CS 147, Spring 2012
Computer Systems Performance Analysis
Topics
CS 147 is intended to teach you how to evaluate the performance of
computer systems in various ways. The concentration will be on
experimental methods, but I am not promising that I will restrict
myself to experimentation as the only technique. Topics that will
certainly be covered in the course include:
- Types of measurement techniques and how to select them
- Types of workloads and how to select and characterize them
- Common mistakes in performance measurement and benchmarking
- Review of probability and statistics
- Methods of statistical analysis
- Experimental design techniques
- Methods of data presentation
- Methods of measuring data in real computer systems
Other topics that might be covered (depending mostly on time) include:
- Discrete-event simulation techniques
- Random-number generation
- Matching simulation models to real computer systems
- Basic queueing theory
- Markov processes
- M/M/k queues
Goals
When you complete this course, you should be able to:
- Identify mistakes in popular benchmarks
- Design a robust and meaningful experimental measurement
- Analyze the validity of someone else's performance claims
- Statistically analyze the results of an experiment or simulation
- Present data in a clear and cohesive fashion
Prerequisites
The prerequisites listed in the catalog are Math 35 and CS 70.
I also recommend CS 105, CS 131, and CS 140 as
reasonable corequisites, since in your project you will be working
with complex software systems. If you try to take CS 147 having had
only CS 70 and with no other CS courses in the same term, the project
will probably give you trouble.
Textbook
The text book for this course is Raj Jain, The Art of Computer
Systems Performance Analysis, Wiley, 1991. Even if you don't
take the course, I strongly recommend this book to anyone who does experimental
computer science.
Workload
Homework and Exams
There will be a small number of homework assignments early in the
course, to provide you with practice in statistical analysis. There
may also be a midterm exam.
Project
The primary graded component of the course will be a project in
performance analysis. Depending on the number of students, the
projects will be done by small teams of 2-4 people. I
expect that most people will do some sort of experimental measurement,
though other options can be proposed.
Each team will present their project to the class at the end of the
term, and will critique the other teams' projects.
Examples of typical projects
might include:
- Measure and characterize the load on the CS network as a
function of assignment due dates,
- Measure and contrast the performance of several Linux file
systems.
- Compare the performance of an Apple laptop to a PC-based one.
- Identify the impact of an MP3 player on the behavior of a
compiler.
- Compare the performance of an algorithm implemented in Java with
that of the same algorithm implemented in another language.
- Characterize how Knuth's scheduler responds in practice to
various
nice values.
- Characterize the performance of several database systems.
- Measure the performance of a Web site under various conditions.
- Implement several heuristics for the Traveling Salesman
Problem and compare their performance.
- (If simulation is covered) Create a simulation of the serving
lines at Hoch-Shanahan, and propose solutions to the crowding problems.
© 2012, Geoff Kuenning
This page is maintained by Geoff
Kuenning.