The State of Experimental Evaluation of Software and Systems in Computer Science

03 Oct
Monday, 10/03/2011 8:00am to 9:30am

Peter Sweeney
IBM TJ Watson

Computer Science Building, Room 150

Host: Systems Lunch

As hardware and software continues to evolve into increasingly complex systems, our ability to understand their behavior and measure their performance is increasingly difficult. 

Nevertheless, many areas of computer science use experiments to identify performance bottlenecks and to evaluate innovations. In the last few years, researchers have identified some disturbing flaws in the way that experiments are performed in computer science. 

This talk presents two of these flaws. 

First, changing a seemingly innocuous aspect of an experimental setup can result in a systems researcher drawing wrong conclusions from an experiment. What appears to be an innocuous aspect in the experimental setup may in fact introduce a significant bias in an evaluation of native (C and C++) applications. 

Second, performance analysts profile their programs to find methods that are worth optimizing: the "hot" methods; however, four commonly used Java profilers (xprof , hprof , jprofile, and yourkit) often disagree on the identity of the hot methods. This talk demonstrates that these profilers all violate a fundamental requirement for sampling based profilers: to be correct, a sampling-based profiler must collect samples randomly. 

Unfortunately, the flaws discussed above are not the full extent of the problem. If computer science is to be taken seriously as a scientific discipline, we as a community need to do a better job designing experiments and evaluating their results. 

I will discuss some current efforts being made by the community to improve experimental evaluation in computer science. 

Bio: 

Peter F. Sweeney is a Research Staff Member in the Program Technology Department at the IBM T.J. Watson Research Center in Hawthorne, New York. His current research interests are performance analysis and tuning of computer systems with a focus on automation. In the past, he has focused on object-oriented optimization. Peter received a Master's degree in Computer Science from Columbia University SEAS and he joined IBM Research in 1985. Peter is a senior member of ACM and a co-author of the paper "Adaptive optimization in the Jalapeno JVM", which received the 2010 ACM SIGPLAN most influential OOPSLA 2000 paper award.