Experimentation in Software Engineering

CLAES WOHLIN is a professor of software engineering at Blekinge Institute of Technology. His research interests include empirical methods in software engineering, software processes and software quality. He is a member of the Royal Swedish Academy of Engineering Sciences and a senior member of the IEEE. PER RUNESON is a professor of software engineering at Lund University. His research interests include empirical research on software testing and quality, and methods for such research. Prior to this, he worked as a consulting expert in industry, and he is a senior member of the IEEE. MARTIN HOST is a professor of software engineering at Lund University. His research interests include software quality and software process improvement, as well as empirical research methods in software engineering. MAGNUS C. OHLSSON is a quality assurance specialist at System Verification AB in Malmo, focusing on process improvement. His primary field of interest is verification and validation, with an emphasis on achieving proper quality efficiently throughout every step of the development process. BJORN REGNELL is a professor of software engineering at Lund University. His research interests include software requirement engineering, software product management, and empirical research on software engineering. ANDERS WESSLEN is a senior system architect at ST-Ericsson in Lund. His focus is on system-wide architectures, as well as system design and requirements.

1 Introduction.- 1.1 Software engineering context.- 1.2 Science and software engineering.- 2 Empirical strategies.- 2.1 Overview of empirical strategies.- 2.2 Surveys.- 2.3 Case studies.- 2.4 Experiments.- 2.5 Empirical strategies comparison.- 2.6 Empiricism in a software engineering context.- 3 Measurement.- 3.1 Basic concepts.- 3.2 Measurements in software engineering.- 4 Experiment process.- 4.1 Variables, treatments, objects and subjects.- 4.2 Process.- 5 Definition.- 5.1 Define experiment.- 5.2 Example.- 5.3 Summary.- 6 Planning.- 6.1 Context selection.- 6.2 Hypothesis formulation.- 6.3 Variables selection.- 6.4 Selection of subjects.- 6.5 Experiment design.- 6.6 Instrumentation.- 6.7 Validity evaluation.- 6.8 Detailed description of validity threats.- 6.9 Priority among types of validity threats.- 7 Operation.- 7.1 Preparation.- 7.2 Execution.- 7.3 Data validation.- 8 Analysis and interpretation.- 8.1 Descriptive statistics.- 8.2 Data set reduction.- 8.3 Hypothesis testing.- 9 Presentation and package.- 9.1 An experiment report outline.- 10 Literature survey.- 10.1 Inspection experiments.- 10.2 Other experiments in Software Engineering.- 10.3 Resources.- 11 Example: Experiment process.- 11.1 Definition.- 11.2 Planning.- 11.3 Operation.- 11.4 Analysis and interpretation.- 11.5 Summary and conclusions.- 12 Example: C versus C++.- 12.1 Introduction and problem statement.- 12.2 Experiment planning.- 12.3 Analysis and interpretation.- 12.4 Conclusions and further work.- 13 Exercises.- 13.1 Understanding.- 13.2 Training.- 13.3 Reviewing.- 13.4 Assignments.- Appendix A: Statistical tables.- Appendix B: Experiment process overview.- References.- About the authors.