Systems Benchmarking

Samuel Kounev is a Professor of Computer Science and Chair of Software Engineering at the University of Würzburg (Germany). He has been actively involved in the Standard Performance Evaluation Corporation (SPEC), the largest standardization consortium in the area of computer systems benchmarking, since 2002. He serves as the elected chair of the SPEC Research Group, which he initiated in 2010 with the goal of providing a platform for collaborative research efforts between academia and industry in the area of quantitative system evaluation. Samuel is also co-founder and Steering Committee Co-chair of several conferences in the field, including the ACM/SPEC International Conference on Performance Engineering (ICPE) and the IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS). He has published extensively in the area of systems benchmarking, modeling, and evaluation of performance, energy efficiency, reliability, and security. Klaus-Dieter Lange is a technologist at Hewlett Packard Enterprise (HPE), where he started his professional career in 1997, with a focus on performance and workload characterization, industry-standard benchmark development, server efficiency, and the design of secure enterprise solutions. He has been active in several of the SPEC Steering and Sub-committees since 2005, serves on the SPEC Board of Directors, and has been on the ICPE Steering Committee since its inception. Klaus is the founding chair of the SPECpower Committee, which developed and maintains under his technical leadership among others the SPECpower_ssj2008 benchmark, the SPEC PTDaemon, and the Server Efficiency Rating Tool (SERT) suite. Jóakim von Kistowski is a software architect at DATEV eG, where he focuses on software performance and load testing, driving benchmarking method adoption, software architecture, software testing and evaluation. Jóakim has a strong SPEC background, actively contributing to the SPECpower Committee and serving as elected chair of the SPEC RG Power Working Group.

Part I Foundations.- 1 Benchmarking Basics.- 2 Review of Basic Probability and Statistics.- 3 Metrics.- 4 Statistical Measurements.- 5 Experimental Design.- 6 Measurement Techniques.- 7 Operational Analysis and Basic Queueing Models.- 8 Workloads.- 9 Standardization.- Part II Applications.- 10 The SPEC CPU Benchmark Suite.- 11 Benchmarking the Energy Efficiency of Servers.- 12 Virtualization Benchmarks.- 13 Storage Benchmarks.- 14 TeaStore: A Micro-Service Reference Application for Research Use.- 15 Elasticity of Cloud Platforms.- 16 Performance Isolation.- 17 Resource Demand Estimation.- 18 Software and System Security.