Fundamentals of Software Testing

Bernard Homès, founder of the ISTQB and an IEEE Senior member, has over 40 years of experience in IT and software testing. He headed the design for the Advanced Level syllabus and holds numerous testing certifications. This is his eighth book on software testing, which will once more focus on helping customers to improve the efficiency of their testing processes.

Preface xi

Glossary xvii

Chapter 1 Fundamentals of Testing 1

1.1 What is testing? 1

1.1.1 Software and systems context 1

1.1.2 Causes of software defects 3

1.1.3 Role of testing in software development, maintenance and operations 5

1.1.4 Tests and quality 5

1.1.5 Terminology 7

1.2 What is testing? 8

1.2.1 Origin of defects 8

1.2.2 Common goals of testing 9

1.2.3 Examples of objectives for testing 9

1.2.4 Test and debugging 10

1.3 Paradoxes and main principles 11

1.3.1 Testing identifies the presence of defects 11

1.3.2 Exhaustive testing is impossible 11

1.3.3 Early testing 12

1.3.4 Defect clustering 12

1.3.5 Pesticide paradox 13

1.3.6 Testing is context dependent 13

1.3.7 Absence-of-errors fallacy 14

1.4 Test activities, testware and test roles 14

1.4.1 Planning 15

1.4.2 Monitoring and control 16

1.4.3 Test analysis and design 17

1.4.4 Test implementation 20

1.4.5 Test execution 21

1.4.6 Reporting 23

1.4.7 Test completion activities 24

1.4.8 The value of traceability 25

1.4.9 Impact of context on the test process 25

1.5 Roles in testing 26

1.6 Essential skills and "good practices" in testing 26

1.6.1 Generic skills 26

1.6.2 Specific skills 27

1.6.3 Whole team approach 27

1.6.4 Independence of testing 28

1.6.5 Levels of independence 29

1.6.6 Adapt to objectives 30

1.6.7 Destructive or constructive? 32

1.6.8 People skills 32

1.6.9 Change of perspective 33

1.7 Testers and code of ethics (FL 1.6) 33

1.7.1 Public 34

1.7.2 Client and employer 35

1.7.3 Product 35

1.7.4 Judgment 37

1.7.5 Management 37

1.7.6 Profession 38

1.7.7 Colleagues 39

1.7.8 Self 40

1.8 Sample exam questions 41

Chapter 2 Testing Throughout the Software Life Cycle 47

2.1 Testing through the software development life cycle 47

2.1.1 Sequential models 48

2.1.2 Iterative models 50

2.1.3 Incremental model 52

2.1.4 RAD 54

2.1.5 Agile models 55

2.1.6 Selection of a development model 61

2.1.7 Positioning tests 62

2.1.8 Test-first and shift-left approaches 63

2.2 Test levels and test types 64

2.2.1 Component-level testing or component tests 65

2.2.2 Integration-level testing or integration tests 66

2.2.3 System tests 68

2.2.4 Acceptance tests 70

2.2.5 Other levels 72

2.3 Types of tests 72

2.3.1 Functional tests 73

2.3.2 Nonfunctional tests 74

2.3.3 Tests based on the structure or architecture of the software 76

2.3.4 Tests associated with changes 77

2.3.5 Comparisons and examples 79

2.4 Test and maintenance 80

2.4.1 Maintenance context 81

2.4.2 Evolutive maintenance 81

2.4.3 Corrective maintenance 82

2.4.4 Retirement and replacement 83

2.4.5 Regression test policies 83

2.4.6 SLA validation and acceptance 86

2.5 Oracles 86

2.5.1 Problems with oracles 87

2.5.2 Sources of oracles 87

2.5.3 Oracle usage 88

2.6 Process improvements 89

2.6.1 Objectives 89

2.6.2 Measurements 89

2.6.3 Retrospectives and improvements 89

2.7 Specific cases 90

2.7.1 Performance tests 90

2.7.2 Maintainability tests 91

2.8 Sample exam questions 91

Chapter 3 Static Testing 97

3.1 Static techniques and the test process 97

3.2 Review process 100

3.2.1 Types of reviews 101

3.2.2 Roles and responsibilities during reviews 106

3.2.3 Phases of reviews 109

3.2.4 Success factors for reviews 122

3.2.5 Comparison of the types of reviews 123

3.3 Static analysis by tools 125

3.3.1 Types of static analysis 125

3.3.2 Types of defects that can be identified 130

3.3.3 Data flow analysis 131

3.4 Added value of static activities 136

3.5 Sample exam questions 137

Chapter 4 Test Design Techniques 141

4.1 The test development process 143

4.1.1 Terminology 143

4.1.2 Traceability 144

4.2 Categories of test design techniques 146

4.2.1 Black box, white box or gray box 147

4.2.2 Experience-based techniques 148

4.2.3 Test characteristics 149

4.2.4 Limitations and assumptions 149

4.3 Black-box techniques 151

4.3.1 Equivalence partitioning 152

4.3.2 Boundary value analysis 158

4.3.3 Decision tables 161

4.3.4 Other combinational techniques 166

4.3.5 State transition testing 166

4.3.6 Use case testing 173

4.3.7 Limitations and assumptions 175

4.4 Structure-based techniques 175

4.4.1 Statement testing and coverage 178

4.4.2 Decision testing and coverage 183

4.4.3 Other structure-based techniques 186

4.4.4 MC/DC coverage 188

4.4.5 Limitations and assumptions of structure-based testing 189

4.4.6 Coverage level and exit criteria 190

4.5 Experience-based technique 190

4.5.1 Attacks 191

4.5.2 Defect taxonomies 192

4.5.3 Error guessing and ad hoc testing 193

4.5.4 Exploratory testing 194

4.5.5 Limitations and assumptions 195

4.6 Collaboration-based test approaches 196

4.6.1 Collaborative user stories 196

4.6.2 Acceptance criteria 197

4.6.3 Acceptance test-driven development 197

4.7 Choosing test techniques 198

4.8 Sample exam questions 200

Chapter 5 Test Management 209

5.1 Test organization 209

5.1.1 Independence levels 209

5.1.2 Roles and responsibilities 212

5.1.3 Human and contractual aspects 214

5.2 Test planning and estimation 215

5.2.1 Planning and evaluation activities 218

5.2.2 Test planning activities 229

5.2.3 Test documentation 231

5.2.4 Entry and exit criteria for test activities 236

5.3 Test progress monitoring and control (FL 5.3) 240

5.4 Reporting 243

5.4.1 What to report, to whom and how? 243

5.4.2 Statistics and graphs 245

5.5 Transverse processes and activities 248

5.5.1 Test data definition 248

5.5.2 Configuration management (FL 5.4) 249

5.5.3 Change management 250

5.6 Risk management (FL 5.2) 250

5.6.1 Principles of risk management 251

5.6.2 Project risks and product risks 255

5.6.3 Introduction to risk management 256

5.7 Defect management (FL 5.5) 259

5.7.1 Introduction to defect management 260

5.7.2 Defect identification 261

5.7.3 Actions applied to defects 266

5.7.4 Defect disposition 266

5.8 Sample exam questions 267

Chapter 6 Tools Support for Testing 277

6.1 Types of test tools 277

6.1.1 Test tool classification 278

6.1.2 Tools supporting test management 278

6.1.3 Tools supporting requirement management 279

6.1.4 Tools supporting static tests 279

6.1.5 Modeling tools 280

6.1.6 Tools supporting test design and test data creation 280

6.1.7 Tools supporting test execution 281

6.1.8 Tools supporting test environment management 281

6.1.9 Tools supporting test data comparison 282

6.1.10 Tools supporting test coverage measurement 282

6.1.11 Other test supporting tools 282

6.2 Assumptions and limitations of test tools 283

6.2.1 Advantages and risks of the tools 283

6.2.2 Specific considerations for some tools 285

6.3 Selecting and introducing tools in an organization 289

6.3.1 Main principles 289

6.3.2 Tool selection process 290

6.3.3 Test tool implementation 293

6.3.4 To build or to buy test tools? 294

6.4 Sample exam questions 295

Chapter 7 Mock Exam 299

Chapter 8 Templates and Models 313

8.1 Master test plan 313

8.2 Test plan 315

8.2.1 Test plan as per IEEE 829-1998 315

8.2.2 Test plan as per IEEE 829-2008 315

8.3 Test design document 317

8.3.1 Test design specifications as per IEEE 829-1998 317

8.3.2 Test design document as per IEEE 829-2008 317

8.4 Test case 318

8.4.1 Test case document as per IEEE 829-1998 318

8.4.2 Test case document as per IEEE 829-2008 318

8.5 Test procedure 319

8.5.1 Test procedure document as per IEEE 829-1998 319

8.5.2 Test procedure document as per IEEE 829-2008 319

8.6 Test log 320

8.6.1 Test log as per IEEE 829-1998 320

8.6.2 Test log as per IEEE 829-2008 320

8.7 Defect report 320

8.7.1 Defect report as per IEEE 829-1998 320

8.7.2 Defect report as per IEEE 829-2008 321

8.8 Test report 322

8.8.1 Test report as per IEEE 829-1998 322

8.8.2 Interim test report as per IEEE 829-2008 322

8.8.3 Level test report as per IEEE 829-2008 323

8.8.4 Master test report as per IEEE 829-2008 323

Chapter 9 Answers to the Questions 325

9.1 Answers to the end-of-chapter questions 325

9.2 Correct answers to the sample paper questions 327

References 329

Index 333