A Practical Guide to Continuous Delivery

Eberhard Wolff, a Fellow at innoQ in Germany, has more than 15 years of experience as an architect and consultant working at the intersection of business and technology. He gives talks and keynote addresses at several international conferences, has served on multiple conference program committees, and has written more than 100 articles and books. His technological focus is on modern architectures-often involving cloud, continuous delivery, DevOps, microservices, and NoSQL. He is author of Microservices: Flexible Software Architecture.

Table of Contents

Preface xvii
Part I: Foundations 1
Chapter 1: Continuous Delivery: What and How? 3
1.1 Introduction: What Is Continuous Delivery? 3
1.2 Why Software Releases are So Complicated 3
1.2.1 Continuous Integration Creates Hope 4
1.2.2 Slow and Risky Processes 4
1.2.3 It’s Possible to be Fast 4
1.3 Values of Continuous Delivery 4
1.3.1 Regularity 5
1.3.2 Traceability/Confirmability 6
1.3.3 Regression 6
1.4 Benefits of Continuous Delivery 6
1.4.1 Continuous Delivery for Time to Market 7
1.4.2 One Example 7
1.4.3 Implementing a Feature and Bringing It into Production 7
1.4.4 On to the Next Feature 7
1.4.5 Continuous Delivery Generates Competitive Advantages 8
1.4.6 Without Continuous Delivery 8
1.4.7 Continuous Delivery and Lean Startup 9
1.4.8 Effects on the Development Process 9
1.4.9 Continuous Delivery to Minimize Risk 10
1.4.10 Faster Feedback and Lean 13
1.5 Generations and Structure of a Continuous Delivery Pipeline 14
1.5.1 The Example 16
1.6 Conclusion 17
Endnotes 17
Chapter 2: Providing Infrastructure 19
2.1 Introduction 19
2.1.1 Infrastructure Automation: An Example 20
2.2 Installation Scripts 21
2.2.1 Problems of Classical Installation Scripts 21
2.3 Chef 24
2.3.1 Chef versus Puppet 25
2.3.2 Other Alternatives 26
2.3.3 Technical Foundations 26
2.3.4 Chef Solo 33
2.3.5 Chef Solo: Conclusion 35
2.3.6 Knife and Chef Server 35
2.3.7 Chef Server: Conclusion 39
2.4 Vagrant 40
2.4.1 An Example with Chef and Vagrant 41
2.4.2 Vagrant: Conclusion 43
2.5 Docker 43
2.5.1 Docker’s Solution 44
2.5.2 Creating Docker Containers 46
2.5.3 Running the Example Application with Docker 49
2.5.4 Docker and Vagrant 51
2.5.5 Docker Machine 53
2.5.6 Complex Configurations with Docker 55
2.5.7 Docker Compose 57
2.6 Immutable Server 60
2.6.1 Disadvantages of Idempotency 60
2.6.2 Immutable Server and Docker 61
2.7 Infrastructure as Code 61
2.7.1 Testing Infrastructure as Code 63
2.8 Platform as a Service (PaaS) 63
2.9 Handling Data and Databases 65
2.9.1 Handling Schemas 66
2.9.2 Test and Master Data 67
2.10 Conclusion 68
Endnotes 69

Part II: The Continuous Delivery Pipeline 71
Chapter 3: Build Automation and Continuous Integration 73
3.1 Introduction 73
3.1.1 Build Automation: An Example 74
3.2 Build Automation and Build Tools 74
3.2.1 Build Tools in the Java World 75
3.2.2 Ant 76
3.2.3 Maven 76
3.2.4 Gradle 81
3.2.5 Additional Build Tools 84
3.2.6 Choosing the Right Tool 84
3.3 Unit Tests 86
3.3.1 Writing Good Unit Tests 87
3.3.2 TDD—Test-Driven Development 89
3.3.3 Clean Code and Software Craftsmanship 90
3.4 Continuous Integration 91
3.4.1 Jenkins 92
3.4.2 Continuous Integration Infrastructure 97
3.4.3 Conclusion 99
3.5 Measuring Code Quality 101
3.5.1 SonarQube 102
3.6 Managing Artifacts 105
3.6.1 Integration into the Build 108
3.6.2 Advanced Features of Repositories 109
3.7 Conclusion 110
Endnotes 111
Chapter 4: Acceptance Tests 113
4.1 Introduction 113
4.1.1 Acceptance Tests: An Example 113
4.2 The Test Pyramid 114
4.3 What Are Acceptance Tests? 117
4.3.1 Automated Acceptance Tests 117
4.3.2 More Than Just an Increase in Efficiency 117
4.3.3 Manual Tests 119
4.3.4 What about the Customer? 119
4.3.5 Acceptance versus Unit Tests 119
4.3.6 Test Environments 120
4.4 GUI-Based Acceptance Tests 121
4.4.1 Problems of GUI Tests 121
4.4.2 Abstractions against Fragile GUI Tests 122
4.4.3 Automation with Selenium 122
4.4.4 Web Driver API 122
4.4.5 Tests without Web Browser: HtmlUnit 123
4.4.6 Selenium Web Driver API 123
4.4.7 Selenium IDE 123
4.4.8 Problems with Automated GUI Tests 124
4.4.9 Executing GUI Tests 125
4.4.10 Exporting the Tests as Code 125
4.4.11 Manual Modifications of the Test Cases 125
4.4.12 Test Data 126
4.4.13 Page Object 126
4.5 Alternative Tools for GUI Tests 127
4.5.1 PhantomJS 127
4.5.2 Windmill 127
4.6 Textual Acceptance Tests 129
4.6.1 Behavior-Driven Development 129
4.6.2 Different Adaptors 131
4.7 Alternative Frameworks 133
4.8 Strategies for Acceptance Tests 134
4.8.1 The Right Tool 134
4.8.2 Rapid Feedback 135
4.8.3 Test Coverage 135
4.9 Conclusion 136
Endnotes 137
Chapter 5: Capacity Tests 139
5.1 Introduction 139
5.1.1 Capacity Tests: An Example 139
5.2 Capacity Tests—How? 140
5.2.1 Objectives of Capacity Tests 140
5.2.2 Data Volumes and Environments 140
5.2.3 Performance Tests Only at the End of the Implementation? 141
5.2.4 Capacity Tests = Risk Management 141
5.2.5 Simulating Users 141
5.2.6 Documenting Performance Requirements 142
5.2.7 Hardware for Capacity Tests 142
5.2.8 Cloud and Virtualization 143
5.2.9 Minimizing Risk by Continuous Testing 143
5.2.10 Capacity Tests—Sensible or Not? 144
5.3 Implementing Capacity Tests 145
5.4 Capacity Tests with Gatling 146
5.4.1 Demo versus Real Life 150
5.5 Alternatives to Gatling 151
5.5.1 Grinder 152
5.5.2 Apache JMeter 152
5.5.3 Tsung 152
5.5.4 Commercial Solutions 152
5.6 Conclusion 153
Endnotes 153
Chapter 6: Exploratory Testing 155
6.1 Introduction 155
6.1.1 Exploratory Tests: An Example 155
6.2 Why Exploratory Tests? 156
6.2.1 Sometimes Manual Testing Is Still Better 156
6.2.2 Test by the Customers 156
6.2.3 Manual Tests for Non-Functional Requirements 157
6.3 How to Go About It? 157
6.3.1 Missions Guide the Tests 157
6.3.2 Automated Environment 158
6.3.3 Showcases as a Basis 158
6.3.4 Example: An E-Commerce Application 158
6.3.5 Beta Tests 159
6.3.6 Session-Based Tests 159
6.4 Conclusion 162
Endnotes 162
Chapter 7: Deploy—The Rollout in Production 163
7.1 Introduction 163
7.1.1 Deployment: An Example 164
7.2 Rollout and Rollback 164
7.2.1 Benefits 164
7.2.2 Disadvantages 164
7.3 Roll Forward 165
7.3.1 Benefits 165
7.3.2 Disadvantages 166
7.4 Blue/Green Deployment 166
7.4.1 Benefits 166
7.4.2 Disadvantages 167
7.5 Canary Releasing 168
7.5.1 Benefits 168
7.5.2 Disadvantages 169
7.6 Continuous Deployment 169
7.6.1 Benefits 170
7.6.2 Disadvantages 171
7.7 Virtualization 171
7.7.1 Physical Hosts 172
7.8 Beyond Web Applications 172
7.9 Conclusion 174
Endnotes 174
Chapter 8: Operations 175
8.1 Introduction 175
8.1.1 Operate—An Example 176
8.2 Challenges in Operations 176
8.3 Log Files 177
8.3.1 What Should Be Logged? 178
8.3.2 Tools for Processing Log Files 180
8.3.3 Logging in the Example Application 181
8.4 Analyzing Logs of the Example Application 182
8.4.1 Analyses with Kibana 184
8.4.2 ELK—Scalability 185
8.5 Other Technologies for Logs 189
8.6 Advanced Log Techniques 190
8.6.1 Anonymization 190
8.6.2 Performance 191
8.6.3 Time 191
8.6.4 Ops Database 191
8.7 Monitoring 191
8.8 Metrics with Graphite 192
8.9 Metrics in the Example Application 194
8.9.1 Structure of the Example 194
8.10 Other Monitoring Solutions 197
8.11 Additional Challenges When Operating an Application 198
8.11.1 Scripts 198
8.11.2 Applications in a Client’s Data Center 198
8.12 Conclusion 199
Endnotes 199

Part III: Management, Organization, and Architecture for Continuous Delivery 201
Chapter 9 Introducing Continuous Delivery into Your Enterprise 203
9.1 Introduction 203
9.2 Continuous Delivery Right from the Start 203
9.3 Value Stream Mapping 204
9.3.1 Value Stream Mapping Describes the Sequence of Events 205
9.3.2 Optimizations 205
9.4 Additional Measures for Optimization 206
9.4.1 Quality Investments 207
9.4.2 Costs 207
9.4.3 Benefits 207
9.4.4 Do not Check in on a Red Build! 208
9.4.5 Stop the Line 208
9.4.6 5 Whys 209
9.4.7 DevOps 209
9.5 Conclusion 210
Endnotes 211
Chapter 10: Continuous Delivery and DevOps 213
10.1 Introduction 213
10.2 What Is DevOps? 213
10.2.1 Problems 214
10.2.2 The Client Perspective 214
10.2.3 Pioneer: Amazon 215
10.2.4 DevOps 215
10.3 Continuous Delivery and DevOps 217
10.3.1 DevOps: More Than Continuous Delivery 217
10.3.2 Individual Responsibility and Self-Organization 218
10.3.3 Technology Decisions 218
10.3.4 Less Central Control 219
10.3.5 Technology Pluralism 219
10.3.6 Exchange Between Teams 220
10.3.7 Architecture 220
10.4 Continuous Delivery without DevOps? 221
10.4.1 Terminating the Continuous Delivery Pipeline 222
10.5 Conclusion 223
Endnotes 224
Chapter 11: Continuous Delivery, DevOps, and Software Architecture 225
11.1 Introduction 225
11.2 Software Architecture 225
11.2.1 Why Software Architecture? 226
11.3 Optimizing Architecture for Continuous Delivery 228
11.3.1 Smaller Deployment Units 229
11.4 Interfaces 230
11.4.1 Postel’s Law or the Robustness Principle 231
11.4.2 Design for Failure 231
11.4.3 State 232
11.5 Databases 233
11.5.1 Keeping Databases Stable 233
11.5.2 Database = Component 234
11.5.3 Views and Stored Procedures 234
11.5.4 A Database per Component 235
11.5.5 NoSQL Databases 235
11.6 Microservices 235
11.6.1 Microservices and Continuous Delivery 236
11.6.2 Introducing Continuous Delivery with Microservices 237
11.6.3 Microservices Entail Continuous Delivery 237
11.6.4 Organization 237
11.7 Handling New Features 238
11.7.1 Feature Branches 238
11.7.2 Feature Toggles 238
11.7.3 Benefits 239
11.7.4 Use Cases for Feature Toggles 240
11.7.5 Disadvantages 240
11.8 Conclusion 241
Endnotes 242
Chapter 12: Conclusion: What Are the Benefits? 243
Endnotes 244
Index 245