The DevOps Handbook

Gene Kim is a multiple award-winning CTO, researcher and bestselling author. He was founder and CTO of Tripwire for 13 years. He has written six books. Since 2014, he has been the founder of IT Revolution and the organizer of the DevOps Enterprise Summit. Jez Humble is co-author of several books on software including Shingo Publication Award winner Accelerate, Jolt Award winner Continuous Delivery, and The DevOps Handbook. He has spent his career tinkering with code, infrastructure, and product development in companies of varying sizes across three continents. He works for Google Cloud as a technology advocate and teaches at UC Berkeley. Patrick Debois is the Director of DevOps Relations and Advisor at Snyk. In 2009 he coined the word DevOps by organizing the first devopsdays event, as is now often known as one of the grandfathers of DevOps. He organized conferences all over the world to collect and spread new ideas. John Willis is Senior Director of the Global Transformation Office at Red Hat. Prior to Red Hat, he was the Director of Ecosystem Development for Docker. John was one of the earliest cloud evangelists and is considered one of the founders of the DevOps movement. John is the author of 7 IBM Redbooks, as well as co-author of the The DevOps Handbook and Beyond the Phoenix Project. Nicole Forsgren, PhD, is a Partner at Microsoft Research. She is author of the Shingo Publication Award-winning book Accelerate: The Science of Lean Software and DevOps and is best known as lead investigator on the largest DevOps studies to date. She has been a successful entrepreneur (with an exit to Google), professor, performance engineer, and sysadmin. Her work has been published in several peer-reviewed journals.

CONTENTS

Figures & Tables xii

Note from the Publisher on the Second Edition xv

Foreword to the Second Edition: Nicole Forsgren xix

Foreword to the First Edition: John Allspaw xxi

Preface: Aha! xxiii

Introduction xxxi

Part I—The Three Ways

Part I Introduction 3

01 Agile, Continuous Delivery, and the Three Ways 7

NEW Case Study: Approaching Cruising Altitude:

American Airlines' DevOps Journey Part 1 (2020) 15

02 The First Way: The Principles of Flow 19

NEW Case Study: Flow and Constraint Management in

Healthcare (2021) 29

03 The Second Way: The Principles of Feedback 33

NEW Case Study: Pulling the Andon Cord at

Excella (2019) 39

04 The Third Way: The Principles of Continual Learning and

Experimentation 45

NEW Case Study: The Story of Bell Labs 54

Part 1 Conclusion 57

Part II—Where to Start

Part II Introduction 61

05 Selecting Which Value Stream to Start With 63

Case Study: Nordstrom's DevOps Transformation 63

NEW Case Study: Kessel Run: The Brownfield

Transformation of a Mid-Air Refueling System (2020) 69

NEW Case Study: Scaling DevOps Across the Business:

American Airlines' DevOps Journey (Part 2) (2020) 74

viii CONTENTS

NEW Case Study: Saving the Economy From Ruin (With a

Hyperscale PaaS) (2021) 77

06 Understanding the Work in Our Value Stream, Making it

Visible, and Expanding it Across the Organization 81

Case Study: Nordstrom's Experience with Value Stream

Mapping 81

Case Study: Operation InVersion at LinkedIn (2011) 91

07 How to Design Our Organization and Architecture

with Conway's Law in Mind 97

Case Study: Conway's Law at Etsy 98

Case Study: API Enablement at Target (2015) 112

08 How to Get Great Outcomes by Integrating Operations

into the Daily Work of Development 115

Case Study: Big Fish Games 115

NEW Case Study: Better Ways of Working at Nationwide

Building Society 124

Part II Conclusion 129

Part III—The First Way: The Technical Practices of Flow

Part III Introduction 133

09 Create the Foundations of Our Deployment Pipeline 135

Case Study: Enterprise Data Warehouse 135

NEW Case Study: How a Hotel Company Ran $30B of

Revenue in Containers (2020) 143

10 Enable Fast and Reliable Automated Testing 147

Case Study: Google Web Server 148

11 Enable and Practice Continuous Integration 167

Case Study: HP LaserJet Firmware 168

Case Study: Continuous Integration of Bazaarvoice (2012) 173

12 Automate and Enable Low-Risk Releases 177

Case Study: Daily Deployments at CSG International (2013) 181

Case Study: Etsy—Self-Service Developer Deployment:

An Example of Continuous Deployment (2014) 186

Case Study: Dixons Retail—Blue-Green Deployment for

Point-of-Sale System (2008) 193

Case Study: Dark Launch of Facebook Chat (2008) 198

CONTENTS ix

NEW Case Study: Creating a Win-Win for Dev & Ops

at CSG (2016) 201

13 Architect for Low-Risk Releases 207

Case Study: Evolutionary Architecture at Amazon (2002) 202

Case Study: Strangler Pattern at Blackboard Learn (2011) 215

Part III Conclusion 219

Part IV—The Second Way: The Technical Practices of Feedback

Part IV Introduction 223

14 Create Telemetry to Enable Seeing and Solving Problems 225

Case Study: DevOps Transformation at Etsy (2012) 226

Case Study: Creating Self-Service Metrics at

LinkedIn (2011) 237

15 Analyze Telemetry to Better Anticipate Problems and

Achieve Goals 245

Case Study: Telemetry at Netflix (2012–2020) 245

Case Study: Auto-Scaling Capacity at Netflix (2012) 251

Case Study: Advanced Anomaly Detection (2014) 255

16 Enable Feedback So Development and Operations Can

Safely Deploy Code 259

Case Study: Right Media (2006) 259

Case Study: The Launch and Hand-Off Readiness

Review Google (2010) 269

17 Integrate Hypothesis-Driven Development and

A/B Testing into Our Daily Work 273

Case Study: Hypothesis-Driven Development

at Intuit, Inc. (2012) 273

Case Study: Doubling Revenue Growth through Fast

Release Cycle Experimentation at Yahoo! Answers (2010) 278

18 Create Review and Coordination Processes to Increase

Quality of Our Current Work 281

Case Study: Peer Review at GitHub (2011) 286

NEW Case Study: From Six-Eye Principle to Release at

Scale at Adidas (2021) 286

Case Study: Code Reviews at Google (2010) 290

Case Study: Pair Programming Replacing Broken Code

Review Processes at Pivotal Labs (2011) 293

Part IV Conclusion 299

x CONTENTS

Part V—The Third Way: The Technical Practices of Continual Learning

and Experimentation

Part V Introduction 303

19 Enable and Inject Learning into Daily Work 305

Case Study: AWS US-EAST and Netflix (2011) 305

NEW Case Study: Turning an Outage into a Powerful

Learning Opportunity at CSG (2020) 318

20 Convert Local Discoveries into Global Improvements 321

Case Study: Standardizing a New Technology Stack

at Etsy (2010) 332

NEW Case Study: Crowdsourcing Technology Governance at

Target (2018) 333

21 Reserve Time to Create Organizational Learning

and Improvement 335

Case Study: 30-Day Challenge at Target (2015) 335

Case Study: Internal Technology Conferences at

Nationwide Insurance, Capital One, and Target (2014) 342

Part V Conclusion 347

Part VI—The Technological Practices of Integrating Information

Security, Change Management, and Compliance

Part VI Introduction 351

22 Information Security Is Everyone's Job Every Day 353

Case Study: Static Security Testing at Twitter (2009) 360

Case Study: 18F Automating Compliance for the Federal

Government with Compliance Masonry (2016) 369

Case Study: Instrumenting the Environment at

Etsy (2010) 373

NEW Case Study: Shifting Security Left at Fannie

Mae (2020) 376

23 Protecting the Deployment Pipeline, and Integrating into

Change Management and Other Security and Compliance

Controls 379

Case Study: Automated Infrastructure Changes as

Standard Changes at Salesforce.com (2012) 383

CONTENTS xi

Case Study: PCI Compliance and a Cautionary Tale of

Separating Duties at Etsy (2014) 385

NEW Case Study: Biz and Tech Partnership toward 10 "No Fear

Releases" Per Day at Capital One (2020) 387

Case Study: Proving Compliance in Regulated

Environments (2015) 389

Case Study: Relying on Production Telemetry for

ATM Systems 392

Part VI Conclusion 395

A Call to Action: Conclusion to the DevOps Handbook 397

Afterword to the Second Edition 401

Appendices 409

Bibliography 423

Notes 441

Index 461

Acknowledgments 479

Author Biographies 482