The DevOps Handbook

CONTENTS

Figures & Tables xii

Note from the Publisher on the Second Edition xv

Foreword to the Second Edition: Nicole Forsgren xix

Foreword to the First Edition: John Allspaw xxi

Preface: Aha! xxiii

Introduction xxxi

Part I—The Three Ways

Part I Introduction 3

01 Agile, Continuous Delivery, and the Three Ways 7

NEW Case Study: Approaching Cruising Altitude:

American Airlines' DevOps Journey Part 1 (2020) 15

02 The First Way: The Principles of Flow 19

NEW Case Study: Flow and Constraint Management in

Healthcare (2021) 29

03 The Second Way: The Principles of Feedback 33

NEW Case Study: Pulling the Andon Cord at

Excella (2019) 39

04 The Third Way: The Principles of Continual Learning and

Experimentation 45

NEW Case Study: The Story of Bell Labs 54

Part 1 Conclusion 57

Part II—Where to Start

Part II Introduction 61

05 Selecting Which Value Stream to Start With 63

Case Study: Nordstrom's DevOps Transformation 63

NEW Case Study: Kessel Run: The Brownfield

Transformation of a Mid-Air Refueling System (2020) 69

NEW Case Study: Scaling DevOps Across the Business:

American Airlines' DevOps Journey (Part 2) (2020) 74

viii CONTENTS

NEW Case Study: Saving the Economy From Ruin (With a

Hyperscale PaaS) (2021) 77

06 Understanding the Work in Our Value Stream, Making it

Visible, and Expanding it Across the Organization 81

Case Study: Nordstrom's Experience with Value Stream

Mapping 81

Case Study: Operation InVersion at LinkedIn (2011) 91

07 How to Design Our Organization and Architecture

with Conway's Law in Mind 97

Case Study: Conway's Law at Etsy 98

Case Study: API Enablement at Target (2015) 112

08 How to Get Great Outcomes by Integrating Operations

into the Daily Work of Development 115

Case Study: Big Fish Games 115

NEW Case Study: Better Ways of Working at Nationwide

Building Society 124

Part II Conclusion 129

Part III—The First Way: The Technical Practices of Flow

Part III Introduction 133

09 Create the Foundations of Our Deployment Pipeline 135

Case Study: Enterprise Data Warehouse 135

NEW Case Study: How a Hotel Company Ran $30B of

Revenue in Containers (2020) 143

10 Enable Fast and Reliable Automated Testing 147

Case Study: Google Web Server 148

11 Enable and Practice Continuous Integration 167

Case Study: HP LaserJet Firmware 168

Case Study: Continuous Integration of Bazaarvoice (2012) 173

12 Automate and Enable Low-Risk Releases 177

Case Study: Daily Deployments at CSG International (2013) 181

Case Study: Etsy—Self-Service Developer Deployment:

An Example of Continuous Deployment (2014) 186

Case Study: Dixons Retail—Blue-Green Deployment for

Point-of-Sale System (2008) 193

Case Study: Dark Launch of Facebook Chat (2008) 198

CONTENTS ix

NEW Case Study: Creating a Win-Win for Dev & Ops

at CSG (2016) 201

13 Architect for Low-Risk Releases 207

Case Study: Evolutionary Architecture at Amazon (2002) 202

Case Study: Strangler Pattern at Blackboard Learn (2011) 215

Part III Conclusion 219

Part IV—The Second Way: The Technical Practices of Feedback

Part IV Introduction 223

14 Create Telemetry to Enable Seeing and Solving Problems 225

Case Study: DevOps Transformation at Etsy (2012) 226

Case Study: Creating Self-Service Metrics at

LinkedIn (2011) 237

15 Analyze Telemetry to Better Anticipate Problems and

Achieve Goals 245

Case Study: Telemetry at Netflix (2012–2020) 245

Case Study: Auto-Scaling Capacity at Netflix (2012) 251

Case Study: Advanced Anomaly Detection (2014) 255

16 Enable Feedback So Development and Operations Can

Safely Deploy Code 259

Case Study: Right Media (2006) 259

Case Study: The Launch and Hand-Off Readiness

Review Google (2010) 269

17 Integrate Hypothesis-Driven Development and

A/B Testing into Our Daily Work 273

Case Study: Hypothesis-Driven Development

at Intuit, Inc. (2012) 273

Case Study: Doubling Revenue Growth through Fast

Release Cycle Experimentation at Yahoo! Answers (2010) 278

18 Create Review and Coordination Processes to Increase

Quality of Our Current Work 281

Case Study: Peer Review at GitHub (2011) 286

NEW Case Study: From Six-Eye Principle to Release at

Scale at Adidas (2021) 286

Case Study: Code Reviews at Google (2010) 290

Case Study: Pair Programming Replacing Broken Code

Review Processes at Pivotal Labs (2011) 293

Part IV Conclusion 299

x CONTENTS

Part V—The Third Way: The Technical Practices of Continual Learning

and Experimentation

Part V Introduction 303

19 Enable and Inject Learning into Daily Work 305

Case Study: AWS US-EAST and Netflix (2011) 305

NEW Case Study: Turning an Outage into a Powerful

Learning Opportunity at CSG (2020) 318

20 Convert Local Discoveries into Global Improvements 321

Case Study: Standardizing a New Technology Stack

at Etsy (2010) 332

NEW Case Study: Crowdsourcing Technology Governance at

Target (2018) 333

21 Reserve Time to Create Organizational Learning

and Improvement 335

Case Study: 30-Day Challenge at Target (2015) 335

Case Study: Internal Technology Conferences at

Nationwide Insurance, Capital One, and Target (2014) 342

Part V Conclusion 347

Part VI—The Technological Practices of Integrating Information

Security, Change Management, and Compliance

Part VI Introduction 351

22 Information Security Is Everyone's Job Every Day 353

Case Study: Static Security Testing at Twitter (2009) 360

Case Study: 18F Automating Compliance for the Federal

Government with Compliance Masonry (2016) 369

Case Study: Instrumenting the Environment at

Etsy (2010) 373

NEW Case Study: Shifting Security Left at Fannie

Mae (2020) 376

23 Protecting the Deployment Pipeline, and Integrating into

Change Management and Other Security and Compliance

Controls 379

Case Study: Automated Infrastructure Changes as

Standard Changes at Salesforce.com (2012) 383

CONTENTS xi

Case Study: PCI Compliance and a Cautionary Tale of

Separating Duties at Etsy (2014) 385

NEW Case Study: Biz and Tech Partnership toward 10 "No Fear

Releases" Per Day at Capital One (2020) 387

Case Study: Proving Compliance in Regulated

Environments (2015) 389

Case Study: Relying on Production Telemetry for

ATM Systems 392

Part VI Conclusion 395

A Call to Action: Conclusion to the DevOps Handbook 397

Afterword to the Second Edition 401

Appendices 409

Bibliography 423

Notes 441

Index 461

Acknowledgments 479

Author Biographies 482