Establishing SRE Foundations

Foreword xxi
Preface xxv
Acknowledgments xxix
About the Author xxxiii

Part I: Foundations 1 Chapter 1: Introduction to SRE 3
1.1 Why SRE? 3
1.2 Alignment Using SRE 13
1.3 Why Does SRE Work? 17
1.4 Summary 19

Chapter 2: The Challenge 21
2.1 Misalignment 22
2.2 Collective Ownership 23
2.3 Ownership Using SRE 25
2.4 The Challenge Statement 38
2.5 Coaching 39
2.6 Summary 41

Chapter 3: SRE Basic Concepts 43
3.1 Service Level Indicators 43
3.2 Service Level Objectives 45
3.3 Error Budgets 47
3.4 Error Budget Policies 53
3.5 SRE Concept Pyramid 55
3.6 Alignment Using the SRE Concept Pyramid 59
3.7 Summary 63

Chapter 4: Assessing the Status Quo 65
4.1 Where Is the Organization? 65
4.2 Where Are the People? 69
4.3 Where Is the Tech? 71
4.4 Where Is the Culture? 74
4.5 Where Is the Process? 79
4.6 SRE Maturity Model 81
4.7 Posing Hypotheses 81
4.8 Summary 86

Part II: Running the Transformation 87 Chapter 5: Achieving Organizational Buy-In 89
5.1 Getting People Behind SRE 89
5.2 SRE Marketing Funnel 92
5.3 SRE Coaches 96
5.4 Top-Down Buy-In 99
5.5 Bottom-Up Buy-In 117
5.6 Lateral Buy-In 122
5.7 Buy-In Staggering 123
5.8 Team Coaching 124
5.9 Traversing the Organization 126
5.10 Organizational Coaching 131
5.11 Summary 133

Chapter 6: Laying Down the Foundations 135
6.1 Introductory Talks by Team 135
6.2 Conveying the Basics 136
6.3 SLI Standardization 147
6.4 Enabling Logging 154
6.5 Teaching the Log Query Language 156
6.6 Defining Initial SLOs 157
6.7 Default SLOs 163
6.8 Providing Basic Infrastructure 164
6.9 Engaging Champions 167
6.10 Dealing with Detractors 168
6.11 Creating Documentation 171
6.12 Broadcast Success 172
6.13 Summary 174

Chapter 7: Reacting to Alerts on SLO Breaches 175
7.1 Environment Selection 175
7.2 Responsibilities 177
7.3 Ways of Working 180
7.4 Setting Up On-Call Rotations 185
7.5 On-Call Management Tools 188
7.6 Out-of-Hours On-Call 193
7.7 Systematic Knowledge Sharing 196
7.8 Broadcast Success 208
7.9 Summary 209

Chapter 8: Implementing Alert Dispatching 211
8.1 Alert Escalation 212
8.2 Defining an Alert Escalation Policy 214
8.3 Defining Stakeholder Groups 216
8.4 Triggering Stakeholder Notifications 218
8.5 Defining Stakeholder Rings 219
8.6 Defining Effective Stakeholder Notifications 222
8.7 Getting the Stakeholders Subscribed 225
8.8 Broadcast Success 226
8.9 Summary 227

Chapter 9: Implementing Incident Response 229
9.1 Incident Response Foundations 229
9.2 Incident Priorities 230
9.3 Complex Incident Coordination 248
9.4 Incident Postmortems 268
9.5 Effective Postmortem Criteria 269
9.6 Mashing Up the Tools 294
9.7 Service Status Broadcast 298
9.8 Documenting the Incident Response Process 301
9.9 Broadcast Success 302
9.10 Summary 303

Chapter 10: Setting Up an Error Budget Policy 305
10.1 Motivation 305
10.2 Terminology 307
10.3 Error Budget Policy Structure 308
10.4 Error Budget Policy Conditions 309
10.5 Error Budget Policy Consequences 311
10.6 Error Budget Policy Governance 312
10.7 Extending the Error Budget Policy 314
10.8 Agreeing to the Error Budget Policy 318
10.9 Storing the Error Budget Policy 319
10.10 Enacting the Error Budget Policy 320
10.11 Reviewing the Error Budget Policy 321
10.12 Related Concepts 322
10.13 Summary 324

Chapter 11: Enabling Error Budget-Based Decision-Making 325
11.1 Reliability Decision-Making Taxonomy 325
11.2 Implementing SRE Indicators 330
11.3 Process Indicators, Not People KPIs 359
11.4 Decisions Versus Indicators 359
11.5 Decision-Making Workflows 362
11.6 Summary 388

Chapter 12: Implementing Organizational Structure 391
12.1 SRE Principles Versus Organizational Structure 393
12.2 Who Builds It, Who Runs It? 394
12.3 You Build It, You Run It 403
12.4 You Build It, You and SRE Run It 406
12.5 You Build It, SRE Run It 421
12.6 Cost Optimization 424
12.7 Team Topologies 426
12.8 Choosing a Model 432
12.9 A New Role: SRE 440
12.10 SRE Career Path 450
12.11 Communicating the Chosen Model 456
12.12 Introducing the Chosen Model 457
12.13 Summary 462

Part III: Measuring and Sustaining the Transformation 465 Chapter 13: Measuring the SRE Transformation 467
13.1 Testing Transformation Hypotheses 467
13.2 Outages Not Detected Internally 469
13.3 Services Exhausting Error Budgets Prematurely 470
13.4 Executives' Perceptions 471
13.5 Reliability Perception by Users and Partners 472
13.6 Summary 473

Chapter 14: Sustaining the SRE Movement 475
14.1 Maturing the SRE CoP 475
14.2 SRE Minutes 475
14.3 Availability Newsletter 476
14.4 SRE Column in the Engineering Blog 477
14.5 Promote Long-Form SRE Wiki Articles 477
14.6 SRE Broadcasting 478
14.7 Combining SRE and CD Indicators 479
14.8 SRE Feedback Loops 483
14.9 New Hypotheses 484
14.10 Providing Learning Opportunities 486
14.11 Supporting SRE Coaches 487
14.12 Summary 489

Chapter 15: The Road Ahead 491
15.1 Service Catalog 492
15.2 SLAs 494
15.3 Regulatory Compliance 494
15.4 SRE Infrastructure 495
15.5 Game Days 496

Appendix: Topics for Quick Reference 499

Index 507