Systems Performance
Enterprise and the Cloud
Published on 7. November 2013
Book
Paperback/Softback
784 pages
978-0-13-339009-4 (ISBN)
Description
The Complete Guide to Optimizing Systems Performance
Written by the winner of the 2013 LISA Award for Outstanding Achievement in System Administration
Large-scale enterprise, cloud, and virtualized computing systems have introduced serious performance challenges. Now, internationally renowned performance expert Brendan Gregg has brought together proven methodologies, tools, and metrics for analyzing and tuning even the most complex environments. Systems Performance: Enterprise and the Cloud focuses on Linux (R) and Unix (R) performance, while illuminating performance issues that are relevant to all operating systems. You'll gain deep insight into how systems work and perform, and learn methodologies for analyzing and improving system and application performance. Gregg presents examples from bare-metal systems and virtualized cloud tenants running Linux-based Ubuntu (R), Fedora (R), CentOS, and the illumos-based Joyent (R) SmartOS (TM) and OmniTI OmniOS (R). He systematically covers modern systems performance, including the "traditional" analysis of CPUs, memory, disks, and networks, and new areas including cloud computing and dynamic tracing. This book also helps you identify and fix the "unknown unknowns" of complex performance: bottlenecks that emerge from elements and interactions you were not aware of. The text concludes with a detailed case study, showing how a real cloud customer issue was analyzed from start to finish.
Coverage includes
* Modern performance analysis and tuning: terminology, concepts, models, methods, and techniques
* Dynamic tracing techniques and tools, including examples of DTrace, SystemTap, and perf
* Kernel internals: uncovering what the OS is doing
* Using system observability tools, interfaces, and frameworks
* Understanding and monitoring application performance
* Optimizing CPUs: processors, cores, hardware threads, caches, interconnects, and kernel scheduling
* Memory optimization: virtual memory, paging, swapping, memory architectures, busses, address spaces, and allocators
* File system I/O, including caching
* Storage devices/controllers, disk I/O workloads, RAID, and kernel I/O
* Network-related performance issues: protocols, sockets, interfaces, and physical connections
* Performance implications of OS and hardware-based virtualization, and new issues encountered with cloud computing
* Benchmarking: getting accurate results and avoiding common mistakes
This guide is indispensable for anyone who operates enterprise or cloud environments: system, network, database, and web admins; developers; and other professionals. For students and others new to optimization, it also provides exercises reflecting Gregg's extensive instructional experience.
Written by the winner of the 2013 LISA Award for Outstanding Achievement in System Administration
Large-scale enterprise, cloud, and virtualized computing systems have introduced serious performance challenges. Now, internationally renowned performance expert Brendan Gregg has brought together proven methodologies, tools, and metrics for analyzing and tuning even the most complex environments. Systems Performance: Enterprise and the Cloud focuses on Linux (R) and Unix (R) performance, while illuminating performance issues that are relevant to all operating systems. You'll gain deep insight into how systems work and perform, and learn methodologies for analyzing and improving system and application performance. Gregg presents examples from bare-metal systems and virtualized cloud tenants running Linux-based Ubuntu (R), Fedora (R), CentOS, and the illumos-based Joyent (R) SmartOS (TM) and OmniTI OmniOS (R). He systematically covers modern systems performance, including the "traditional" analysis of CPUs, memory, disks, and networks, and new areas including cloud computing and dynamic tracing. This book also helps you identify and fix the "unknown unknowns" of complex performance: bottlenecks that emerge from elements and interactions you were not aware of. The text concludes with a detailed case study, showing how a real cloud customer issue was analyzed from start to finish.
Coverage includes
* Modern performance analysis and tuning: terminology, concepts, models, methods, and techniques
* Dynamic tracing techniques and tools, including examples of DTrace, SystemTap, and perf
* Kernel internals: uncovering what the OS is doing
* Using system observability tools, interfaces, and frameworks
* Understanding and monitoring application performance
* Optimizing CPUs: processors, cores, hardware threads, caches, interconnects, and kernel scheduling
* Memory optimization: virtual memory, paging, swapping, memory architectures, busses, address spaces, and allocators
* File system I/O, including caching
* Storage devices/controllers, disk I/O workloads, RAID, and kernel I/O
* Network-related performance issues: protocols, sockets, interfaces, and physical connections
* Performance implications of OS and hardware-based virtualization, and new issues encountered with cloud computing
* Benchmarking: getting accurate results and avoiding common mistakes
This guide is indispensable for anyone who operates enterprise or cloud environments: system, network, database, and web admins; developers; and other professionals. For students and others new to optimization, it also provides exercises reflecting Gregg's extensive instructional experience.
More details
Other editions
New editions
Person
Brendan Gregg, lead performance engineer at Joyent, analyzes performance and scalability throughout the software stack. As performance lead and kernel engineer at Sun Microsystems (and later Oracle), his work included developing the ZFS L2ARC, a pioneering file system technology for improving performance using flash memory. He has invented and developed many performance tools, including some that ship with Mac OS X and Oracle (R) Solaris (TM) 11. His recent work has included performance visualizations for Linux and illumos kernel analysis. For contributions to system administration, and his work on performance analysis methodologies, he is the recipient of the USENIX 2013 LISA Award for Outstanding Achievement in System Administration. He is also a coauthor of Dtrace: Dynamic Tracing in Oracle Solaris, Mac OS X and FreeBSD (Prentice Hall, 2011), and Solaris (TM) Performance and Tools: DTrace and MDB Techniques for Solaris 10 and OpenSolaris, (Prentice Hall, 2007).
Content
Acknowledgments xxxiii
About the Author xxxv
Chapter 1: Introduction 1
1.1 Systems Performance 1
1.2 Roles 2
1.3 Activities 3
1.4 Perspectives 4
1.5 Performance Is Challenging 4
1.6 Latency 6
1.7 Dynamic Tracing 7
1.8 Cloud Computing 8
1.9 Case Studies 9
Chapter 2: Methodology 15
2.1 Terminology 16
2.2 Models 17
2.3 Concepts 18
2.4 Perspectives 32
2.5 Methodology 35
2.6 Modeling 57
2.7 Capacity Planning 65
2.8 Statistics 69
2.9 Monitoring 74
2.10 Visualizations 76
2.11 Exercises 82
2.12 References 82
Chapter 3: Operating Systems 85
3.1 Terminology 86
3.2 Background 87
3.3 Kernels 105
3.4 Exercises 113
3.5 References 113
Chapter 4: Observability Tools 115
4.1 Tool Types 116
4.2 Observability Sources 120
4.3 DTrace 133
4.4 SystemTap 144
4.5 perf 149
4.6 Observing Observability 150
4.7 Exercises 151
4.8 References 151
Chapter 5: Applications 153
5.1 Application Basics 153
5.2 Application Performance Techniques 158
5.3 Programming Languages 163
5.4 Methodology and Analysis 167
5.5 Exercises 186
5.6 References 187
Chapter 6: CPUs 189
6.1 Terminology 190
6.2 Models 191
6.3 Concepts 193
6.4 Architecture 199
6.5 Methodology 214
6.6 Analysis 224
6.7 Experimentation 254
6.8 Tuning 256
6.9 Exercises 260
6.10 References 262
Chapter 7: Memory 265
7.1 Terminology 266
7.2 Concepts 267
7.3 Architecture 272
7.4 Methodology 289
7.5 Analysis 295
7.6 Tuning 314
7.7 Exercises 319
7.8 References 320
Chapter 8: File Systems 323
8.1 Terminology 324
8.2 Models 325
8.3 Concepts 326
8.4 Architecture 337
8.5 Methodology 353
8.6 Analysis 362
8.7 Experimentation 383
8.8 Tuning 387
8.9 Exercises 391
8.10 References 392
Chapter 9: Disks 395
9.1 Terminology 396
9.2 Models 397
9.3 Concepts 399
9.4 Architecture 407
9.5 Methodology 421
9.6 Analysis 431
9.7 Experimentation 465
9.8 Tuning 467
9.9 Exercises 470
9.10 References 471
Chapter 10: Network 473
10.1 Terminology 474
10.2 Models 474
10.3 Concepts 476
10.4 Architecture 483
10.5 Methodology 493
10.6 Analysis 503
10.7 Experimentation 535
10.8 Tuning 536
10.9 Exercises 542
10.10 References 543
Chapter 11: Cloud Computing 545
11.1 Background 546
11.2 OS Virtualization 551
11.3 Hardware Virtualization 563
11.4 Comparisons 581
11.5 Exercises 583
11.6 References 584
Chapter 12: Benchmarking 587
12.1 Background 588
12.2 Benchmarking Types 597
12.3 Methodology 602
12.4 Benchmark Questions 613
12.5 Exercises 614
12.6 References 615
Chapter 13: Case Study 617
13.1 Case Study: The Red Whale 617
13.2 Comments 633
13.3 Additional Information 634
13.4 References 634
Appendix A: USE Method: Linux 637
Physical Resources 637
Software Resources 640
Reference 641
Appendix B: USE Method: Solaris 643
Physical Resources 643
Software Resources 646
References 647
Appendix C: sar Summary 649
Linux 649
Solaris 650
Appendix D: DTrace One-Liners 651
syscall Provider 651
proc Provider 655
profile Provider 655
sched Provider 657
fbt Provider 658
pid Provider 659
io Provider 660
sysinfo Provider 660
vminfo Provider 661
ip Provider 661
tcp provider 662
udp provider 663
Appendix E: DTrace to SystemTap 665
Functionality 665
Terminology 666
Probes 666
Built-in Variables 667
Functions 668
Example 1: Listing syscall Entry Probes 668
Example 2: Summarize read() Returned Size 668
Example 3: Count syscalls by Process Name 670
Example 4: Count syscalls by syscall Name, for Process ID 123 671
Example 5: Count syscalls by syscall Name, for "httpd" Processes 672
Example 6: Trace File open()s with Process Name and Path Name 672
Example 7: Summarize read() Latency for "mysqld" Processes 672
Example 8: Trace New Processes with Process Name and Arguments 673
Example 9: Sample Kernel Stacks at 100 Hz 674
References 674
Appendix F: Solutions to Selected Exercises 675
Chapter 2 Methodology 675
Chapter 3 Operating Systems 675
Chapter 6 CPUs 675
Chapter 7 Memory 676
Chapter 8 File Systems 676
Chapter 9 Disks 677
Chapter 11 Cloud Computing 677
Appendix G: Systems Performance Who's Who 679
Glossary 683
Bibliography 689
Index 697
About the Author xxxv
Chapter 1: Introduction 1
1.1 Systems Performance 1
1.2 Roles 2
1.3 Activities 3
1.4 Perspectives 4
1.5 Performance Is Challenging 4
1.6 Latency 6
1.7 Dynamic Tracing 7
1.8 Cloud Computing 8
1.9 Case Studies 9
Chapter 2: Methodology 15
2.1 Terminology 16
2.2 Models 17
2.3 Concepts 18
2.4 Perspectives 32
2.5 Methodology 35
2.6 Modeling 57
2.7 Capacity Planning 65
2.8 Statistics 69
2.9 Monitoring 74
2.10 Visualizations 76
2.11 Exercises 82
2.12 References 82
Chapter 3: Operating Systems 85
3.1 Terminology 86
3.2 Background 87
3.3 Kernels 105
3.4 Exercises 113
3.5 References 113
Chapter 4: Observability Tools 115
4.1 Tool Types 116
4.2 Observability Sources 120
4.3 DTrace 133
4.4 SystemTap 144
4.5 perf 149
4.6 Observing Observability 150
4.7 Exercises 151
4.8 References 151
Chapter 5: Applications 153
5.1 Application Basics 153
5.2 Application Performance Techniques 158
5.3 Programming Languages 163
5.4 Methodology and Analysis 167
5.5 Exercises 186
5.6 References 187
Chapter 6: CPUs 189
6.1 Terminology 190
6.2 Models 191
6.3 Concepts 193
6.4 Architecture 199
6.5 Methodology 214
6.6 Analysis 224
6.7 Experimentation 254
6.8 Tuning 256
6.9 Exercises 260
6.10 References 262
Chapter 7: Memory 265
7.1 Terminology 266
7.2 Concepts 267
7.3 Architecture 272
7.4 Methodology 289
7.5 Analysis 295
7.6 Tuning 314
7.7 Exercises 319
7.8 References 320
Chapter 8: File Systems 323
8.1 Terminology 324
8.2 Models 325
8.3 Concepts 326
8.4 Architecture 337
8.5 Methodology 353
8.6 Analysis 362
8.7 Experimentation 383
8.8 Tuning 387
8.9 Exercises 391
8.10 References 392
Chapter 9: Disks 395
9.1 Terminology 396
9.2 Models 397
9.3 Concepts 399
9.4 Architecture 407
9.5 Methodology 421
9.6 Analysis 431
9.7 Experimentation 465
9.8 Tuning 467
9.9 Exercises 470
9.10 References 471
Chapter 10: Network 473
10.1 Terminology 474
10.2 Models 474
10.3 Concepts 476
10.4 Architecture 483
10.5 Methodology 493
10.6 Analysis 503
10.7 Experimentation 535
10.8 Tuning 536
10.9 Exercises 542
10.10 References 543
Chapter 11: Cloud Computing 545
11.1 Background 546
11.2 OS Virtualization 551
11.3 Hardware Virtualization 563
11.4 Comparisons 581
11.5 Exercises 583
11.6 References 584
Chapter 12: Benchmarking 587
12.1 Background 588
12.2 Benchmarking Types 597
12.3 Methodology 602
12.4 Benchmark Questions 613
12.5 Exercises 614
12.6 References 615
Chapter 13: Case Study 617
13.1 Case Study: The Red Whale 617
13.2 Comments 633
13.3 Additional Information 634
13.4 References 634
Appendix A: USE Method: Linux 637
Physical Resources 637
Software Resources 640
Reference 641
Appendix B: USE Method: Solaris 643
Physical Resources 643
Software Resources 646
References 647
Appendix C: sar Summary 649
Linux 649
Solaris 650
Appendix D: DTrace One-Liners 651
syscall Provider 651
proc Provider 655
profile Provider 655
sched Provider 657
fbt Provider 658
pid Provider 659
io Provider 660
sysinfo Provider 660
vminfo Provider 661
ip Provider 661
tcp provider 662
udp provider 663
Appendix E: DTrace to SystemTap 665
Functionality 665
Terminology 666
Probes 666
Built-in Variables 667
Functions 668
Example 1: Listing syscall Entry Probes 668
Example 2: Summarize read() Returned Size 668
Example 3: Count syscalls by Process Name 670
Example 4: Count syscalls by syscall Name, for Process ID 123 671
Example 5: Count syscalls by syscall Name, for "httpd" Processes 672
Example 6: Trace File open()s with Process Name and Path Name 672
Example 7: Summarize read() Latency for "mysqld" Processes 672
Example 8: Trace New Processes with Process Name and Arguments 673
Example 9: Sample Kernel Stacks at 100 Hz 674
References 674
Appendix F: Solutions to Selected Exercises 675
Chapter 2 Methodology 675
Chapter 3 Operating Systems 675
Chapter 6 CPUs 675
Chapter 7 Memory 676
Chapter 8 File Systems 676
Chapter 9 Disks 677
Chapter 11 Cloud Computing 677
Appendix G: Systems Performance Who's Who 679
Glossary 683
Bibliography 689
Index 697