Solaris Performance and Tools
DTrace and MDB Techniques for Solaris 10 and OpenSolaris (paperback)
Published on 20. April 2015
Book
Paperback/Softback
496 pages
978-0-13-413186-3 (ISBN)
Description
"The Solaris (TM)Internals volumes are simply the best and most comprehensive treatment of the Solaris (and OpenSolaris) Operating Environment. Any person using Solaris--in any capacity--would be remiss not to include these two new volumes in their personal library. With advanced observability tools in Solaris (like DTrace), you will more often find yourself in what was previously unchartable territory. Solaris (TM) Internals, Second Edition, provides us a fantastic means to be able to quickly understand these systems and further explore the Solaris architecture--especially when coupled with OpenSolaris source availability."
--Jarod Jenson, chief systems architect, Aeysis
"The Solaris (TM) Internals volumes by Jim Mauro and Richard McDougall must be on your bookshelf if you are interested in in-depth knowledge of Solaris operating system internals and architecture. As a senior Unix engineer for many years, I found the first edition of Solaris (TM) Internals the only fully comprehensive source for kernel developers, systems programmers, and systems administrators. The new second edition, with the companion performance and debugging book, is an indispensable reference set, containing many useful and practical explanations of Solaris and its underlying subsystems, including tools and methods for observing and analyzing any system running Solaris 10 or OpenSolaris."
--Marc Strahl, senior UNIX engineer
Solaris (TM) Performance and Tools provides comprehensive coverage of the powerful utilities bundled with Solaris 10 and OpenSolaris, including the Solaris Dynamic Tracing facility, DTrace, and the Modular Debugger, MDB. It provides a systematic approach to understanding performance and behavior, including:
Analyzing CPU utilization by the kernel and applications, including reading and understanding hardware counters
Process-level resource usage and profiling
Disk IO behavior and analysis
Memory usage at the system and application level
Network performance
Monitoring and profiling the kernel, and gathering kernel statistics
Using DTrace providers and aggregations
MDB commands and a complete MDB tutorial
The Solaris (TM) Internals volumes make a superb reference for anyone using Solaris 10 and OpenSolaris.
--Jarod Jenson, chief systems architect, Aeysis
"The Solaris (TM) Internals volumes by Jim Mauro and Richard McDougall must be on your bookshelf if you are interested in in-depth knowledge of Solaris operating system internals and architecture. As a senior Unix engineer for many years, I found the first edition of Solaris (TM) Internals the only fully comprehensive source for kernel developers, systems programmers, and systems administrators. The new second edition, with the companion performance and debugging book, is an indispensable reference set, containing many useful and practical explanations of Solaris and its underlying subsystems, including tools and methods for observing and analyzing any system running Solaris 10 or OpenSolaris."
--Marc Strahl, senior UNIX engineer
Solaris (TM) Performance and Tools provides comprehensive coverage of the powerful utilities bundled with Solaris 10 and OpenSolaris, including the Solaris Dynamic Tracing facility, DTrace, and the Modular Debugger, MDB. It provides a systematic approach to understanding performance and behavior, including:
Analyzing CPU utilization by the kernel and applications, including reading and understanding hardware counters
Process-level resource usage and profiling
Disk IO behavior and analysis
Memory usage at the system and application level
Network performance
Monitoring and profiling the kernel, and gathering kernel statistics
Using DTrace providers and aggregations
MDB commands and a complete MDB tutorial
The Solaris (TM) Internals volumes make a superb reference for anyone using Solaris 10 and OpenSolaris.
Reviews / Votes
"In total, the two books Solaris Performance and Tools & Solaris Internals reviewed here present a new level of knowledge about the internals of Solaris, what they do, how they behave, and how to analyze that behavior. The books are a must for developers, system programmers, and systems administrators who work with Solaris 8, 9, or 10. They are especially useful for users of Solaris 10 and OpenSolaris because of their exploration of the new tools in those releases. These books receive my highest recommendation. "-Peter Baer Galvin, Contributing Editor, Sys Admin MagazineMore details
Persons
Richard McDougall is a Distinguished Engineer at Sun Microsystems, specializing in operating systems technology and systems performance.
Jim Mauro is a Senior Staff Engineer in the Performance, Architecture, and Applications Engineering group at Sun Microsystems, where his most recent efforts have focused on Solaris performance on Opteron platforms.
Brendan Gregg is a Solaris consultant and instructor teaching classes for Sun Microsystems across Australia and Asia. He is also an OpenSolaris contributor and community leader, and has written numerous software packages, including the DTraceToolkit.
Jim Mauro is a Senior Staff Engineer in the Performance, Architecture, and Applications Engineering group at Sun Microsystems, where his most recent efforts have focused on Solaris performance on Opteron platforms.
Brendan Gregg is a Solaris consultant and instructor teaching classes for Sun Microsystems across Australia and Asia. He is also an OpenSolaris contributor and community leader, and has written numerous software packages, including the DTraceToolkit.
Content
Foreword xxi Preface xxiii About the Authors xxxi Acknowledgments xxxiii PART ONE: Observability Methods 1 Chapter 1: Introduction to Observability Tools 3 1.1 Observability Tools 4
1.2 Drill-Down Analysis 7
1.3 About Part One 8
Chapter 2: CPUs 11 2.1 Tools for CPU Analysis 11
2.2 vmstat Tool 13
2.3 CPU Utilization 14
2.4 CPU Saturation 15
2.5 psrinfo Command 15
2.6 uptime Command 15
2.7 sar Command 16
2.8 Clock Tick Woes 19
2.9 mpstat Command 20
2.10 Who Is Using the CPU? 23
2.11 CPU Run Queue Latency 24
2.12 CPU Statistics Internals 26
2.13 Using DTrace to Explain Events from Performance Tools 29
2.14 DTrace Versions of runq-sz, %runocc 31
2.15 DTrace Probes for CPU States 33
Chapter 3: Processes 35 3.1 Tools for Process Analysis 35
3.2 Process Statistics Summary: prstat 37
3.3 Process Status: ps 41
3.4 Tools for Listing and Controlling Processes 45
3.5 Process Introspection Commands 47
3.6 Examining User-Level Locks in a Process 52
3.7 Tracing Processes 53
3.8 Java Processes 60
Chapter 4: Disk Behavior and Analysis 67 4.1 Terms for Disk Analysis 67
4.2 Random vs. Sequential I/O 69
4.3 Storage Arrays 70
4.4 Sector Zoning 71
4.5 Max I/O Size 72
4.6 iostat Utility 73
4.7 Disk Utilization 74
4.8 Disk Saturation 75
4.9 Disk Throughput 76
4.10 iostat Reference 76
4.11 Reading iostat 82
4.12 iostat Internals 85
4.13 sar -d 87
4.14 Trace Normal Form (TNF) Tracing for I/O 88
4.15 DTrace for I/O 88
4.16 Disk I/O Time 97
4.17 DTraceToolkit Commands 101
4.18 DTraceTazTool 108
Chapter 5: File Systems 109 5.1 Layers of File System and I/O 109
5.2 Observing Physical I/O 111
5.3 File System Latency 112
5.4 Causes of Read/Write File System Latency 114
5.5 Observing File System "Top End" Activity 118
5.6 File System Caches 119
5.7 NFS Statistics 133
Chapter 6: Memory 135 6.1 Tools for Memory Analysis 135
6.2 vmstat(1M) Command 137
6.3 Types of Paging 138
6.4 Physical Memory Allocation 142
6.5 Relieving Memory Pressure 144
6.6 Scan Rate as a Memory Health Indicator 146
6.7 Process Virtual and Resident Set Size 148
6.8 Using pmap to Inspect Process Memory Usage 149
6.9 Calculating Process Memory Usage with ps and pmap 150
6.10 Displaying Page-Size Information with pmap 153
6.11 Using DTrace for Memory Analysis 154
6.12 Obtaining Memory Kstats 157
6.13 Using the Perl Kstat API to Look at Memory Statistics 158
6.14 System Memory Allocation Kstats 158
6.15 Kernel Memory with kstat 160
6.16 System Paging Kstats 161
6.17 Observing MMU Performance Impact with trapstat 163
6.18 Swap Space 164
Chapter 7: Networks 173 7.1 Terms for Network Analysis 173
7.2 Packets Are Not Bytes 175
7.3 Network Utilization 176
7.4 Network Saturation 177
7.5 Network Errors 177
7.6 Misconfigurations 177
7.7 Systemwide Statistics 178
7.8 Per-Process Network Statistics 189
7.9 TCP Statistics 191
7.10 IP Statistics 196
7.11 ICMP Statistics 199
Chapter 8: Performance Counters 2018.1 Introducing CPU Caches 203
8.2 cpustat Command 206
8.3 cputrack Command 215
8.4 busstat Command 216
Chapter 9: Kernel Monitoring 221 9.1 Tools for Kernel Monitoring 221
9.2 Profiling the Kernel and Drivers 222
9.3 Analyzing Kernel Locks 223
9.4 DTrace lockstat Provider 227
9.5 DTrace Kernel Profiling 229
9.6 Interrupt Statistics: vmstat -i 230
9.7 Interrupt Analysis: intrstat 230
PART TWO: Observability Infrastructure 233 Chapter 10: Dynamic Tracing 235 10.1 Introduction to DTrace 235
10.2 The Basics 236
10.3 Inspecting Java Applications with DTrace 257
10.4 DTrace Architecture 265
10.5 Summary 271
10.6 Probe Reference 271
10.7 MDB Reference 294
Chapter 11: Kernel Statistics 295 11.1 C-Level Kstat Interface 295
11.2 Command-Line Interface 307
11.3 Using Perl to Access kstats
11.4 Snooping a Program's kstat Use with DTrace 317
11.5 Adding Statistics to the Solaris Kernel 317
11.6 Additional Information 323
PART THREE: Debugging 325 Chapter 12: The Modular Debugger 327 12.1 Introduction to the Modular Debugger 327
12.2 MDB Concepts 330
Chapter 13: An MDB Tutorial 335 13.1 Invoking MDB 335
13.2 MDB Command Syntax 336
13.3 Working with Debugging Targets 353
13.4 GDB-to-MDB Reference 357
13.5 dcmd and Walker Reference 359
Chapter 14: Debugging Kernels 367 14.1 Working with Kernel Cores 367
14.2 Examining User Process Stacks within a Kernel Image 382
14.3 Switching MDB to Debug a Specific Process 385
14.4 kmdb, the Kernel Modular Debugger 388
14.5 Kernel Built-In MDB dcmds 395
APPENDICES Appendix A Tunables and Settings 401 Appendix B DTrace One-Liners 407 Appendix C Java DTrace Scripts 409 Appendix D Sample Perl Kstat Utilities 413 Bibliography 429 Index 433
1.2 Drill-Down Analysis 7
1.3 About Part One 8
Chapter 2: CPUs 11 2.1 Tools for CPU Analysis 11
2.2 vmstat Tool 13
2.3 CPU Utilization 14
2.4 CPU Saturation 15
2.5 psrinfo Command 15
2.6 uptime Command 15
2.7 sar Command 16
2.8 Clock Tick Woes 19
2.9 mpstat Command 20
2.10 Who Is Using the CPU? 23
2.11 CPU Run Queue Latency 24
2.12 CPU Statistics Internals 26
2.13 Using DTrace to Explain Events from Performance Tools 29
2.14 DTrace Versions of runq-sz, %runocc 31
2.15 DTrace Probes for CPU States 33
Chapter 3: Processes 35 3.1 Tools for Process Analysis 35
3.2 Process Statistics Summary: prstat 37
3.3 Process Status: ps 41
3.4 Tools for Listing and Controlling Processes 45
3.5 Process Introspection Commands 47
3.6 Examining User-Level Locks in a Process 52
3.7 Tracing Processes 53
3.8 Java Processes 60
Chapter 4: Disk Behavior and Analysis 67 4.1 Terms for Disk Analysis 67
4.2 Random vs. Sequential I/O 69
4.3 Storage Arrays 70
4.4 Sector Zoning 71
4.5 Max I/O Size 72
4.6 iostat Utility 73
4.7 Disk Utilization 74
4.8 Disk Saturation 75
4.9 Disk Throughput 76
4.10 iostat Reference 76
4.11 Reading iostat 82
4.12 iostat Internals 85
4.13 sar -d 87
4.14 Trace Normal Form (TNF) Tracing for I/O 88
4.15 DTrace for I/O 88
4.16 Disk I/O Time 97
4.17 DTraceToolkit Commands 101
4.18 DTraceTazTool 108
Chapter 5: File Systems 109 5.1 Layers of File System and I/O 109
5.2 Observing Physical I/O 111
5.3 File System Latency 112
5.4 Causes of Read/Write File System Latency 114
5.5 Observing File System "Top End" Activity 118
5.6 File System Caches 119
5.7 NFS Statistics 133
Chapter 6: Memory 135 6.1 Tools for Memory Analysis 135
6.2 vmstat(1M) Command 137
6.3 Types of Paging 138
6.4 Physical Memory Allocation 142
6.5 Relieving Memory Pressure 144
6.6 Scan Rate as a Memory Health Indicator 146
6.7 Process Virtual and Resident Set Size 148
6.8 Using pmap to Inspect Process Memory Usage 149
6.9 Calculating Process Memory Usage with ps and pmap 150
6.10 Displaying Page-Size Information with pmap 153
6.11 Using DTrace for Memory Analysis 154
6.12 Obtaining Memory Kstats 157
6.13 Using the Perl Kstat API to Look at Memory Statistics 158
6.14 System Memory Allocation Kstats 158
6.15 Kernel Memory with kstat 160
6.16 System Paging Kstats 161
6.17 Observing MMU Performance Impact with trapstat 163
6.18 Swap Space 164
Chapter 7: Networks 173 7.1 Terms for Network Analysis 173
7.2 Packets Are Not Bytes 175
7.3 Network Utilization 176
7.4 Network Saturation 177
7.5 Network Errors 177
7.6 Misconfigurations 177
7.7 Systemwide Statistics 178
7.8 Per-Process Network Statistics 189
7.9 TCP Statistics 191
7.10 IP Statistics 196
7.11 ICMP Statistics 199
Chapter 8: Performance Counters 2018.1 Introducing CPU Caches 203
8.2 cpustat Command 206
8.3 cputrack Command 215
8.4 busstat Command 216
Chapter 9: Kernel Monitoring 221 9.1 Tools for Kernel Monitoring 221
9.2 Profiling the Kernel and Drivers 222
9.3 Analyzing Kernel Locks 223
9.4 DTrace lockstat Provider 227
9.5 DTrace Kernel Profiling 229
9.6 Interrupt Statistics: vmstat -i 230
9.7 Interrupt Analysis: intrstat 230
PART TWO: Observability Infrastructure 233 Chapter 10: Dynamic Tracing 235 10.1 Introduction to DTrace 235
10.2 The Basics 236
10.3 Inspecting Java Applications with DTrace 257
10.4 DTrace Architecture 265
10.5 Summary 271
10.6 Probe Reference 271
10.7 MDB Reference 294
Chapter 11: Kernel Statistics 295 11.1 C-Level Kstat Interface 295
11.2 Command-Line Interface 307
11.3 Using Perl to Access kstats
11.4 Snooping a Program's kstat Use with DTrace 317
11.5 Adding Statistics to the Solaris Kernel 317
11.6 Additional Information 323
PART THREE: Debugging 325 Chapter 12: The Modular Debugger 327 12.1 Introduction to the Modular Debugger 327
12.2 MDB Concepts 330
Chapter 13: An MDB Tutorial 335 13.1 Invoking MDB 335
13.2 MDB Command Syntax 336
13.3 Working with Debugging Targets 353
13.4 GDB-to-MDB Reference 357
13.5 dcmd and Walker Reference 359
Chapter 14: Debugging Kernels 367 14.1 Working with Kernel Cores 367
14.2 Examining User Process Stacks within a Kernel Image 382
14.3 Switching MDB to Debug a Specific Process 385
14.4 kmdb, the Kernel Modular Debugger 388
14.5 Kernel Built-In MDB dcmds 395
APPENDICES Appendix A Tunables and Settings 401 Appendix B DTrace One-Liners 407 Appendix C Java DTrace Scripts 409 Appendix D Sample Perl Kstat Utilities 413 Bibliography 429 Index 433