Effective Python

Preface xvii

Acknowledgments xxiii

About the Author xxv



Chapter 1: Pythonic Thinking 1

Item 1: Know Which Version of Python You're Using 1

Item 2: Follow the PEP 8 Style Guide 3

Item 3: Never Expect Python to Detect Errors at Compile Time 6

Item 4: Write Helper Functions Instead of Complex Expressions 8

Item 5: Prefer Multiple-Assignment Unpacking Over Indexing 11

Item 6: Always Surround Single-Element Tuples with Parentheses 16

Item 7: Consider Conditional Expressions for Simple Inline Logic 19

Item 8: Prevent Repetition with Assignment Expressions 24

Item 9: Consider match for Destructuring in Flow Control; Avoid When if Statements Are Sufficient 30



Chapter 2: Strings and Slicing 41

Item 10: Know the Differences Between bytes and str 41

Item 11: Prefer Interpolated F-Strings over C-Style Format Strings and str.format 47

Item 12: Understand the Difference Between repr and str when Printing Objects 58

Item 13: Prefer Explicit String Concatenation over Implicit, Especially in Lists 62

Item 14: Know How to Slice Sequences 67

Item 15: Avoid Striding and Slicing in a Single Expression 70

Item 16: Prefer Catch-All Unpacking Over Slicing 72



Chapter 3: Loops and Iterators 77

Item 17: Prefer enumerate over range 77

Item 18: Use zip to Process Iterators in Parallel 79

Item 19: Avoid else Blocks After for and while Loops 82

Item 20: Never Use for Loop Variables After the Loop Ends 85

Item 21: Be Defensive when Iterating over Arguments 87

Item 22: Never Modify Containers While Iterating over Them; Use Copies or Caches Instead 92

Item 23: Pass Iterators to any and all for Efficient Short-Circuiting Logic 98

Item 24: Consider itertools for Working with Iterators and Generators 102



Chapter 4: Dictionaries 109

Item 25: Be Cautious when Relying on Dictionary Insertion Ordering 109

Item 26: Prefer get over in and KeyError to Handle Missing Dictionary Keys 117

Item 27: Prefer defaultdict over setdefault to Handle Missing Items in Internal State 122

Item 28: Know How to Construct Key-Dependent Default Values with __missing__ 124

Item 29: Compose Classes Instead of Deeply Nesting Dictionaries, Lists, and Tuples 127



Chapter 5: Functions 135

Item 30: Know That Function Arguments Can Be Mutated 135

Item 31: Return Dedicated Result Objects Instead of Requiring Function Callers to Unpack More Than Three Variables 138

Item 32: Prefer Raising Exceptions to Returning None 142

Item 33: Know How Closures Interact with Variable Scope and nonlocal 145

Item 34: Reduce Visual Noise with Variable Positional Arguments 150

Item 35: Provide Optional Behavior with Keyword Arguments 153

Item 36: Use None and Docstrings to Specify Dynamic Default Arguments 157

Item 37: Enforce Clarity with Keyword-Only and Positional-Only Arguments 161

Item 38: Define Function Decorators with functools.wraps 166

Item 39: Prefer functools.partial over lambda Expressions for Glue Functions 169



Chapter 6: Comprehensions and Generators 173

Item 40: Use Comprehensions Instead of map and filter 173

Item 41: Avoid More Than Two Control Subexpressions in Comprehensions 176

Item 42: Reduce Repetition in Comprehensions with Assignment Expressions 178

Item 43: Consider Generators Instead of Returning Lists 182

Item 44: Consider Generator Expressions for Large List Comprehensions 184

Item 45: Compose Multiple Generators with yield from 186

Item 46: Pass Iterators into Generators as Arguments Instead of Calling the send Method 188

Item 47: Manage Iterative State Transitions with a Class Instead of the Generator throw Method 195



Chapter 7: Classes and Interfaces 201

Item 48: Accept Functions Instead of Classes for Simple Interfaces 201

Item 49: Prefer Object-Oriented Polymorphism over Functions with isinstance Checks 205

Item 50: Consider functools.singledispatch for Functional-Style Programming Instead of Object-Oriented Polymorphism 210

Item 51: Prefer dataclasses for Defining Lightweight Classes 217

Item 52: Use @classmethod Polymorphism to Construct Objects Generically 230

Item 53: Initialize Parent Classes with super 235

Item 54: Consider Composing Functionality with Mix-in Classes 240

Item 55: Prefer Public Attributes over Private Ones 245

Item 56: Prefer dataclasses for Creating Immutable Objects 250

Item 57: Inherit from collections.abc Classes for Custom Container Types 260



Chapter 8: Metaclasses and Attributes 265

Item 58: Use Plain Attributes Instead of Setter and Getter Methods 265

Item 59: Consider @property Instead of Refactoring Attributes 270

Item 60: Use Descriptors for Reusable @property Methods 274

Item 61: Use __getattr__, __getattribute__, and __setattr__ for Lazy Attributes 279

Item 62: Validate Subclasses with __init_subclass__ 285

Item 63: Register Class Existence with __init_subclass__ 293

Item 64: Annotate Class Attributes with __set_name__ 299

Item 65: Consider Class Body Definition Order to Establish Relationships Between Attributes 303

Item 66: Prefer Class Decorators over Metaclasses for Composable Class Extensions 310



Chapter 9: Concurrency and Parallelism 319

Item 67: Use subprocess to Manage Child Processes 320

Item 68: Use Threads for Blocking I/O; Avoid for Parallelism 324

Item 69: Use Lock to Prevent Data Races in Threads 330

Item 70: Use Queue to Coordinate Work Between Threads 333

Item 71: Know How to Recognize When Concurrency Is Necessary 344

Item 72: Avoid Creating New Thread Instances for On-Demand Fan-out 349

Item 73: Understand How Using Queue for Concurrency Requires Refactoring 353

Item 74: Consider ThreadPoolExecutor When Threads Are Necessary for Concurrency 361

Item 75: Achieve Highly Concurrent I/O with Coroutines 364

Item 76: Know How to Port Threaded I/O to asyncio 368

Item 77: Mix Threads and Coroutines to Ease the Transition to asyncio 381

Item 78: Maximize Responsiveness of asyncio Event Loops with async-friendly Worker Threads 389

Item 79: Consider concurrent.futures for True Parallelism 393



Chapter 10: Robustness 399

Item 80: Take Advantage of Each Block in try/except/else/finally 399

Item 81: assert Internal Assumptions and raise Missed Expectations 404

Item 82: Consider contextlib and with Statements for Reusable try/finally Behavior 408

Item 83: Always Make try Blocks as Short as Possible 412

Item 84: Beware of Exception Variables Disappearing 414

Item 85: Beware of Catching the Exception Class 416

Item 86: Understand the Difference Between Exception and BaseException 419

Item 87: Use traceback for Enhanced Exception Reporting 424

Item 88: Consider Explicitly Chaining Exceptions to Clarify Tracebacks 428

Item 89: Always Pass Resources into Generators and Have Callers Clean Them Up Outside 436

Item 90: Never Set __debug__ to False 442

Item 91: Avoid exec and eval Unless You're Building a Developer Tool 445



Chapter 11: Performance 447

Item 92: Profile Before Optimizing 448

Item 93: Optimize Performance-Critical Code Using timeit Microbenchmarks 453

Item 94: Know When and How to Replace Python with Another Programming Language 458

Item 95: Consider ctypes to Rapidly Integrate with Native Libraries 462

Item 96: Consider Extension Modules to Maximize Performance and Ergonomics 467

Item 97: Rely on Precompiled Bytecode and File System Caching to Improve Startup Time 475

Item 98: Lazy-Load Modules with Dynamic Imports to Reduce Startup Time 478

Item 99: Consider memoryview and bytearray for Zero-Copy Interactions with bytes 485



Chapter 12: Data Structures & Algorithms 493

Item 100: Sort by Complex Criteria Using the key Parameter 493

Item 101: Know the Difference Between sort and sorted 499

Item 102: Consider Searching Sorted Sequences with bisect 501

Item 103: Prefer deque for Producer-Consumer Queues 504

Item 104: Know How to Use heapq for Priority Queues 509

Item 105: Use datetime Instead of time for Local Clocks 519

Item 106: Use decimal When Precision Is Paramount 523

Item 107: Make pickle Serialization Maintainable with copyreg 526



Chapter 13: Testing and Debugging 533

Item 108: Verify Related Behaviors in TestCase Subclasses 533

Item 109: Prefer Integration Tests over Unit Tests 541

Item 110: Isolate Tests From Each Other with setUp, tearDown, setUpModule, and tearDownModule 547

Item 111: Use Mocks to Test Code with Complex Dependencies 550

Item 112: Encapsulate Dependencies to Facilitate Mocking and Testing 559

Item 113: Use assertAlmostEqual to Control Precision in Floating Point Tests 563

Item 114: Consider Interactive Debugging with pdb 565

Item 115: Use tracemalloc to Understand Memory Usage and Leaks 570



Chapter 14: Collaboration 575

Item 116: Know Where to Find Community-Built Modules 575

Item 117: Use Virtual Environments for Isolated and Reproducible Dependencies 576

Item 118: Write Docstrings for Every Function, Class, and Module 582

Item 119: Use Packages to Organize Modules and Provide Stable APIs 588

Item 120: Consider Module-Scoped Code to Configure Deployment Environments 593

Item 121: Define a Root Exception to Insulate Callers from APIs 595

Item 122: Know How to Break Circular Dependencies 600

Item 123: Consider warnings to Refactor and Migrate Usage 605

Item 124: Consider Static Analysis via typing to Obviate Bugs 613

Item 125: Prefer Open Source Projects for Bundling Python Programs over zipimport and zipapp 621



Index 627