Effective Python

Brett Slatkin is a Principal Software Engineer at Google in the Office of the CTO, focusing on emerging technologies. He co-founded Google Surveys, launched Google Cloud’s first product (App Engine), and co-created the PubSubHubbub protocol—all using Python. Brett has been writing Python code professionally for the past 19 years and has made numerous contributions to open-source projects.

Preface     xvii

Acknowledgments     xxiii

About the Author     xxv

 

Chapter 1: Pythonic Thinking     1

     Item 1: Know Which Version of Python You’re Using     1

     Item 2: Follow the PEP 8 Style Guide     3

     Item 3: Never Expect Python to Detect Errors at Compile Time     6

     Item 4: Write Helper Functions Instead of Complex Expressions     8

     Item 5: Prefer Multiple-Assignment Unpacking Over Indexing     11

     Item 6: Always Surround Single-Element Tuples with Parentheses     16

     Item 7: Consider Conditional Expressions for Simple Inline Logic     19

     Item 8: Prevent Repetition with Assignment Expressions     24

     Item 9: Consider match for Destructuring in Flow Control; Avoid When if Statements Are Sufficient     30

 

Chapter 2: Strings and Slicing     41

     Item 10: Know the Differences Between bytes and str     41

     Item 11: Prefer Interpolated F-Strings over C-Style Format Strings and str.format     47

     Item 12: Understand the Difference Between  repr and str when Printing Objects     58

     Item 13: Prefer Explicit String Concatenation over Implicit, Especially in Lists     62

     Item 14: Know How to Slice Sequences     67

     Item 15: Avoid Striding and Slicing in a Single Expression     70

     Item 16: Prefer Catch-All Unpacking Over Slicing     72

 

Chapter 3: Loops and Iterators     77

     Item 17: Prefer enumerate over range     77

     Item 18: Use zip to Process Iterators in Parallel     79

     Item 19: Avoid else Blocks After for and while Loops     82

     Item 20: Never Use for Loop Variables After the Loop Ends     85

     Item 21: Be Defensive when Iterating over Arguments     87

     Item 22: Never Modify Containers While Iterating over Them; Use Copies or Caches Instead     92

     Item 23: Pass Iterators to any and all for Efficient Short-Circuiting Logic     98

     Item 24: Consider itertools for Working with Iterators and Generators     102

 

Chapter 4: Dictionaries     109

     Item 25: Be Cautious when Relying on Dictionary Insertion Ordering     109

     Item 26: Prefer get over in and KeyError to Handle Missing Dictionary Keys     117

     Item 27: Prefer defaultdict over setdefault to Handle Missing Items in Internal State     122

     Item 28: Know How to Construct Key-Dependent Default Values with __missing__     124

     Item 29: Compose Classes Instead of Deeply Nesting Dictionaries, Lists, and Tuples     127

 

Chapter 5: Functions     135

     Item 30: Know That Function Arguments Can Be Mutated     135

     Item 31: Return Dedicated Result Objects Instead of Requiring Function Callers to Unpack More Than Three Variables     138

     Item 32: Prefer Raising Exceptions to Returning None     142

     Item 33: Know How Closures Interact with Variable Scope and nonlocal     145

     Item 34: Reduce Visual Noise with Variable Positional Arguments     150

     Item 35: Provide Optional Behavior with Keyword Arguments     153

     Item 36: Use None and Docstrings to Specify Dynamic Default Arguments     157

     Item 37: Enforce Clarity with Keyword-Only and Positional-Only Arguments     161

     Item 38: Define Function Decorators with functools.wraps     166

     Item 39: Prefer functools.partial over lambda Expressions for Glue Functions     169

 

Chapter 6: Comprehensions and Generators     173

     Item 40: Use Comprehensions Instead of map and filter     173

     Item 41: Avoid More Than Two Control Subexpressions in Comprehensions     176

     Item 42: Reduce Repetition in Comprehensions with Assignment Expressions     178

     Item 43: Consider Generators Instead of Returning Lists     182

     Item 44: Consider Generator Expressions for Large List Comprehensions     184

     Item 45: Compose Multiple Generators with yield from     186

     Item 46: Pass Iterators into Generators as Arguments Instead of Calling the send Method     188

     Item 47: Manage Iterative State Transitions with a Class Instead of the Generator throw Method     195

 

Chapter 7: Classes and Interfaces     201

     Item 48: Accept Functions Instead of Classes for Simple Interfaces     201

     Item 49: Prefer Object-Oriented Polymorphism over Functions with isinstance Checks     205

     Item 50: Consider functools.singledispatch for Functional-Style Programming Instead of Object-Oriented Polymorphism     210

     Item 51: Prefer dataclasses for Defining Lightweight Classes     217

     Item 52: Use @classmethod Polymorphism to Construct Objects Generically     230

     Item 53: Initialize Parent Classes with super     235

     Item 54: Consider Composing Functionality with Mix-in Classes     240

     Item 55: Prefer Public Attributes over Private Ones     245

     Item 56: Prefer dataclasses for Creating Immutable Objects     250

     Item 57: Inherit from collections.abc Classes for Custom Container Types     260

 

Chapter 8: Metaclasses and Attributes     265

     Item 58: Use Plain Attributes Instead of Setter and Getter Methods     265

     Item 59: Consider @property Instead of Refactoring Attributes     270

     Item 60: Use Descriptors for Reusable @property Methods     274

     Item 61: Use __getattr__, __getattribute__, and __setattr__ for Lazy Attributes     279

     Item 62: Validate Subclasses with __init_subclass__     285

     Item 63: Register Class Existence with __init_subclass__     293

     Item 64: Annotate Class Attributes with __set_name__     299

     Item 65: Consider Class Body Definition Order to Establish Relationships Between Attributes     303

     Item 66: Prefer Class Decorators over Metaclasses for Composable Class Extensions     310

 

Chapter 9: Concurrency and Parallelism     319

     Item 67: Use subprocess to Manage Child Processes     320

     Item 68: Use Threads for Blocking I/O; Avoid for Parallelism     324

     Item 69: Use Lock to Prevent Data Races in Threads     330

     Item 70: Use Queue to Coordinate Work Between Threads     333

     Item 71: Know How to Recognize When Concurrency Is Necessary     344

     Item 72: Avoid Creating New Thread Instances for On-Demand Fan-out     349

     Item 73: Understand How Using Queue for Concurrency Requires Refactoring     353

     Item 74: Consider ThreadPoolExecutor When Threads Are Necessary for Concurrency     361

     Item 75: Achieve Highly Concurrent I/O with Coroutines     364

     Item 76: Know How to Port Threaded I/O to asyncio     368

     Item 77: Mix Threads and Coroutines to Ease the Transition to asyncio     381

     Item 78: Maximize Responsiveness of asyncio Event Loops with async-friendly Worker Threads     389

     Item 79: Consider concurrent.futures for True Parallelism     393

 

Chapter 10: Robustness     399

     Item 80: Take Advantage of Each Block in try/except/else/finally     399

     Item 81: assert Internal Assumptions and raise Missed Expectations     404

     Item 82: Consider contextlib and with Statements for Reusable try/finally Behavior     408

     Item 83: Always Make try Blocks as Short as Possible     412

     Item 84: Beware of Exception Variables Disappearing     414

     Item 85: Beware of Catching the Exception Class     416

     Item 86: Understand the Difference Between Exception and BaseException     419

     Item 87: Use traceback for Enhanced Exception Reporting     424

     Item 88: Consider Explicitly Chaining Exceptions to Clarify Tracebacks     428

     Item 89: Always Pass Resources into Generators and Have Callers Clean Them Up Outside     436

     Item 90: Never Set __debug__ to False     442

     Item 91: Avoid exec and eval Unless You’re Building a Developer Tool     445

 

Chapter 11: Performance     447

     Item 92: Profile Before Optimizing     448

     Item 93: Optimize Performance-Critical Code Using timeit Microbenchmarks     453

     Item 94: Know When and How to Replace Python with Another Programming Language     458

     Item 95: Consider ctypes to Rapidly Integrate with Native Libraries     462

     Item 96: Consider Extension Modules to Maximize Performance and Ergonomics     467

     Item 97: Rely on Precompiled Bytecode and File System Caching to Improve Startup Time     475

     Item 98: Lazy-Load Modules with Dynamic Imports to Reduce Startup Time     478

     Item 99: Consider memoryview and bytearray for Zero-Copy Interactions with bytes     485

 

Chapter 12: Data Structures & Algorithms     493

     Item 100: Sort by Complex Criteria Using the key Parameter     493

     Item 101: Know the Difference Between sort and sorted     499

     Item 102: Consider Searching Sorted Sequences with bisect     501

     Item 103: Prefer deque for Producer-Consumer Queues     504

     Item 104: Know How to Use heapq for Priority Queues     509

     Item 105: Use datetime Instead of time for Local Clocks     519

     Item 106: Use decimal When Precision Is Paramount     523

     Item 107: Make pickle Serialization Maintainable with copyreg     526

 

Chapter 13: Testing and Debugging     533

     Item 108: Verify Related Behaviors in TestCase Subclasses     533

     Item 109: Prefer Integration Tests over Unit Tests     541

     Item 110: Isolate Tests From Each Other with setUp, tearDown, setUpModule, and tearDownModule     547

     Item 111: Use Mocks to Test Code with Complex Dependencies     550

     Item 112: Encapsulate Dependencies to Facilitate Mocking and Testing     559

     Item 113: Use assertAlmostEqual to Control Precision in Floating Point Tests     563

     Item 114: Consider Interactive Debugging with pdb     565

     Item 115: Use tracemalloc to Understand Memory Usage and Leaks     570

 

Chapter 14: Collaboration     575

     Item 116: Know Where to Find Community-Built Modules     575

     Item 117: Use Virtual Environments for Isolated and Reproducible Dependencies     576

     Item 118: Write Docstrings for Every Function, Class, and Module     582

     Item 119: Use Packages to Organize Modules and Provide Stable APIs     588

     Item 120: Consider Module-Scoped Code to Configure Deployment Environments     593

     Item 121: Define a Root Exception to Insulate Callers from APIs     595

     Item 122: Know How to Break Circular Dependencies     600

     Item 123: Consider warnings to Refactor and Migrate Usage     605

     Item 124: Consider Static Analysis via typing to Obviate Bugs     613

     Item 125: Prefer Open Source Projects for Bundling Python Programs over zipimport and zipapp     621

 

Index     627