Build Better Write Less. Code More. Python 2 And 3 Covers Serious .

Documenting Your API Changes . . . . . . . . . . . . . . . . . . Marking Deprecated Functions with the warnings Module . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christophe de Vienne on Developing APIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Handling Timestamps and Time Zones The Problem of Missing Time Zones . . . . . . . . . . Building Default datetime Objects . . . . . . . . . . . Time Zone–Aware Timestamps with dateutil . . . . Serializing Time Zone–Aware datetime Objects . Solving Ambiguous Times . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Distributing Your Software A Bit of setup.py History . . . . . . . . . . . . Packaging with setup.cfg . . . . . . . . . . . . The Wheel Format Distribution Standard . Sharing Your Work with the World . . . . . Entry Points . . . . . . . . . . . . . . . . . . . . . Visualizing Entry Points . . . . . . Using Console Scripts . . . . . . . Using Plugins and Drivers . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . Nick Coghlan on Packaging . . . . . . . . . . . . . . . . . . . 50 50 52 54 55 56 57 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Unit Testing The Basics of Testing . . . . . . . . . . . . . . . . . . . . . . . . . Some Simple Tests . . . . . . . . . . . . . . . . . . . Skipping Tests . . . . . . . . . . . . . . . . . . . . . . Running Particular Tests . . . . . . . . . . . . . . . Running Tests in Parallel . . . . . . . . . . . . . . . Creating Objects Used in Tests with Fixtures . Running Test Scenarios . . . . . . . . . . . . . . . . Controlled Tests Using Mocking . . . . . . . . . . Revealing Untested Code with coverage . . . . Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . . Setting Up a Virtual Environment . . . . . . . . . Using virtualenv with tox . . . . . . . . . . . . . . . Re-creating an Environment . . . . . . . . . . . . . Using Different Python Versions . . . . . . . . . . Integrating Other Tests . . . . . . . . . . . . . . . . Testing Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Robert Collins on Testing . . . . . . . . . . . . . . . . . . . . . . xii   Contents in Detail 41 43 45 45 58 60 61 64 67 68 69 71 73 74 75 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 76 78 79 81 81 83 84 88 90 91 92 94 95 95 96 97

7 Methods and Decorators 99 Decorators and When to Use Them . . . . . . . . . . . . . . . Creating Decorators . . . . . . . . . . . . . . . . . . Writing Decorators . . . . . . . . . . . . . . . . . . Stacking Decorators . . . . . . . . . . . . . . . . . . Writing Class Decorators . . . . . . . . . . . . . . How Methods Work in Python . . . . . . . . . . . . . . . . . . Static Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Class Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Abstract Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . Mixing Static, Class, and Abstract Methods . . . . . . . . . Putting Implementations in Abstract Methods . The Truth About super . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Creating Pure Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Creating a Generator . . . . . . . . . . . . . . . . . . . . . . . . . . Returning and Passing Values with yield . . . . . . . . . . . . . Inspecting Generators . . . . . . . . . . . . . . . . . . . . . . . . . . List Comprehensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Functional Functions Functioning . . . . . . . . . . . . . . . . . . . . . . . . . . Applying Functions to Items with map() . . . . . . . . . . . . . Filtering Lists with filter() . . . . . . . . . . . . . . . . . . . . . . . . Getting Indexes with enumerate() . . . . . . . . . . . . . . . . . . Sorting a List with sorted() . . . . . . . . . . . . . . . . . . . . . . . Finding Items That Satisfy Conditions with any() and all() . Combining Lists with zip() . . . . . . . . . . . . . . . . . . . . . . . A Common Problem Solved . . . . . . . . . . . . . . . . . . . . . . Useful itertools Functions . . . . . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Functional Programming 119 9 The Abstract Syntax Tree, Hy, and Lisp-like Attributes Looking at the AST . . . . . . . . . . . . . . . . . . . . Writing a Program Using the AST . . The AST Objects . . . . . . . . . . . . . . Walking Through an AST . . . . . . . . Extending flake8 with AST Checks . . . . . . . . . Writing the Class . . . . . . . . . . . . . . Ignoring Irrelevant Code . . . . . . . . . Checking for the Correct Decorator . Looking for self . . . . . . . . . . . . . . . A Quick Introduction to Hy . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . Paul Tagliamonte on the AST and Hy . . . . . . . . 100 100 101 102 103 107 108 109 110 112 114 114 117 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 121 121 123 124 125 126 127 127 127 128 128 129 129 132 134 135 136 137 138 139 140 141 141 142 143 145 147 147 Contents in Detail   xiii

10 Performances and Optimizations Data Structures . . . . . . . . . . . . . . . . . . . . . . Understanding Behavior Through Profiling . . . cProfile . . . . . . . . . . . . . . . . . . . . Disassembling with the dis Module . Defining Functions Efficiently . . . . . . . . . . . . Ordered Lists and bisect . . . . . . . . . . . . . . . namedtuple and Slots . . . . . . . . . . . . . . . . . Memoization . . . . . . . . . . . . . . . . . . . . . . . Faster Python with PyPy . . . . . . . . . . . . . . . . Achieving Zero Copy with the Buffer Protocol . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . Victor Stinner on Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Scaling and Architecture Multithreading in Python and Its Limitations . Multiprocessing vs. Multithreading . . . . . . . Event-Driven Architecture . . . . . . . . . . . . . . Other Options and asyncio . . . . . . . . . . . . Service-Oriented Architecture . . . . . . . . . . . Interprocess Communication with ZeroMQ . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . 177 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Managing Relational Databases RDBMSs, ORMs, and When to Use Them . . . . . . . . Database Backends . . . . . . . . . . . . . . . . . . . . . . . Streaming Data with Flask and PostgreSQL . . . . . . Writing the Data-Streaming Application . Building the Application . . . . . . . . . . . . Dimitri Fontaine on Databases . . . . . . . . . . . . . . . Using six for Python 2 and 3 Support . . . . . . . . . . Strings and Unicode . . . . . . . . . . . . . . . Handling Python Modules Moves . . . . . . The modernize Module . . . . . . . . . . . . . Using Python Like Lisp to Make a Single Dispatcher . Creating Generic Methods in Lisp . . . . . Generic Methods with Python . . . . . . . . Context Managers . . . . . . . . . . . . . . . . . . . . . . . Less Boilerplate with attr . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 179 181 182 184 185 186 187 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Write Less, Code More Index 152 154 154 156 158 159 162 167 169 170 174 174 187 190 190 191 193 195 201 201 202 203 203 203 204 205 207 210 213 215

Acknowledgments Writing this first book has been a tremendous effort. Looking back, I had no clue how crazy this journey would be but also no idea how fulfilling it would turn out to be. They say that if you want to go fast you should go alone, but that if you want to go far you should go together. This is the fourth edition of the original book I wrote, and I would not have made it here without the people who helped along the way. This is a team effort and I would like to thank everyone who participated. Most of the interviewees gave me their time and trust without a second thought, and I owe a lot of what we teach in this book to them: Doug Hellmann for his great advice about building libraries, Joshua Harlow for his good humor and knowledge about distributed systems, Christophe de Vienne for his experience in building frameworks, Victor Stinner for his incredible CPython knowledge, Dimitri Fontaine for his database wisdom, Robert Collins for messing up with testing, Nick Coghlan for his work in getting Python into better shape, and Paul Tagliamonte for his amazing hacker spirit.

Thanks to the No Starch crew for working with me on bringing this book to a brand new level — especially to Liz Chadwick for her editing skills, Laurel Chun for keeping me on track, and Mike Driscoll for his technical insight. My gratitude also goes to the free software communities who shared their knowledge and helped me grow, especially to the Python community which always has been welcoming and enthusiastic. xvi   Acknowledgments

Introduction If you’re reading this, the odds are good you’ve been working with Python for some time already. Maybe you learned it using some tutorials, delved into some existing programs, or started from scratch. Whatever the case, you’ve hacked your way into learning it. That’s exactly how I got familiar with Python up until I started working on big open source projects 10 years ago. It is easy to think that you know and understand Python once you’ve written your first program. The language is that simple to grasp. However, it takes years to master it and to develop a deep comprehension of its advantages and shortcomings. When I started Python, I built my own Python libraries and applications on a “garage project” scale. Things changed once I started working with hundreds of developers on software that thousands of users rely on. For example, the OpenStack platform—a project I contribute to—represents over 9 million lines of Python code, which collectively needs to be concise,

efficient, and scalable to the needs of whatever cloud computing application its users require. When you have a project of this size, things like testing and documentation absolutely require automation, or else they won’t get done at all. I thought I knew a lot about Python before working on projects of this scale—a scale I could hardly imagine when I started out—but I’ve learned a lot more. I’ve also had the opportunity to meet some of the best Python hackers in the industry and learn from them. They’ve taught me everything from general architecture and design principles to various helpful tips and tricks. Through this book, I hope to share the most important things I’ve learned so that you can build better Python programs—and build them more efficiently, too! The first version of this book, The Hacker’s Guide to Python, came out in 2014. Now Serious Python is the fourth edition, with updated and entirely new contents. I hope you enjoy it! Who Should Read This Book and Why This book is intended for Python coders and developers who want to take their Python skills to the next level. In it, you’ll find methods and advice that will help you get the most out of Python and build future-proof programs. If you’re already working on a project, you’ll be able to apply the techniques discussed right away to improve your current code. If you’re starting your first project, you’ll be a

Mike Driscoll has been programming with Python for more than a decade. He has been writing about Python on his blog, The Mouse vs. The Python, for many years. Mike is the author of several Python books including Python 101, Python Interviews, and ReportLab: PDF Processing with Python. You can find Mike on Twitter or GitHub via his handle .