Python Cookbook, Third Edition
THIRD EDITIONPython CookbookDavid Beazley and Brian K. Jones
Python Cookbook, Third Editionby David Beazley and Brian K. JonesCopyright 2013 David Beazley and Brian Jones. All rights reserved.Printed in the United States of America.Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions arealso available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.Editors: Meghan Blanchette and Rachel RoumeliotisProduction Editor: Kristen BorgCopyeditor: Jasmine KwitynProofreader: BIM Proofreading ServicesMay 2013:Indexer: WordCo Indexing ServicesCover Designer: Karen MontgomeryInterior Designer: David FutatoIllustrator: Robert RomanoThird EditionRevision History for the Third Edition:2013-05-08: First release2014-03-07: Second releaseSee http://oreilly.com/catalog/errata.csp?isbn 9781449340377 for release details.Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’ReillyMedia, Inc. Python Cookbook, the image of a springhaas, and related trade dress are trademarks of O’ReillyMedia, Inc.Many of the designations used by manufacturers and sellers to distinguish their products are claimed astrademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trade‐mark claim, the designations have been printed in caps or initial caps.While every precaution has been taken in the preparation of this book, the publisher and authors assumeno responsibility for errors or omissions, or for damages resulting from the use of the information containedherein.ISBN: 978-1-449-34037-7[LSI]
Table of ContentsPreface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi1. Data Structures and Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1. Unpacking a Sequence into Separate Variables1.2. Unpacking Elements from Iterables of Arbitrary Length1.3. Keeping the Last N Items1.4. Finding the Largest or Smallest N Items1.5. Implementing a Priority Queue1.6. Mapping Keys to Multiple Values in a Dictionary1.7. Keeping Dictionaries in Order1.8. Calculating with Dictionaries1.9. Finding Commonalities in Two Dictionaries1.10. Removing Duplicates from a Sequence while Maintaining Order1.11. Naming a Slice1.12. Determining the Most Frequently Occurring Items in a Sequence1.13. Sorting a List of Dictionaries by a Common Key1.14. Sorting Objects Without Native Comparison Support1.15. Grouping Records Together Based on a Field1.16. Filtering Sequence Elements1.17. Extracting a Subset of a Dictionary1.18. Mapping Names to Sequence Elements1.19. Transforming and Reducing Data at the Same Time1.20. Combining Multiple Mappings into a Single Mapping135781112131517182021232426282932332. Strings and Text. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372.1. Splitting Strings on Any of Multiple Delimiters2.2. Matching Text at the Start or End of a String2.3. Matching Strings Using Shell Wildcard Patterns2.4. Matching and Searching for Text Patterns37384042iii
2.5. Searching and Replacing Text2.6. Searching and Replacing Case-Insensitive Text2.7. Specifying a Regular Expression for the Shortest Match2.8. Writing a Regular Expression for Multiline Patterns2.9. Normalizing Unicode Text to a Standard Representation2.10. Working with Unicode Characters in Regular Expressions2.11. Stripping Unwanted Characters from Strings2.12. Sanitizing and Cleaning Up Text2.13. Aligning Text Strings2.14. Combining and Concatenating Strings2.15. Interpolating Variables in Strings2.16. Reformatting Text to a Fixed Number of Columns2.17. Handling HTML and XML Entities in Text2.18. Tokenizing Text2.19. Writing a Simple Recursive Descent Parser2.20. Performing Text Operations on Byte Strings454647485052535457586164656669783. Numbers, Dates, and Times. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 833.1. Rounding Numerical Values3.2. Performing Accurate Decimal Calculations3.3. Formatting Numbers for Output3.4. Working with Binary, Octal, and Hexadecimal Integers3.5. Packing and Unpacking Large Integers from Bytes3.6. Performing Complex-Valued Math3.7. Working with Infinity and NaNs3.8. Calculating with Fractions3.9. Calculating with Large Numerical Arrays3.10. Performing Matrix and Linear Algebra Calculations3.11. Picking Things at Random3.12. Converting Days to Seconds, and Other Basic Time Conversions3.13. Determining Last Friday’s Date3.14. Finding the Date Range for the Current Month3.15. Converting Strings into Datetimes3.16. Manipulating Dates Involving Time Zones8384878990929496971001021041061071091104. Iterators and Generators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1134.1. Manually Consuming an Iterator4.2. Delegating Iteration4.3. Creating New Iteration Patterns with Generators4.4. Implementing the Iterator Protocol4.5. Iterating in Reverse4.6. Defining Generator Functions with Extra Stateiv Table of Contents113114115117119120
4.7. Taking a Slice of an Iterator4.8. Skipping the First Part of an Iterable4.9. Iterating Over All Possible Combinations or Permutations4.10. Iterating Over the Index-Value Pairs of a Sequence4.11. Iterating Over Multiple Sequences Simultaneously4.12. Iterating on Items in Separate Containers4.13. Creating Data Processing Pipelines4.14. Flattening a Nested Sequence4.15. Iterating in Sorted Order Over Merged Sorted Iterables4.16. Replacing Infinite while Loops with an Iterator1221231251271291311321351361385. Files and I/O. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1415.1. Reading and Writing Text Data5.2. Printing to a File5.3. Printing with a Different Separator or Line Ending5.4. Reading and Writing Binary Data5.5. Writing to a File That Doesn’t Already Exist5.6. Performing I/O Operations on a String5.7. Reading and Writing Compressed Datafiles5.8. Iterating Over Fixed-Sized Records5.9. Reading Binary Data into a Mutable Buffer5.10. Memory Mapping Binary Files5.11. Manipulating Pathnames5.12. Testing for the Existence of a File5.13. Getting a Directory Listing5.14. Bypassing Filename Encoding5.15. Printing Bad Filenames5.16. Adding or Changing the Encoding of an Already Open File5.17. Writing Bytes to a Text File5.18. Wrapping an Existing File Descriptor As a File Object5.19. Making Temporary Files and Directories5.20. Communicating with Serial Ports5.21. Serializing Python 611631651661671701716. Data Encoding and Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1756.1. Reading and Writing CSV Data6.2. Reading and Writing JSON Data6.3. Parsing Simple XML Data6.4. Parsing Huge XML Files Incrementally6.5. Turning a Dictionary into XML6.6. Parsing, Modifying, and Rewriting XML6.7. Parsing XML Documents with Namespaces175179183186189191193Table of Contents v
6.8. Interacting with a Relational Database6.9. Decoding and Encoding Hexadecimal Digits6.10. Decoding and Encoding Base646.11. Reading and Writing Binary Arrays of Structures6.12. Reading Nested and Variable-Sized Binary Structures6.13. Summarizing Data and Performing Statistics1951971991992032147. Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2177.1. Writing Functions That Accept Any Number of Arguments7.2. Writing Functions That Only Accept Keyword Arguments7.3. Attaching Informational Metadata to Function Arguments7.4. Returning Multiple Values from a Function7.5. Defining Functions with Default Arguments7.6. Defining Anonymous or Inline Functions7.7. Capturing Variables in Anonymous Functions7.8. Making an N-Argument Callable Work As a Callable with FewerArguments7.9. Replacing Single Method Classes with Functions7.10. Carrying Extra State with Callback Functions7.11. Inlining Callback Functions7.12. Accessing Variables Defined Inside a Closure2172192202212222242252272312322352388. Classes and Objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2438.1. Changing the String Representation of Instances8.2. Customizing String Formatting8.3. Making Objects Support the Context-Management Protocol8.4. Saving Memory When Creating a Large Number of Instances8.5. Encapsulating Names in a Class8.6. Creating Managed Attributes8.7. Calling a Method on a Parent Class8.8. Extending a Property in a Subclass8.9. Creating a New Kind of Class or Instance Attribute8.10. Using Lazily Computed Properties8.11. Simplifying the Initialization of Data Structures8.12. Defining an Interface or Abstract Base Class8.13. Implementing a Data Model or Type System8.14. Implementing Custom Containers8.15. Delegating Attribute Access8.16. Defining More Than One Constructor in a Class8.17. Creating an Instance Without Invoking init8.18. Extending Classes with Mixins8.19. Implementing Stateful Objects or State Machinesvi Table of 287291293294299
8.20. Calling a Method on an Object Given the Name As a String8.21. Implementing the Visitor Pattern8.22. Implementing the Visitor Pattern Without Recursion8.23. Managing Memory in Cyclic Data Structures8.24. Making Classes Support Comparison Operations8.25. Creating Cached Instances3053063113173213239. Metaprogramming. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3299.1. Putting a Wrapper Around a Function9.2. Preserving Function Metadata When Writing Decorators9.3. Unwrapping a Decorator9.4. Defining a Decorator That Takes Arguments9.5. Defining a Decorator with User Adjustable Attributes9.6. Defining a Decorator That Takes an Optional Argument9.7. Enforcing Type Checking on a Function Using a Decorator9.8. Defining Decorators As Part of a Class9.9. Defining Decorators As Classes9.10. Applying Decorators to Class and Static Methods9.11. Writing Decorators That Add Arguments to Wrapped Functions9.12. Using Decorators to Patch Class Definitions9.13. Using a Metaclass to Control Instance Creation9.14. Capturing Class Attribute Definition Order9.15. Defining a Metaclass That Takes Optional Arguments9.16. Enforcing an Argument Signature on *args and **kwargs9.17. Enforcing Coding Conventions in Classes9.18. Defining Classes Programmatically9.19. Initializing Class Members at Definition Time9.20. Implementing Multiple Dispatch with Function Annotations9.21. Avoiding Repetitive Property Methods9.22. Defining Context Managers the Easy Way9.23. Executing Code with Local Side Effects9.24. Parsing and Analyzing Python Source9.25. Disassembling Python Byte 6436737037437638238438638839210. Modules and Packages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39710.1. Making a Hierarchical Package of Modules10.2. Controlling the Import of Everything10.3. Importing Package Submodules Using Relative Names10.4. Splitting a Module into Multiple Files10.5. Making Separate Directories of Code Import Under a CommonNamespace10.6. Reloading ModulesTable of Contents397398399401404406 vii
10.7. Making a Directory or Zip File Runnable As a Main Script10.8. Reading Datafiles Within a Package10.9. Adding Directories to sys.path10.10. Importing Modules Using a Name Given in a String10.11. Loading Modules from a Remote Machine Using Import Hooks10.12. Patching Modules on Import10.13. Installing Packages Just for Yourself10.14. Creating a New Python Environment10.15. Distributing Packages40740840941141242843143243311. Network and Web Programming. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43711.1. Interacting with HTTP Services As a Client11.2. Creating a TCP Server11.3. Creating a UDP Server11.4. Generating a Range of IP Addresses from a CIDR Address11.5. Creating a Simple REST-Based Interface11.6. Implementing a Simple Remote Procedure Call with XML-RPC11.7. Communicating Simply Between Interpreters11.8. Implementing Remote Procedure Calls11.9. Authenticating Clients Simply11.10. Adding SSL to Network Services11.11. Passing a Socket File Descriptor Between Processes11.12. Understanding Event-Driven I/O11.13. Sending and Receiving Large Arrays43744144544744945445645846146447047548112. Concurrency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48512.1. Starting and Stopping Threads12.2. Determining If a Thread Has Started12.3. Communicating Between Threads12.4. Locking Critical Sections12.5. Locking with Deadlock Avoidance12.6. Storing Thread-Specific State12.7. Creating a Thread Pool12.8. Performing Simple Parallel Programming12.9. Dealing with the GIL (and How to Stop Worrying About It)12.10. Defining an Actor Task12.11. Implementing Publish/Subscribe Messaging12.12. Using Generators As an Alternative to Threads12.13. Polling Multiple Thread Queues12.14. Launching a Daemon Process on Unix48548849149750050450550951351652052453153413. Utility Scripting and System Administration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539viii Table of Contents
13.1. Accepting Script Input via Redirection, Pipes, or Input Files13.2. Terminating a Program with an Error Message13.3. Parsing Command-Line Options13.4. Prompting for a Password at Runtime13.5. Getting the Terminal Size13.6. Executing an External Command and Getting Its Output13.7. Copying or Moving Files and Directories13.8. Creating and Unpacking Archives13.9. Finding Files by Name13.10. Reading Configuration Files13.11. Adding Logging to Simple Scripts13.12. Adding Logging to Libraries13.13. Making a Stopwatch Timer13.14. Putting Limits on Memory and CPU Usage13.15. Launching a Web 6314. Testing, Debugging, and Exceptions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56514.1. Testing Output Sent to stdout14.2. Patching Objects in Unit Tests14.3. Testing for Exceptional Conditions in Unit Tests14.4. Logging Test Output to a File14.5. Skipping or Anticipating Test Failures14.6. Handling Multiple Exceptions14.7. Catching All Exceptions14.8. Creating Custom Exceptions14.9. Raising an Exception in Response to Another Exception14.10. Reraising the Last Exception14.11. Issuing Warning Messages14.12. Debugging Basic Program Crashes14.13. Profiling and Timing Your Program14.14. Making Your Programs Run . C Extensions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59715.1. Accessing C Code Using ctypes15.2. Writing a Simple C Extension Module15.3. Writing an Extension Function That Operates on Arrays15.4. Managing Opaque Pointers in C Extension Modules15.5. Defining and Exporting C APIs from Extension Modules15.6. Calling Python from C15.7. Releasing the GIL in C Extensions15.8. Mixing Threads from C and Python15.9. Wrapping C Code with Swig599605609612614619625625627Table of Contents ix
15.10. Wrapping Existing C Code with Cython15.11. Using Cython to Write High-Performance Array Operations15.12. Turning a Function Pointer into a Callable15.13. Passing NULL-Terminated Strings to C Libraries15.14. Passing Unicode Strings to C Libraries15.15. Converting C Strings to Python15.16. Working with C Strings of Dubious Encoding15.17. Passing Filenames to C Extensions15.18. Passing Open Files to C Extensions15.19. Reading File-Like Objects from C15.20. Consuming an Iterable from C15.21. Diagnosing Segmentation Faults632638643644648653654657658659662663A. Further Reading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 665Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 667x Table of Contents
PrefaceSince 2008, the Python world has been watching the slow evolution of Python 3. It wasalways known that the adoption of Python 3 would likely take a long time. In fact, evenat the time of this writing (2013), most working Python programmers continue to usePython 2 in production. A lot has been made about the fact that Python 3 is not backwardcompatible with past versions. To be sure, backward compatibility is an issue for anyonewith an existing code base. However, if you shift your view toward the future, you’ll findthat Python 3 offers much more than meets the eye.Just as Python 3 is about the future, this edition of the Python Cookbook represents amajor change over past editions. First and foremost, this is meant to be a very forwardlooking book. All of the recipes have been written and tested with Python 3.3 withoutregard to past Python versions or the “old way” of doing things. In fact, many of therecipes will only work with Python 3.3 and above. Doing so may be a calculated risk,but the ultimate goal is to write a book of recipes based on the most modern tools andidioms possible. It is hoped that the recipes can serve as a guide for people writing newcode in Python 3 or those who hope to modernize existing code.Needless to say, writing a book of recipes in this style presents a certain editorial chal‐lenge. An online search for Python recipes returns literally thousands of useful recipeson sites such as ActiveState’s Python recipes or Stack Overflow. However, most of theserecipes are steeped in history and the past. Besides being written almost exclusively forPython 2, they often contain workarounds and hacks related to differences between oldversions of Python (e.g., version 2.3 versus 2.4). Moreover, they often use outdatedtechniques that have simply become a built-in feature of Python 3.3. Finding recipesexclusively focused on Python 3 can be a bit more difficult.Rather than attempting to seek out Python 3-specific recipes, the topics of this book aremerely inspired by existing code and techniques. Using these ideas as a springboard,the writing is an original work that has been deliberately written with the most modernPython programming techniques possible. Thus, it can serve as a reference for anyonewho wants to write their code in a modern style.xi
In choosing which recipes to include, there is a certain realization that it is simplyimpossible to write a book that covers every possible thing that someone might do withPython. Thus, a priority has been given to topics that focus on the core Python languageas well as tasks that are common to a wide variety of application domains. In addition,many of the recipes aim to illustrate features that are new to Python 3 and more likelyto be unknown to even experienced programmers using older versions. There is also acertain preference to recipes that illustrate a generally applicable programming tech‐nique (i.e., programming patterns) as opposed to those that narrowly try to address avery specific practical problem. Although certain third-party packages get coverage, amajority of the recipes focus on the core language and standard library.Who This Book Is ForThis book is aimed at more experienced Python programmers who are looking todeepen their understanding of the language and modern programming idioms. Muchof the material focuses on some of the more advanced techniques used
10.7. Making a Directory or Zip File Runnable As a Main Script 407 10.8. Reading Datafiles Within a Package 408
Python Programming for the Absolute Beginner Second Edition. CONTENTS CHAPTER 1 GETTING STARTED: THE GAME OVER PROGRAM 1 Examining the Game Over Program 2 Introducing Python 3 Python Is Easy to Use 3 Python Is Powerful 3 Python Is Object Oriented 4 Python Is a "Glue" Language 4 Python Runs Everywhere 4 Python Has a Strong Community 4 Python Is Free and Open Source 5 Setting Up Python on .
Python 2 versus Python 3 - the great debate Installing Python Setting up the Python interpreter About virtualenv Your first virtual environment Your friend, the console How you can run a Python program Running Python scripts Running the Python interactive shell Running Python as a service Running Python as a GUI application How is Python code .
Python is readable 5 Python is complete—"batteries included" 6 Python is cross-platform 6 Python is free 6 1.3 What Python doesn't do as well 7 Python is not the fastest language 7 Python doesn't have the most libraries 8 Python doesn't check variable types at compile time 8 1.4 Why learn Python 3? 8 1.5 Summary 9
SAP has developed a new radio frequency (RF) concept. This RF cookbook helps developers to begin working in the RF framework. It answers frequently asked questions and helps to avoid common errors. This RF cookbook also provides some useful tips about the standard layout and screen structure that should be applied in the standard transactions.File Size: 299KBPage Count: 59Explore further[PDF] SAP EWM RF Cookbook - Free Download PDFdlscrib.comEWM RF Cookbook SAP blog of John Kristensenjksap.wordpress.comRF Cookbook - Part I Description - SAP Communityarchive.sap.comRF Cookbook - Part I Descriptiondocshare01.docshare.tipsSAP EWM RF Framework - SlideSharewww.slideshare.netRecommended to you based on what's popular Feedback
site "Python 2.x is legacy, Python 3.x is the present and future of the language". In addition, "Python 3 eliminates many quirks that can unnecessarily trip up beginning programmers". However, note that Python 2 is currently still rather widely used. Python 2 and 3 are about 90% similar. Hence if you learn Python 3, you will likely
Related titles Essential System Administration Learning Python Linux Networking Cookbook Linux Security Cookbook Mac OS X for Unix Geeks Programming Python Python Cookbook Python in a Nutshell Unix in a Nutshell oreilly.com oreilly.com is more than a complete catalog ofO'Reilly books. You'll also find links to news, events, articles .
There are currently two versions of Python in use; Python 2 and Python 3. Python 3 is not backward compatible with Python 2. A lot of the imported modules were only available in Python 2 for quite some time, leading to a slow adoption of Python 3. However, this not really an issue anymore. Support for Python 2 will end in 2020.
Active Filter Cookbook, CMOS Cookbook, TTL Cook book, RTL Cookbook (out of print), TVT Cookbook, Cheap Video Cookbook, Son of Cheap Video, The Hex adecimal Chronicles, The Incredible Secret M
