book

Fluent Python

by Luciano Ramalho

August 2015

Intermediate

790 pages

18h 48m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Who This Book Is ForWho This Book Is Not ForHow This Book Is OrganizedHands-On ApproachHardware Used for TimingsSoapbox: My Personal PerspectivePython JargonPython Version CoveredConventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgments
A Pythonic Card DeckHow Special Methods Are UsedEmulating Numeric TypesString RepresentationArithmetic OperatorsBoolean Value of a Custom TypeOverview of Special MethodsWhy len Is Not a MethodChapter SummaryFurther Reading
Overview of Built-In SequencesList Comprehensions and Generator ExpressionsList Comprehensions and ReadabilityListcomps Versus map and filterCartesian ProductsGenerator ExpressionsTuples Are Not Just Immutable ListsTuples as RecordsTuple UnpackingNested Tuple UnpackingNamed TuplesTuples as Immutable ListsSlicingWhy Slices and Range Exclude the Last ItemSlice ObjectsMultidimensional Slicing and EllipsisAssigning to SlicesUsing + and * with SequencesBuilding Lists of ListsAugmented Assignment with SequencesA += Assignment Puzzlerlist.sort and the sorted Built-In FunctionManaging Ordered Sequences with bisectSearching with bisectInserting with bisect.insortWhen a List Is Not the AnswerArraysMemory ViewsNumPy and SciPyDeques and Other QueuesChapter SummaryFurther Reading
Generic Mapping Typesdict ComprehensionsOverview of Common Mapping MethodsHandling Missing Keys with setdefaultMappings with Flexible Key Lookupdefaultdict: Another Take on Missing KeysThe __missing__ MethodVariations of dictSubclassing UserDictImmutable MappingsSet Theoryset LiteralsSet ComprehensionsSet Operationsdict and set Under the HoodA Performance ExperimentHash Tables in DictionariesPractical Consequences of How dict WorksHow Sets Work—Practical ConsequencesChapter SummaryFurther Reading
Character IssuesByte EssentialsStructs and Memory ViewsBasic Encoders/DecodersUnderstanding Encode/Decode ProblemsCoping with UnicodeEncodeErrorCoping with UnicodeDecodeErrorSyntaxError When Loading Modules with Unexpected EncodingHow to Discover the Encoding of a Byte SequenceBOM: A Useful GremlinHandling Text FilesEncoding Defaults: A MadhouseNormalizing Unicode for Saner ComparisonsCase FoldingUtility Functions for Normalized Text MatchingExtreme “Normalization”: Taking Out DiacriticsSorting Unicode TextSorting with the Unicode Collation AlgorithmThe Unicode DatabaseDual-Mode str and bytes APIsstr Versus bytes in Regular Expressionsstr Versus bytes on os FunctionsChapter SummaryFurther Reading
Treating a Function Like an ObjectHigher-Order FunctionsModern Replacements for map, filter, and reduceAnonymous FunctionsThe Seven Flavors of Callable ObjectsUser-Defined Callable TypesFunction IntrospectionFrom Positional to Keyword-Only ParametersRetrieving Information About ParametersFunction AnnotationsPackages for Functional ProgrammingThe operator ModuleFreezing Arguments with functools.partialChapter SummaryFurther Reading
Case Study: Refactoring StrategyClassic StrategyFunction-Oriented StrategyChoosing the Best Strategy: Simple ApproachFinding Strategies in a ModuleCommandChapter SummaryFurther Reading

Decorators 101When Python Executes DecoratorsDecorator-Enhanced Strategy PatternVariable Scope RulesClosuresThe nonlocal DeclarationImplementing a Simple DecoratorHow It WorksDecorators in the Standard LibraryMemoization with functools.lru_cacheGeneric Functions with Single DispatchStacked DecoratorsParameterized DecoratorsA Parameterized Registration DecoratorThe Parameterized Clock DecoratorChapter SummaryFurther Reading
Variables Are Not BoxesIdentity, Equality, and AliasesChoosing Between == and isThe Relative Immutability of TuplesCopies Are Shallow by DefaultDeep and Shallow Copies of Arbitrary ObjectsFunction Parameters as ReferencesMutable Types as Parameter Defaults: Bad IdeaDefensive Programming with Mutable Parametersdel and Garbage CollectionWeak ReferencesThe WeakValueDictionary SkitLimitations of Weak ReferencesTricks Python Plays with ImmutablesChapter SummaryFurther Reading
Object RepresentationsVector Class ReduxAn Alternative Constructorclassmethod Versus staticmethodFormatted DisplaysA Hashable Vector2dPrivate and “Protected” Attributes in PythonSaving Space with the __slots__ Class AttributeThe Problems with __slots__Overriding Class AttributesChapter SummaryFurther Reading
Vector: A User-Defined Sequence TypeVector Take #1: Vector2d CompatibleProtocols and Duck TypingVector Take #2: A Sliceable SequenceHow Slicing WorksA Slice-Aware __getitem__Vector Take #3: Dynamic Attribute AccessVector Take #4: Hashing and a Faster ==Vector Take #5: FormattingChapter SummaryFurther Reading
Interfaces and Protocols in Python CulturePython Digs SequencesMonkey-Patching to Implement a Protocol at RuntimeAlex Martelli’s WaterfowlSubclassing an ABCABCs in the Standard LibraryABCs in collections.abcThe Numbers Tower of ABCsDefining and Using an ABCABC Syntax DetailsSubclassing the Tombola ABCA Virtual Subclass of TombolaHow the Tombola Subclasses Were TestedUsage of register in PracticeGeese Can Behave as DucksChapter SummaryFurther Reading
Subclassing Built-In Types Is TrickyMultiple Inheritance and Method Resolution OrderMultiple Inheritance in the Real WorldCoping with Multiple Inheritance1. Distinguish Interface Inheritance from Implementation Inheritance2. Make Interfaces Explicit with ABCs3. Use Mixins for Code Reuse4. Make Mixins Explicit by Naming5. An ABC May Also Be a Mixin; The Reverse Is Not True6. Don’t Subclass from More Than One Concrete Class7. Provide Aggregate Classes to Users8. “Favor Object Composition Over Class Inheritance.”Tkinter: The Good, the Bad, and the UglyA Modern Example: Mixins in Django Generic ViewsChapter SummaryFurther Reading
Operator Overloading 101Unary OperatorsOverloading + for Vector AdditionOverloading * for Scalar MultiplicationRich Comparison OperatorsAugmented Assignment OperatorsChapter SummaryFurther Reading
Sentence Take #1: A Sequence of WordsWhy Sequences Are Iterable: The iter FunctionIterables Versus IteratorsSentence Take #2: A Classic IteratorMaking Sentence an Iterator: Bad IdeaSentence Take #3: A Generator FunctionHow a Generator Function WorksSentence Take #4: A Lazy ImplementationSentence Take #5: A Generator ExpressionGenerator Expressions: When to Use ThemAnother Example: Arithmetic Progression GeneratorArithmetic Progression with itertoolsGenerator Functions in the Standard LibraryNew Syntax in Python 3.3: yield fromIterable Reducing FunctionsA Closer Look at the iter FunctionCase Study: Generators in a Database Conversion UtilityGenerators as CoroutinesChapter SummaryFurther Reading
Do This, Then That: else Blocks Beyond ifContext Managers and with BlocksThe contextlib UtilitiesUsing @contextmanagerChapter SummaryFurther Reading
How Coroutines Evolved from GeneratorsBasic Behavior of a Generator Used as a CoroutineExample: Coroutine to Compute a Running AverageDecorators for Coroutine PrimingCoroutine Termination and Exception HandlingReturning a Value from a CoroutineUsing yield fromThe Meaning of yield fromUse Case: Coroutines for Discrete Event SimulationAbout Discrete Event SimulationsThe Taxi Fleet SimulationChapter SummaryFurther Reading
Example: Web Downloads in Three StylesA Sequential Download ScriptDownloading with concurrent.futuresWhere Are the Futures?Blocking I/O and the GILLaunching Processes with concurrent.futuresExperimenting with Executor.mapDownloads with Progress Display and Error HandlingError Handling in the flags2 ExamplesUsing futures.as_completedThreading and Multiprocessing AlternativesChapter SummaryFurther Reading
Thread Versus Coroutine: A Comparisonasyncio.Future: Nonblocking by DesignYielding from Futures, Tasks, and CoroutinesDownloading with asyncio and aiohttpRunning Circles Around Blocking CallsEnhancing the asyncio downloader ScriptUsing asyncio.as_completedUsing an Executor to Avoid Blocking the Event LoopFrom Callbacks to Futures and CoroutinesDoing Multiple Requests for Each DownloadWriting asyncio ServersAn asyncio TCP ServerAn aiohttp Web ServerSmarter Clients for Better ConcurrencyChapter SummaryFurther Reading
Data Wrangling with Dynamic AttributesExploring JSON-Like Data with Dynamic AttributesThe Invalid Attribute Name ProblemFlexible Object Creation with __new__Restructuring the OSCON Feed with shelveLinked Record Retrieval with PropertiesUsing a Property for Attribute ValidationLineItem Take #1: Class for an Item in an OrderLineItem Take #2: A Validating PropertyA Proper Look at PropertiesProperties Override Instance AttributesProperty DocumentationCoding a Property FactoryHandling Attribute DeletionEssential Attributes and Functions for Attribute HandlingSpecial Attributes that Affect Attribute HandlingBuilt-In Functions for Attribute HandlingSpecial Methods for Attribute HandlingChapter SummaryFurther Reading
Descriptor Example: Attribute ValidationLineItem Take #3: A Simple DescriptorLineItem Take #4: Automatic Storage Attribute NamesLineItem Take #5: A New Descriptor TypeOverriding Versus Nonoverriding DescriptorsOverriding DescriptorOverriding Descriptor Without __get__Nonoverriding DescriptorOverwriting a Descriptor in the ClassMethods Are DescriptorsDescriptor Usage TipsDescriptor docstring and Overriding DeletionChapter SummaryFurther Reading
A Class FactoryA Class Decorator for Customizing DescriptorsWhat Happens When: Import Time Versus RuntimeThe Evaluation Time ExercisesMetaclasses 101The Metaclass Evaluation Time ExerciseA Metaclass for Customizing DescriptorsThe Metaclass __prepare__ Special MethodClasses as ObjectsChapter SummaryFurther Reading
Further Reading
Chapter 3: in Operator Performance TestChapter 3: Compare the Bit Patterns of HashesChapter 9: RAM Usage With and Without __slots__Chapter 14: isis2json.py Database Conversion ScriptChapter 16: Taxi Fleet Discrete Event SimulationChapter 17: Cryptographic ExamplesChapter 17: flags2 HTTP Client ExamplesChapter 19: OSCON Schedule Scripts and Tests

Content preview from Fluent Python

Chapter 7. Function Decorators and Closures

There’s been a number of complaints about the choice of the name “decorator” for this feature. The major one is that the name is not consistent with its use in the GoF book.¹ The name decorator probably owes more to its use in the compiler area—a syntax tree is walked and annotated.

PEP 318 — Decorators for Functions and Methods

Function decorators let us “mark” functions in the source code to enhance their behavior in some way. This is powerful stuff, but mastering it requires understanding closures.

One of the newest reserved keywords in Python is nonlocal, introduced in Python 3.0. You can have a profitable life as a Python programmer without ever using it if you adhere to a strict regimen of class-centered object orientation. However, if you want to implement your own function decorators, you must know closures inside out, and then the need for nonlocal becomes obvious.

Aside from their application in decorators, closures are also essential for effective asynchronous programming with callbacks, and for coding in a functional style whenever it makes sense.

The end goal of this chapter is to explain exactly how function decorators work, from the simplest registration decorators to the rather more complicated parameterized ones. However, before we reach that goal we need to cover: