Python (Advanced) training course

Concurrency

A section on executing multiple tasks in parallel, based on an understanding of the underlying hardware (CPU and operating system), a brief history of concurrency, and working with async (asyncio), threads and multiprocessing. With a discussion of the Global Interpreter Lock, free threading, interprocess communication and shared memory, plus best practises via the actor model (CSP - Concurrent Sequential Programming).

• Concurrency basic principles
• Multitasking on a single CPU; context switching
• True concurrency
• I/O bound and CPU bound tasks
• Processes and the operating system
• Use processes and not threads
• Python and the Global Interpreter Lock (GIL)
• Python 3.13 and free threading
• Async and the event loop
• Await and coroutines
• asyncio.gather and true concurrency with asyncio
• async generators
• Threading and the main thread
• The threading module
• Launching a function on a new thread
• Thread joining, thread cancelling
• Race conditions and mutex locks
• Thread coordination with events, flags and sempahores
• The threading APIs
• Thread pools
• Concurrency with multiprocessing
• Process lifetime management
• The actor model with queues, processes and object passing
• Shared memory with multiprocessing

Context Managers and \_\_init_subclass__
 

Context managers and the with statement are a powerful and integral part of Python. Even basic file I/O uses the with statement. This section explains how context managers and the with statement work, how to write your own context managers, and the tools available in the `contextlib` module to help.

\_\_init_subclass__ is a reasonably recent addition to Python and an amazing tool for customizing class creation. Particularly useful for frameworks that need just a little bit of magic, \_\_init_subclass__ can be used for many of the things that previously required metaclasses. An advanced feature of Python object orientation with practical uses.

* Try/finally for resource management
* With statement for file I/O and locks
* Context managers and exception handling: `pytest.raises(...)`
* Context managers returning a value: `mock.patch(...) as mock`
* The context manager protocol
    - \_\_enter__
    - \_\_exit__
    - Returning a value
    - Handling exceptions
    - The full Python equivalent code
    - \_\_aenter__ and \_\_aexit__ for `async with`
* `contextlib.contextmanager`: context managers written as a generator
* \_\_init_subclass__ method
* Keyword arguments and class construction syntax
* Example with class level caches

Iterators, Generators, References and Assignment

* The iteration protocol
* \_\_iter__ and \_\_next__ methods
* Delegating to builtin iterators (like listiterator)
* The builting in `next` and `iter` functions
* Stateful iteration with generator functions
* Coroutine behaviour: suspend and resume execution, yielding values
* The `yield` keyword
* Adding iteration support to objects
* References, assignment and mutability
* Names not variables (references not pointers)
* Mutable objects and multiple references
* Identity versus equality
* Call by object
* Object copying


Imports, Modules and Namespaces

Modules and packages are the larger units of abstraction that Python provides, along with functions and classes, for building systems with. Much of the state of a running system lives in the global namespaces of imported modules, the `sys.modules` import cache. We'll look at import syntax, the import machinery, the importance of namespaces and factoring code into modules and packages.

* Import syntax variations
* namespaces and variable lookups
* Global variables
* sys.modules and the import cache
* Module objects
* Module level functionality: \_\_dir\__ and \_\_getattr__
* Packages and the filesystem
* Refactoring modules into libraries
* Refactoring libraries into packages
* Relative import syntax
* Module reloading (how to do it and why not to do it)
* `__import__` and the `importlib` module
* Circular imports, avoiding and fixing
* Executable modules and packages

Networking

Web scraping and API clients use HTTP client code whilst API servers live on the server side and we'll cover both sides of the equations and the request -> response cycle that API clients and servers are built on top of.

• HTTP request response cycle
• The requests module
• HTTP clients for REST APIs
• pip, requirements.txt and venv for project management
• Sockets , TCP/IP
• TCP and UDP, IPv4, IPv6

The Core Python Object Model

A look at the whole Python object model from attributes to inheritance to super and the method resolution order. Perhaps the most important module in the course this module describes the behaviour of all Python objects, inside and out.

* Objects as abstractions, for thinking
* Objects, methods and instantiation
* The class statement
* The explicit self
* The initialiser method \_\_init__
* Bound methods
* Attributes and the built-in attribute access functions
* Class attributes
* Class methods
* Properties
* Private attributes
* Single inheritance
* Inheritance to extend and modify the parent
* The use of super
* Cooperative multiple inheritance
* Mixin Classes
* Attribute lookup and the method resolution order
* The type system: isinstance and issubclass
* Inside Python objects and classes
* The instance dictionary
* Slots
* Class dictionaries
* Composition over inheritance
* Magic methods and Python protocols
* Operator overloading
* The string representation protocol
* The container protocols
* Implementing custom containers

Testing with pytest

* virtual environments and pipenv (installing pytest)
* pytest command line for collecting and running tests
* Simple test functions and asserts
* Test fixtures and conftest.py
* Testing exceptions with `pytest.raises`
* Test parameterisation for test combinations
* Test marking for running test subsets
* Principles of testing (unit tests versus end to end testing, building test helpers etc)
* Mocking and patching
* Testing API servers with Flask and pytest test client

Floats, Unicode, and Regular Expressions

The minimum every developer must know about floating point numbers, unicode, and character encodings. Along with a fast paced but in depth look at regular expressions. The exercises illustrate a Test Driven Development approach with unittest.

• Floating point numbers
• Unicode, encodings and strings
• Regular expressions

List Comprehensions and Generator Expressions

* For loops and iteration
* Tuple unpacking, multiple iteration variables
* Keeping track of iteration with `enumerate`
* Combining iterators with `zip`, iteration over pairs
* Building dictionaries with zip and item, value pairs: `dict(zip(keys, values))`
* List comprehension syntax
* How to read list comprehensions (from the middle)
* List comprehensions with filters
* List comprehensions and query like syntax
* Eager versus lazy: list comprehensions versus generator expressions
* Generator expressions as one shot iterators
* Ordered dictionaries
* Ternary expressions

Advanced Python OOP

An module on some advanced features of the Python object model. Useful for building larger systems, a full understanding of the Python object model, and the creation of frameworks or understanding the mechanisumes used by frameworks.

• Two phase object creation
• Weak references
• Destructors
• Metaclasses