Tutorial: Asynchonous Python with Twisted (and asyncio); Part one

Repository

• https://github.com/twisted/twisted

What Will I Learn?

In part one of this multipart tutorial, aimed at teaching you the use of asynchronous programming paradigms using Twisted, you will learn about two of the five pillars of the Twisted architecture: The Reactor and Deferreds.

• You will learn about event loops and how, while there can be only one event loop implementation in your applications, that there are ways to still use multiple event-loop based framework.
• You will learn how Twisted can be used in conjunction with other popular event based frameworks such as Qt and Python-3 (>= 3.5) asyncio.
• You will learn about different typed of Python such as functions, methods, lambdas and function objects.
• You will learn about the use of Deferreds for performing callbacks in an asynchronous programming setting.
• You will learn about the similarities and differences between Twisted Deferreds and asyncio Futures, and how to glue usage of the two together in a single program.

Requirements

To follow this tutorial, you should:

• Have a good working knowledge of Python

Difficulty

• Intermediate

Asynchronous Python with Twisted; Part 1

This tutorial is based on a half-day training I gave on the subject at my place of employment last week. In this (multi-part) tutorial we will look at asynchronous programming in Python. The primary focus in this tutorial shall be on the Twisted framework. We shall though, occasionally visit the Python 3 (>= 3.5) asyncio facilities; primarily from an interoperability perspective. Before you dismiss this tutorial as irrelevant because you believe asyncio supersedes and deprecates asynchronous frameworks such as Twisted, I should probably spend a few words on why a few like this on the relationship between Twisted and asyncio is premature at best and completely misguided at worst.

Asyncio is now part of the standard python library and as such, as of Python 3.5 (or 3.4 if you consider an earlier API incomplete version) provides the Python language with asynchronous primitives at a low level. In contrast, Twisted is a mature and full fledged asynchronous framework including facilities for asynchronous programming at multiple abstraction levels, supporting a variety of protocols and transports. If all the async operations you want to do are against sockets and timed events, and support for older versions of Python is a none-issue, then using just asyncio might be a good idea. If however you want things like HTTP(s), DNS, SSH, Tor, Access to a database like PostgreSQL or Redis, you will need to either use a batteries-included framework like Twisted, or choose third party libraries built on top of asyncio. One issue with the later at the moment of writing is that, with asyncio being relatively new, and big changes between asyncio from Python 3.4 to 3.5, that the current quality of libraries for asyncio on pypi is a bit like Node JS libraries in the early days of Node. There might be five or six libraries doing approximately the same thing and at least two-thirds of these will be low quality code, with chances the remaining ones are just inferior asynchronous wrappers for a synchronous library. It will be up to you to figure out if any of these is worth your time. Yes, the future might look different, but then, as this tutorial will show, at no time will you ever need to choose for asyncio. The two worlds can coexist. And as Python 2 fades out of relevance in say the coming five to ten years, Twisted is likely to move more of its code base to being asyncio based.

Before we continue to discussing asynchronous programming with Twisted, we first need to take a step back and look at what we mean when we talk about asynchronous operations. Consider you are a cook in a diner. It is your job to take orders, prepare plates, fry eggs and hand the finished dish to the customer. There is no one else working the frying plate, just you. You could take a single order, prepare the plate, put the eggs on the frying plate, wait, patiently for the egg to reach the perfect state, put the eggs on the plate, hand the plate to the customer and take the next order. This would be the synchronous option. If the customers would start queuing to outside of the diner, your boss might hire a second cook and maybe a third, and we'dd still be handling things 100% synchronously.

But let us consider the work flow. You as a cook will be spending more than 50% of your time just looking at the cooking plate, waiting for a slow operation, the egg cooking on the plate, to finish. Instead, in an asynchronous model, you could play with the few minutes it takes for the egg to cook, using that time smarter. If there is a queue of customers, you might first take orders for a small set of customers. Create a short queue of orders, put all eggs for all these orders on the plate, then use the time it takes for the eggs to cook to prepare the plates and take the orders for the next batch of customers. It is, I hope, easy to see how this second, asynchronous work schedule makes much more efficient use of your capacity for work. By refusing to block on the blocking 'cook egg' operation, you could manage to maybe do the same amount of work that two or even three people would be able to do if they were to stick to a strictly synchronous model.

Now before we dive into Twisted and asyncio, let us look a bit at a bit broader picture of asynchronous frameworks on Python. Twisted is an old and mature asynchronous framework for Python. Quite a bit older in fact than other well-known asynchronous frameworks for other languages such as NodeJS (2009) for JavaScript and even Boost::Asio (2005) for C++.
Where Twisted is a general purpose asynchronous framework, the Tornado framework is aimed specifically at web oriented asynchronous projects. An interesting piece of glue between Twisted and Tornado is Cyclone. Cyclone basically provides the Tornado API on top of Twisted, allowing for a migration path for Tornado based projects.
Finally there is Python 3 asyncio. Low level asynchronous primitives for Python >= 3.5.
In this tutorial we shall focus on Twisted, while making sure to touch on compatibility with asyncio wherever needed.

The first of the core pillars of Twisted, and the core of any async framework is the event loop.

It is important to realize that using a framework build upon an event loop, you will hand over control of the processor (for the main thread) to the event loop. So while there are different frameworks and libraries that come with their own event loop, in the end, your program can have only one event loop.

One mistake many people make when learning that there can be only one event loop is assuming that there can be only one event-loop based framework. While it can sometimes be a bit of a puzzle combining more than two frameworks in a single application, as we will see later, there are ways to let multiple frameworks use a single shared event loop implementation and thus using the frameworks together.

In twisted, the event loop is captured within the Reactor abstraction. You can think of the reactor, at least the default one as an advanced select loop. In twisted, the Reactor API provides facilities for things like network communication, timed events and threading.

Next to the default select based reactor, it is possible to use a platform specific reactor instead with Twisted, or to use a reactor that binds to the main event loop of for example a GUI framework like Qt.

Some code examples using different event loop implementations.

For completeness, part of the reactor API of the reactor abstraction in Twisted.

Now for comparison, let's have a look at the event loop usage code with asyncio on Python 3.4 and higher.

Just like in Twisted, we can pick different underlying event loops with asyncio, for example the Qt framework event loop.

The API of the event loop abstraction in asyncio provides most of the functionality that the Reactor abstraction API provides in Twisted.

If you find yourself in a situation where you want to use both Twisted ans asyncio based functionality, it is easy to use the asyncio event loop from Twisted and combine the use of both.

Now before we move on to Deferreds in Twisted (and Futures in asyncio), we need to look at the concept of callables.

There are many types of callables in Python. Regular functions, methods of objects, lambdas, function objects, etc, etc.
Let's look at an example of the different types of callables.

While in the Python type system, all these callables have different types, each of them is callable and thus usable as a callback in asynchronous programming.

One special usage pattern for functions is in a closure construct. The above function returns a function. When the returned function is invoked, the lexical scope of the function returning the function will still be active.

By using a closure as above, in the definition of a getattr method, it is possible to define a sort of wildcard method for an object. We will see later, when we get to databases, that wildcard methods can be extremely powerful and compact.

Back to the five pillars of the Twisted architecture. We already covered the Reactor. Now let's move on to the Deferred.

Deffereds in Twisted and Futures in asyncio are comparable abstractions that allow for the decoupling of synchonous blocks of code. Think of Deferreds (and Futures) as an immediate result to a (possibly) unresolved blocking operation. Think of it as a deferred result or a future result. You can't use the result like a value, but you can set callbacks for when results become available.

It is easyest to understand thit by considering a simple task involving waiting. For example pressing the button at the pedestrian crossing and waiting for the light to go green before crossing the street. Above we see an example, bost synconously using blocking operations and asynchonously using a non blocking operation that instantly returns a deferred.

Here we see two more concrete pieces of code. Sychronous code using try/except/finaly and its asynchronous equivalent using a deferred in Twisted. Note that we add multiple callbacks and error handling callbacks (errbacks) on a single deferred. Consider that a deferred can be viewed as a double list of calbacks. First we add both a callback to the success path and an error handler to the error path, then we add the single callback to both the success and the error path.

Here we see a graphical representation of the deferred as double list with the different operations for adding one or two callbacks to a deferred.

Considder we have a deferred returning function that normally fetches a page from a site on the internet. Now what if we add some checking to see if the page name is a valid page name before we fetch the page. If it isn't we won't need to do a blocking call as we know beforehand that the fetch will fail. For those situations we can return an imediately resolving deferred by invoking defer.fail with an exception type.

The same goes for if we were to use a non blocking in memory cache. We invoke defer.succeed with a success value and get an instantly resolved deferred to return from our function.

Python 3, since 3.5 has asyncio language extentions for a concpt quite similar to deferreds. The async keyword (3.4) combined with await (3.5) make for a powerfull syntax for returning a Future from a function containing a blocking function call.

If you write a program using both Twisted and asyncio based libraries, next to making sure you use one single event loop, it is likely that at some point you will need a deferred while you have a future or you will need a future where you have a deferred. While the two are similar, they are not the same, but fortunately you can create a future from a deferred and vice versa.

Curriculum

• Writing a simple STEEM statistics-script in Python with asyncsteem
• Writing an asynchronous STEEM web-app in Python with asyncsteem
• Building an asyncsteem python web app with micro-transaction-based steem user authentication, Redis and Jinja2