Tutorial: Asynchonous Python with Twisted (and asyncio); Part Two

Repository

• https://github.com/twisted/twisted

What Will I Learn?

In part one of this multipart tutorial, aimed at teaching you the use of asynchronous programming paradigms using Twisted, you learned about the first two of the five pillars of the Twisted architecture: The Reactor and Deferreds.

In this second part we move on to the other three pillars: Transports, Protocols and Applications. As in the previous tutorial we will lightly touch on Python3 asyncio for part of the low level stuff. The main things you will learn in this installments are:

• You will learn about the different options for Logging in with Twisted.
• You will learn how to subclass Protocol to write a client or server using sockets.
• You will learn about different types of Transports that next to plain sockets you can use with your Protocols.
• You will learn how socket usage in Twisted compares to the equivalent in Python 3 asyncio.
• You will learn about running a Twisted Application using twistd.
• You will learn how to use timed events in Twisted
• You will learn how to invoke external programs and process their output asynchronously with Twisted.
• You will learn how to use threads in conjunction with Twisted in order to write a wrapper for a synchronous blocking API call.
• You will learn how to run CPU intensive code in a different process using Twistes Spread Perspective Brokers.

Requirements

To follow this tutorial, you should:

• Have a good working knowledge of Python
• Have read part one of this series.

Difficulty

• Intermediate

Asynchronous Python with Twisted; Part 2

In Twisted there are two distinct ways to integrate logging into your asynchronous program. We can either use twisted.logger, a limited configurable facility for logging with a convenient API akin to that of python's own logging facility, or we can use twisted.python.log that runs on top of Python's own logging facility, and as such is highly configurable. Let us have a shoer look at both options.

The code above shows how to start the logger and make it log to a plain text log file.

The standard Twisted logger basically has only two out of the box Log Observers to choose from. One for logging to a JSON file. Usage of the logger is very simple and works with an API not unlike that of the standard logging library you may be used to using.

Alternative to twisted.logger, the twisted.python.log API gives a different API to logging. One though that allows you to use configurebility to a much deeper extent. For one, it can plug in the standard Python logger infrastructure, indirectly using the configurability and modularity of the familiar logging infrastructure. In many cases though, you can use the modularity of the twisted.python.log itself. For example, if you want to use GreyLog as logging infrastructure.

Here we have a sample of using twisted.python.log with the standard logger. Notice that the API for logging may be a bit different then what you are used to in Python. The API is still simple though, and I would like to promote using twisted.python.log over twisted.logger for the reason of flexability and configurability.

Now let us go back to the five pillars of the Twisted architecture. We already discussed Deferreds and the Reactor in part one of this tutorial series. Now it is time to dive into Transports and Protocols.

The above diagram gives us some insights into how Transports and Protocols fit together with the Reactor and the underlying operating system. A Protocol, either user defined or higher level protocols implemented in Twisted or in a 3rd party library, runs on top of a Transport. The transport in turn plugs into the Reactor we discussed in part one of this series.

From the user perspective, a Transport represents one end of a connection, any connection, between endpoints. A protocol, running on top of that, represents one side of an application level protocol. Finally, in Twisted, a ProtocolFactory is what creates a protocol at the moment a Transport level connection is established.

Let us start of with a trivial do-nothing server that does nothing but log when a new connection is made and when that connection is lost. Note that we subclass Protocol and provide a non-default implementation to two of its methods.

Given that our subclass doesn't have any constructor arguments, we can simply use the standard ServerFactory and not subclass it. We simply set its protocol attribute to our protocol. Finaly we bind the factory to our server endpoint when invoking listen.

Now let us make some changes to our do-nothing server. I know it is a matter of taste, but personally I like to inject my dependencies implicitly as constructor arguments so it is easy if we want to mock stuff later on. In this case our code uses
logging as dependency, so we inject the logger as dependency. We add an extra overloaded function for counting bytes received, and we update our class to count and log received bytes.

Now, because we have a constructor argument, we can no longer use the standard factory, so we need to subclass Factory and create our own.

Putting things together, our updated server could look something like this.

The other side of the connection, the client is pretty similar. Again we subclass Protocol, and in this case ClientFactory, and we overload a number of relevant methods. The above client sends a number of characters to the server on connection and then closes the connection.

Now it is time to shortly discuss the fifth and last of the five pillars of the Twisted architecture. The Application. By changing just a few lines in our python code, we change it from a self-sufficient Python script into a Twisted application.
A Twisted Application is started with the twistd tool.

So why care about Applications and twistd? Well, twistd does a bit more than just start the reactor. It takes care of daemonizing your server, having it run under the proper credentials, etc. Next to that, twistd also allows the command-line picking of the reactor and provides basic profiling facilities for your application.

So far we looked at Protocol, lets take a small step back and look at Transport.

Transport provides a generic interface to Protocol to work with. In the above slide we see the API each Transport provides us with.

For completeness and contrast, here is how our simple server would look with Python 3 (>3.5) asyncio. Notice how at its basis, despite somewhat different API's the end result code is actually quite similar.

Back again to Twisted and Transports. In our example we only looked at TCP as Transport, both from the server and from the client side of things. It falls outside of the scope of this tutorial to look at examples of all the different types of transports, but it is important to realize there are other types of transports you can basically just plug into. In Twisted itself we have UDP and Transport Layer Security as base transports. With third party libraries, things like TOR and I2P are also trivial to integrate in your Twisted based project.

Often it is important to implement things like time-outs and other types of timed-events into your asynchronous program. Let us compare a silly synchronous piece of code with its asynchronous counterpart implemented in Twisted.
Pay attention to the syntax of deferLatter.

As our example is supposed to end when done, notice the last function that is supposed to run calls reactor.stop(). Realize that in Twisted, a reactor, once stopped can not be started again.

So how about calling external programs? This may sound strange, but in Twisted, external programs simply fit into the Protocol regime.
We subclass ProcessProtocol and overload relevant methods as we did with our client and our server examples.

Here we have a trivial example of a Protocol for processing the output of involving the ps tool as external program.

So how about using threads in your asynchronous program. If you aren't used to working with threads in Python and are unfamiliar with Python's Global Interpreter Lock, this may come as a bit of a surprise, but for most purposes, using threads with Twisted is a rather bad idea. Most legitimate uses of threads in Python are exactly the type of operations that in Twisted and asyncio are handled by the Reactor or event loop. There basically are just two reasons why you should consider using threads in Twisted. The primary reason is when you wish to build an asynchronous wrapper for an API that only provides synchronous blocking functions.

There is one more potentially useful nugget of knowledge with respect to the use of threads in Python that might be relevant for your usage with Twisted. The numpy library will often release the Global Interpreter Lock when doing calculations on large arrays and matrices. If such calculations are part of your program, then using threads might also make sense from a performance perspective.

For our primary use case, the wrapping of a blocking function. Let us have a look at what Twisted provides for working with threads. The reactor provides us with two methods. The first method, callInThread is meant to be called from the main thread and will make that the supplied callable will be invoked while running in a different thread. The second one, callFromThread is meant to be called from within the other thread and will result in the supplied callable being invoked by the reactor in the main thread. No other reactor methods should ever be called from outside of the main thread.

Now how do we use these two methods to wrap a blocking function? To keep things simple let us start out with a simple example wrapping time.sleep and doing so without deferreds at first. In this example, time.sleep will be called in the main thread, but all invocations of print will be done from the main thread.

The next step is turning things into a real wrapper using deferreds. If you have problems reading the above code, please revisit part one and look at the part on deferreds.

So what if we actually want to do long-running CPU intensive tasks that don't use numpy, and as such are likely to lock up the GIL? For such situations, Twisted provides twisted.spread.pb. Look at a Perspective Broker as a remote object. You subclass pb.Root and any method you wish to expose to the client, you prefix with remote_. In this example server, we define a findkey method that does a number of long-running operations using a crypto library, in order to find a candidate password, knowing the first byte of the resulting key must be a zero. On a PC, this code will tke many seconds to complete.

Now for the client code. Have a look at the code and realize how the multiple levels of callbacks fit together. While the server will block on its CPU intensive calls, all operations calling the remote object from the client will be non-blocking asynchronous. This way we can keep the process with the pb client responsive while the server is doing a long running CPU intensive task.

In this installment of this multi-part Twisted tutorial we completed the low level stuff Twisted provides. In the upcomming part three we will be looking at a number of higher level protocols that Twisted provides, or that are available through third party libraries that work well with Twisted.

Curriculum

• Writing a simple STEEM statistics-script in Python with asyncsteem
• Writing an asynchronous STEEM web-app in Python with asyncsteem
• Building an asyncsteem python web app with micro-transaction-based steem user authentication, Redis and Jinja2
• Asynchonous Python with Twisted (and asyncio); Part One