Imagine that a madman (we'll call him MadHatter) walks up to you and thrusts into your hand a FireCracker. The FireCracker's internal Fuse is burning and will explode the firework in the next few seconds. We will model this scenario using Asynchronous Python code. This sort of task is exactly what Twisted excels at, and we will write a Twisted implementation of the scenario at the end of the article. Before that, however, we will examine what components the task makes necessary in a general async event manager. To this end we shall create our own event manager called Fate (don't worry, this will be very bare bones to fit in a few lines of code, yet suitable for the task we have set it). You may have noticed that the pieces of the system I have described so far have names fitting into a real-life analogy. This is a device to aid memory and understanding; I will make clear as we consider each part whether it is specific to our scenario, or applicable to a more general class of circumstances. Hopefully the choice of naming will be helpful - Twisted also has interesting names, but ones which strangely have very little to do with the function of their bearers (see Conch, Manhole and Jelly). Enter the actors:
We've got this far using only the 'standard' everyday sort of Python. No async magic. Now, why is an event manager useful? Why not simply continue in this vein? Well, we want to be able to simulate the burning fuse. Let's add a light_fuse() method to FireCracker. In naïve, synchronous Python we could do something like this:
Do not do this! Why? There are two obvious issues:
While burning the FireCracker() cannot be handed to anyone else. The MadHatter()'s dark designs will never be realized because directly after lighting the fuse the processor becomes completely taken up with sleeping. The path of execution effectively sits at the time.sleep() statement for several seconds, after which it moves to the next line, exploding in the lighter's own hands. There is simply no way to do anything between the two consecutive lines of code.
Secondly, if you have more than one FireCracker(), or indeed anything else in your model, it will all unhelpfully freeze for several seconds. Only one thing can possibly be happening at a particular time. This is the fundamental limitation of traditional synchronous programming.
We shall now find a way around this by writing Fate() async code to use it. In general, asynchronous programming could be summarized by replacing any blocking operation (e.g. sleeping, waiting for data on sockets, waiting or external command to return) with special event-handling commands. By writing Fate, we will see that the blocking, clunky operations are in fact moved back behind the scenes into the event framework, which intelligently combines them to give the illusion of simultaneous happenings. They are merely being 'switched between'. (Incidentally, this is similar to the way that modern operating systems allow you to run multiple applications at once by rapidly switching between them behind-the-scenes).
On to Fate. We have to provide a way for the developer working on the FireCracker code to 'set' events. The simplest method of doing this is perhaps timer-based events. Lets make a Fuse class that represents a timed event. Just follow along, you'll see precisely how it fits together with everything else in a bit.
So, this is simple. We get a fuse, set the burning time and then we can check in the future whether or not it is fully burned. It is a trifle whether to use '<' or '<=' - it is incredibly unlikely that this will make any difference in your Python implementation, due to the high precision of the Standard Library's time() call. I have opted for the nicer-looking version, avoiding the perhaps philosophical issue of subdivision of time. Now, the way that this will be incorporated into our code is as follows:
We will create Fate and add to it a get_fuse() method. This will take a single argument of the (floating point) number of seconds representing burning time. It will add this to the current time, giving the absolute time at which the fuse will be fully burned, creates a Fuse instance and sets the time on it. It saves this to internal instance memory and returns it.
We will add a lightfuse() method to FireCracker. This will get a Fuse using Fate.getfuse() and tell it what to do when the fuse is fully burned. It will do this by simply calling its set_handler() method with self.explode. We are telling Fate how to handle the fuse being completely burned.
But wait. Fuse doesn't have a set_handler() method. We're going to fix that presently, it was simply more convenient to present the material in this logical order. The following is the whole of Fate, including the slightly expanded Fuse class. It will be explained afterwards. Try to glean the rough functionality from the source.
Whew! Now the explanation:
Now, we'll fill in the bits of the FireCracker scenario to make use of Fate. This will be a good example of how to use Fate for other applications, and, more generally, is illustrative of a standard asynchronous design pattern. Remember that we're replacing the bad, synchronous version one of our light_fuse() method with a better Fate-ful one.
Here is the complete FireCracker program, minus general-purpose Fate code and imports.
Now, our Fate system functions well at the limited tasks set out for it. It will not do any of the following things:
Let's see a quick second example that takes Fate to the limits of its functionality.
Here we have represented a chain of fuses by putting in the "user" code a chaining mechanism: the handler for one Fuse() i.e. ignite() itself creates another fuse. Try following through the "logical flow" of execution through the program. It can be more difficult to follow this than standard, synchronous code, but we gain an advantage when working on any non-trivial program that must accomplish multiple tasks in some sort of concurrency. Now that we've written an event handler and two examples together, its time to introduce Twisted. It is similar to Fate, but much more extensible. Twisted also contains many utility modules for doing things like HTTP downloading and executing external commands.
To get started with Twisted, remember these two approximations:
Here is a simple Twisted program, adapted from the Twisted Docs: from twisted.internet import reactor, defer
So, similarly to our Fuse, a Deferred is Twisted's indication that the returner will fire it at some point in the future. The slow-multiply function returns the Deferred instantly, but uses Twisted's callLater function to call the "callback" method of its Deferred in 3 seconds. This simulates a slow multiplier. Instead of adding a "handler", we add a "callback" function to the Deferred. In this case it's printData() which as to deal with whatever value the Deferred fires with. With these small difference in mind, it ought to be easy for you to rewrite the Mad Hatter example using Twisted. Here is is:
We've made this one more interesting: it may explode in the MadHatter's own hands. The single-letter "d" is commonly used to point to a Deferred object within Twisted programs. This convention can make it easier to keep clear which returned values are actual data (i.e. non-Deferred objects) and which are simply "promises of data" (i.e. Deferreds).
The above should be enough to get you started working within the Twisted asynchronous framework, and you should have a reasonably clear idea of how events fit together. With such programming, you have the advantage of not having to worry about simultaneous variable access and other issues associated with threaded programming, but you have the additional responsibility of making your functions return fast (no "block" for long periods of time). Looking at Fate its clear why this would be a problem: your code would stop the event handler looking for other events that need to happen. It is for this reason that the other parts of the Twisted framework become desirable. In synchronous programming, one might use
to read a web page. This would be bad in Twisted, because while the standard library is occupied with downloading it (which might take several seconds), anything else that ought to run couldn't. The correct Twisted way to do this is:
Internally, Twisted's getPage function downloads small chunks of the page at a time, allowing other events to be handled during downloading. Two important topics not covered here for reasons of brevity are errbacks and deferred chaining. Errbacks are the counterpart to callbacks that are meant to be triggered in the event of something going wrong in a function. They are similar to exceptions, and can be handled with the addErrback method of a Deferred. Chaining is a way to link Deferreds "end-to-end" in a chain, so that one will only fire after another does. This allows us to do the "line-by-line" execution that is given to us for free in synchronous programming. Also useful is the inlineCallbacks generator, which performs a similar function. As a closing remark, it is important to choose correctly when asynchronous code is necessary and when it is not. For simple shell-replacement Python scripts it is clearly an overkill, but for applications performing multiple network requests or handling user input while doing background work Twisted is probably the answer.