Deferred: all the details

previous article has been described the basic principles of Deferred and its application in asynchronous programming. Today we will try to examine in detail the functioning of the Deferred and examples of its use.

So Deferred is a deferred result, the execution result, which will be known after some time. The result is stored in a Deferred, may be an arbitrary value (successful execution) or an error (exception) that occurred in the process of performing an asynchronous operation. Times we are interested in the result of the and operation we have received from some async functions Deferred, we want to perform actions in the moment when the execution result is known. Therefore, Deferred in addition to the result still keeps a chain of handlers: the result handlers (callback) and failure handlers (errback).

Let us consider in detail a chain of handlers:



Handlers are placed in “layers” or levels, the execution is clearly on levels from the top down. At each level there is a callback and errback handlers, one element may be missing. Each level can be done either callback or errback, but not both. The execution of the handler occurs only once, re-entry can not be.

Handlers are callback functions that takes one argument — the result of:

the

def callback(result):
...

Function errback handlers take as a parameter an exception wrapped in class Failure:

the

def errback(failure):
...

Deferred execution starts with the fact that Deferred, you receive result: success or exception. Depending on the result, selects the corresponding branch handler: callback or errback. This is followed by finding the nearest level handler, which has a corresponding handler. In our example in the figure was obtained by a successful execution result and the result was passed to the handler callback1.

Further execution causes the call handlers at lower levels. If callback or errback fails with a return value that is not a Failure, the execution is considered successful and the result is sent to the input callback-handler at the next level. If the process execution handler is thrown exception or returned a value of type Failure, the control will be passed to errback at the next level, which will receive the exception as a parameter.

In our example the handler callback1 returns successfully, its result was passed to the handler callback2, in which the exception was thrown, which led to the transition to the chain errback handlers on the third level, the errback handler and no exception was passed to errback4 that processed the exception, returned successful result of the execution, which now is the result of Deferred, but no more handlers. If Deferred is added another layer of handlers, they will be able to access this result.

Like all other Python objects, the object lives until the Deferred until he has references from other objects. Typically the object returned Deferred, saves it, because it is necessary for the asynchronous operation completes to transfer to Deferred the result. Often other members (add event handlers) do not retain references to Deferred, therefore, the Deferred object will be destroyed at the end of the handler chain. If there is a destruction Deferred, which remained unhandled exception (execution ended with an exception and no more handlers), the screen prints a debug message with a traceback of the exception. This situation is similar to “bounce” an unhandled exception on top-level in a synchronous program.
the

Deferred squared

Return value of callback and errback can also be another Deferred, then the execution of the handler chain of the current Deferred is suspended until the end of the handler chain nested Deferred.



In the above figure example, the handler returns callback2 is not the usual result, and the other Deferred — Deferred2. The execution of the current Deferred is suspended until the result of executing Deferred2. The result Deferred2 successful or the exception becomes the result passed to the next level handlers in the first Deferred. In our example, Deferred2 ended with an exception, which will be transferred to the handler errback2 Deferred.

the

exception Handling in errback

Each exception handler errback is analogous to a try..except block and the except block is usually responds to certain type of exceptions, this behavior is very easy to play with Failure:

the

def errback(failure):
"""
@param failure: an error (exception), wrapped in failure
@type failure: C{Failure}
"""
failure.trap(KeyError)

print "Got key error: %r" % failure.value

return 0

The method trap of class Failure checks whether the wrapped in it with the exception of the heir or directly by the class KeyError. If not, the original exception is thrown again, interrupting the execution of the current errback that will lead to the next errback in the chain of handlers that mimics the behavior of except when type mismatch exception (the control passes to the next block). In the property value is stored the original exception, which can be used for more information about the error.

It is necessary to note that the errback handler to be completed in one of two ways:

the
1. to Return a value, which will be the input value of the next callback that indicates that the exception was handled.
the
2. Throw the original or new exception — exception was not treated or was perevyasko a new exception, the errback handler chain continues.

There is a third option — to return a Deferred, then further execution of the processors will depend on the result of the Deferred.

In our example, we the exception is processed and communicated as a result of 0 (for example, the lack of some key equivalent to its zero value).

the

Preparing for async in advance

As soon as asynchrony, that is, some function instead of immediate values returns a Deferred, asynchronous begins to spread in the feature tree above, forcing to return a Deferred from a function, which used to be synchronous (return the result directly). Consider a hypothetical example of such a transformation:

the

def f():
return 33

def g():
return f()*2

If for some reason the function f cannot return the result immediately, it will return Deferred

the

def f():
return Deferred().callback(33)

But now the function g is forced to return Deferred, catching the chain of handlers:

the

def g():
return f().addCallback(lambda result: result*2)

A similar scheme of “conversion” happens with real functions, we obtain results in the form of Deferred from the underlying tree of function calls, hang them on your Deferred callback handlers that correspond to the old synchronous code our function, if we have exception handlers, are added, and the errback handlers.

In practice, it is better to first identify the places in code that will be asynchronous and will use Deferred than convert synchronous code into asynchronous. Asynchronous code starts with those challenges which can't build the result directly:
the

the
• network I / o;
the
• remote calls RPC;
the
• operations, the implementation of which will be allocated to the thread in the model Worker, etc.

In the process of writing applications is often clear that at this point the asynchronous call, but it's not (not implement an interface with the DBMS, for example). In such a situation, you can use the functions defer.success or defer.fail to create a Deferred which already contains the result. That's as short as possible to rewrite the function f:

the

from twisted.internet import defer

def f():
return defer.success(33)

If we do not know whether the called function to return the result synchronously or Deferred return, and do not want to depend on her behavior, we can wrap it in defer.maybeDeferred that is either option will do the equivalent of Deferred call:

the

from twisted.internet import defer

def g():
return defer.maybeDeferred(f).addCallback(lambda result: result*2)

This version of the function g will work with both synchronous and asynchronous f.

the

in conclusion

To talk about "Deferred" for a long time, as additional reading I recommend the list of materials at the end previous article.

Article based on information from habrahabr.ru