[Twisted-Python] freeing the reactor to do other jobs
exarkun at divmod.com
Fri Nov 7 08:59:23 EST 2008
On Fri, 7 Nov 2008 08:29:50 -0500, Jeff Dyke <jeff.dyke at gmail.com> wrote:
>I'm using the XMLRPC server in twisted and a few methods call other,
>sometimes long running, functions/methods. I'm trying to get my brain
>around how to free the reactor to respond to other requests while this
>A scenario. A call is made to the server, which selects say 10K rows
>from a db and needs to check each row against a table and if they do
>not exist, insert them.
>""" Oversimplified version of the process """
> rows = self.getUserData(user_id)
> for row in rows:
> if self.existsInQueue(row['some_id']):
>I want the caller to wait on a result from this process, but I also
>want the reactor to be able to handle other requests as they come in.
>This function is not directly registered in the xmlrpc server with
>xmlrpc_getData, but is called by that type of method after validation
>that it is allowed to run in this context.
>What i've seen when this has thousands of rows to process is that the
>reactor is tied up and can not respond to requests until complete.
>Which obviously leads to me believe that I'm not using twisted
>correctly/to its potential. I have read the deferred/asynchronous doc
>pages...but am having a hard time getting my head around it and would
>appreciate any advice.
>When i don't specifically need the caller to get the final result,
>i've been suing deferToThread, but feel in some of those instances i
>could possibly write better code, rather then sending to a thread.
One thing you might not have discovered yet is that even if you use
deferToThread, you can still give the final result to the caller. The
XML-RPC support in Twisted supports Deferreds - meaning that if an
xmlrpc_ method returns a Deferred, then no response is sent to the
XML-RPC request until that Deferred fires, and then the result is sent
as the XML-RPC response.
Since most libraries for interacting with RDBMs using SQL present a
blocking interface, Twisted includes twisted.internet.adbapi, a thin
layer on top of DB-API 2.0 which runs all of the blocking stuff in a
threadpool. If most of your time is being spent waiting for rows from
a database, then adbapi might help you out, and since adbapi gives you
Deferreds, this is trivial to integrate into an XML-RPC server.
For other blocking tasks - it depends. If the task is blocking on an
event, then transforming that event into a callback (probably using a
Deferred, since Deferreds are a good tool to use to manage callbacks)
and then putting your code into a callback instead of blocking on the
event is the right thing to do. How exactly you turn a particular
event into a callback depends on the details of the event, though. If
the blocking task is CPU bound, then running it in another thread or
another process can make sense. It's also possible to insert explicit
control-flow yields into the implementation of the CPU bound task (at
least, sometimes) so that the reactor can service other event sources
as the calculation progresses.
More information about the Twisted-Python