Async XMLRPC and job processing

[Sean Davis]

I would like to set up a server that takes XMLRPC requests and
processes them asynchronously. The XMLRPC server part is trivial in
python. The job processing part is the part that I am having trouble
with. I have been looking at how to use threadpool, but I can't see
how to get that working. I would like to have the XMLRPC part of
things do something like:

def method1(a,b,c):
jobid=workRequest(long_method1,[a,b,c])
return(jobid)

def method2(a,b,c):
jobid=workRequest(long_method2,[a,b,c])
return(jobid)

def long_method1(a,b,c)
do lots of heavy computation, etc.
store results in files in a given directory, etc
return result

..... for any number of methods

Again, pretty straightforward. However, I run into problems with the
threadpool and xmlrpc server both waiting. In particular, if I do
something like:

server = SimpleXMLRPCServer.SimpleXMLRPCServer(.....)
server.serve_forever()

Where can tell the threadpool that I have set up to wait
indefinitely? Both are blocking.

Thanks,
Sean

 

[Adonis Vargas]

I would like to set up a server that takes XMLRPC requests and
processes them asynchronously. The XMLRPC server part is trivial in
python. The job processing part is the part that I am having trouble
with. I have been looking at how to use threadpool, but I can't see
how to get that working. I would like to have the XMLRPC part of
things do something like:

def method1(a,b,c):
jobid=workRequest(long_method1,[a,b,c])
return(jobid)

def method2(a,b,c):
jobid=workRequest(long_method2,[a,b,c])
return(jobid)

def long_method1(a,b,c)
do lots of heavy computation, etc.
store results in files in a given directory, etc
return result

.... for any number of methods

Again, pretty straightforward. However, I run into problems with the
threadpool and xmlrpc server both waiting. In particular, if I do
something like:

server = SimpleXMLRPCServer.SimpleXMLRPCServer(.....)
server.serve_forever()

Where can tell the threadpool that I have set up to wait
indefinitely? Both are blocking.

Thanks,
Sean

This site shows how to make it multi-threaded:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/425043

But now you have to contend with the integrity of your data, provided
your going be storing data through these processes. You may want to look
into the Queue module to create a sort of message queue so your data can
be properly synchronized.

Hope this helps.

Adonis