Could you recommend job schedulling solution?

[redbaron]

I've sinlge 8-way node dedicated for executing long running tasks. To
be able to execute multiple tasks on this node it shoud spawn each
task in another process. At the same time it should accept network
connection with new tasks without blocking of client and put it on job
queue.

What is "task" ? Executing just ordinary python function will be
enough. If solution contain some client library which allow easy task
submit it will be great.

 

[Martin]

Hi,

should accept network
connection with new tasks without blocking of client and put it on job
queue.

What is "task" ? Executing just ordinary python function will be
enough. If solution contain some client library which allow easy task
submit it will be great.

I think parallel python will take of that for you
(http://www.parallelpython.com/)

--
http://soup.alt.delete.co.at
http://www.xing.com/profile/Martin_Marcher
http://www.linkedin.com/in/martinmarcher

You are not free to read this message,
by doing so, you have violated my licence
and are required to urinate publicly. Thank you.

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html

 

[redbaron]

I think parallel python will take of that for you
(http://www.parallelpython.com/)

I've found that RPyC (http://rpyc.wikidot.com/) is quite usefull for
my task. It allows me to build RPC service which accepts ordinary
python function from client and return result in synchronous or
asynchronous way. Of course it is able to serve multiple clients and
it's forking server help me to solve GIL problems on intense
calculations inside Python code.

Some limitations are present, like the fact that you couldn't send on
execution any code which use C extensions which are not present on RPC
server, but I didn't expect this could work at all, so in general RPyC
makes me happy.