Advice needed on parallel processing in python

[Glazner]

Hi!

I need to replace an app that does number crunching over a local
network.
it have about 50 computers as slaves
each computer needs to run COM that will do the "job"
right now the system uses MFC threads and DCOM to distribute the load.

as i said, Now i'm trying to replace this system with python.
I already use win32all and read about Pareller Python and Pyro.

a requirement is that messages can be sent to the remote worker.

If anyone can share experience I'll love to hear


Many Thanks,
Yoav Glazner

 

[Jon Clements]

Hi!

I need to replace an app that does number crunching over a local
network.
it have about 50 computers as slaves
each computer needs to run COM that will do the "job"
right now the system uses MFC threads and DCOM to distribute the load.

as i said, Now i'm trying to replace this system with python.
I already use win32all and read about Pareller Python and Pyro.

a requirement is that messages can be sent to the remote worker.

If anyone can share experience I'll love to hear

Many Thanks,
Yoav Glazner

Would Celery suit? http://celeryproject.org/

hth

Jon.

 

[bobicanprogram]

Would Celery suit?http://celeryproject.org/

hth

Jon.

If you can add just one Linux server onto that network you could use
SIMPL-Python to do what you want.

http://www.icanprogram.com/06py/lesson1/lesson1.html

ie. SIMPL messages going seamlessly from Python (Windows) to Python
(Windows) with the SIMPL sandbox being hosted on the Linux node.

Of course if you wanted to add Linux nodes as processing nodes.
Python (Windows) modules could seamlessly exchange messages with those
as well. Same would be true if you wanted to distribute part of your
app into the cloud. All your Python code would look virtually
identical in each of these cases.

As an example of an inexpensive Linux node the SIMPL toolkit has been
ported to the ~$100 Sheeva Plug computer (http://www.icanprogram.com/
simpl/plugsimplbin.self.html) (http://www.plugcomputer.org).

bob