parallel and/or synchronous start/run/stop on multiple boxes

[Shane]

Consider a network of 3 fully-connected boxes i.e. every box as a TCP-
IP connection to every other box.

Suppose you start a python program P on box A. Is there a Python
mechanism for P to send a copy of itself to box B or C then start that
program P on B or C by running a method p in P? Is there a way that P
on A could wait for that result?

Bottom line: we have a number of needs in which it'd very convienent
to write a phython program, install and run it, so that it could
automatically propogate itself to wherever the code needs to go and
run itself so that there could be multiple copies of the code running
on different boxes with synchrounous (maybe asychronous)
communication.

Shane

 

[James Mills]

Consider a network of 3 fully-connected boxes i.e. every box as a TCP-
IP connection to every other box.

Suppose you start a python program P on box A. Is there a Python
mechanism for P to send a copy of itself to box B or C then start that
program P on B or C by running a method p in P? Is there a way that P
on A could wait for that result?

No, python provides no such mechanism. However,
I would highly recommend you take a look at circuits (1)
for this task - as it's been especially developed for
distributed processing in the manner you describe above.

In order for your program to propagate itself, you would
have to rely on ssh (recommended). There are several
good ssh libraries/clients for python, and pexpect might
come in handy as well.

cheers
James

1. http://trac.softcircuit.com.au/circuits/

 

[Ned Deily]

No, python provides no such mechanism.

The multiprocessing module, new in the 2.6 standard library and
available in PyPi as a backport to 2.4 and 2.5, supports managing of
processes on both local and remote machines. The 2.6 module
documentation has an "example/demo of how to use the
managers.SyncManager, Process and others to build a system which can
distribute processes and work via a distributed queue to a 'cluster' of
machines on a network, accessible via SSH".

<http://docs.python.org/library/multiprocessing.html>

 

[James Mills]

The multiprocessing module, new in the 2.6 standard library and
available in PyPi as a backport to 2.4 and 2.5, supports managing of
processes on both local and remote machines. The 2.6 module
documentation has an "example/demo of how to use the
managers.SyncManager, Process and others to build a system which can
distribute processes and work via a distributed queue to a 'cluster' of
machines on a network, accessible via SSH".

Wow! I didn't know that! It seems the multiprocessing
library packs quite a punch!

cheers
James

 

[MRAB]

Consider a network of 3 fully-connected boxes i.e. every box as a TCP-
IP connection to every other box.

Suppose you start a python program P on box A. Is there a Python
mechanism for P to send a copy of itself to box B or C then start that
program P on B or C by running a method p in P? Is there a way that P
on A could wait for that result?

Bottom line: we have a number of needs in which it'd very convienent
to write a phython program, install and run it, so that it could
automatically propogate itself to wherever the code needs to go and
run itself so that there could be multiple copies of the code running
on different boxes with synchrounous (maybe asychronous)
communication.

A sort of Pythonic virus?