managing multiple subprocesses

[Marcos]

Hi guys,
I realise this question has been answered in one form or another many
times before but I can't quite find the solution I need. I am trying to
run multiple subprocesses from a python script and then wait until all
subprocesses have completed before continuing. The subprocesses run on
other machines ie this is a coarse grained parallel computation task.

So the commands I am using are:
for x in range(0, limit):
os.system("xgrid .... ./someprogram %d &" % x)

and then to check whether they are all finished I do a very crude:

joblist = commands.getoutput("xgrid .... -job list")
while joblist != "{jobArray = (); }":
time.sleep(1)
joblist = commands.getoutput("xgrid .... -job list")

Xgrid is a system for parallel computation on Mac OS X. It is quite
good, check it out at www.apple.com/acg/xgrid. But anyway the method I
am using above, while it works, it quite often hangs. I'd like to just
run all the subprocesses at once and then check within the script,
rather than an auxilliary, to see if everything has returned. The
output of the commands is not important as that gets redirected to
files outside the script. I have tried all sorts of popens / excevs /
os.systems / commands etc etc. I realise the subprocess module may have
what I need but I can't get python 2.4 on the Mac so I need a 2.3 based
solution. Any help is much appreciated. Cheers.

 

[Skip Montanaro]

Marcos> I have tried all sorts of popens / excevs / os.systems /
Marcos> commands etc etc.

I think os.spawn* and os.wait will do what you want. I have trouble with
os.spawn* myself, so may just fall back to fork/exec + os.wait in the
parent.

Skip

 

[Donn Cave]

Quoth Skip Montanaro <[email protected]>:
|
| Marcos> I have tried all sorts of popens / excevs / os.systems /
| Marcos> commands etc etc.
|
| I think os.spawn* and os.wait will do what you want. I have trouble with
| os.spawn* myself, so may just fall back to fork/exec + os.wait in the
| parent.

That's probably the ticket. There are endless variations on how you
can use these functions (especially if you include pipe() and the dup
fcntls), and in a way it may be simpler to write your own variation
than work with a packaged one that does approximately what you need.
As long as it doesn't need to run on Windows.

By the way, I never ever use the *p versions of these functions, and
always specify the full path of the executable. I don't use the the
C library versions, and I don't use the Python versions. The latter
aren't actually C library function wrappers, but rather like I think
most shells they contrive to look through PATH themselves, and at any
rate it's difficult to deal with the lookup failure in a useful way
in the child fork. No doubt there are situations where a path lookup
is essential, but it just hasn't been happening to me.

Donn Cave, (e-mail address removed)

 

[Oren Tirosh]

...
os.systems / commands etc etc. I realise the subprocess module may have
what I need but I can't get python 2.4 on the Mac so I need a 2.3 based
solution. Any help is much appreciated. Cheers.

The Python 2.4 subprocess module by Peter Astrand is pure python for
posix systems (a small C extension module is required only for win32
systems). You can bundle it with your code and use it with older
Python versions. It even contains backward compatibility code to
replace True and False if not found so it can run on Python 2.2.

Oren

 

[Fredrik Lundh]

The Python 2.4 subprocess module by Peter Astrand is pure python for
posix systems (a small C extension module is required only for win32
systems). You can bundle it with your code and use it with older
Python versions. It even contains backward compatibility code to
replace True and False if not found so it can run on Python 2.2.

footnote: a stand-alone distribution is available here:

http://www.lysator.liu.se/~astrand/popen5/

direct link to the latest subprocess.py source:

http://tinyurl.com/3mmka

</F>