spawning pyhon apps...

[bruce]

hi...

toying with an idea.. trying to figure out a good/best way to spawn multiple
python scripts from a parent python app. i'm trying to figure out how to
determine when all child apps have completed, or to possibly determine if
any of the child processes have died/halted..

parent app
spawn child1
spawn child2
spawn child3
.
.
.
spawn childn

do i iterate through a os.waitpid(pid) for each pid of the child processes i
create?

is there another approach? code samples/tutorial...??

i've seen various approaches via google, but not just what i'm looking for..

thanks

 

[Jason Scheirer]

hi...

toying with an idea.. trying to figure out a good/best way to spawn multiple
python scripts from a parent python app. i'm trying to figure out how to
determine when all child apps have completed, or to possibly determine if
any of the child processes have died/halted..

parent app
 spawn child1
 spawn child2
 spawn child3
 .
 .
 .
 spawn childn

do i iterate through a os.waitpid(pid) for each pid of the child processes i
create?

is there another approach? code samples/tutorial...??

i've seen various approaches via google, but not just what i'm looking for..

thanks

Investigate the subprocess module, you probably want Popen objects.
You can do a poll loop like

my_popenobjects = [subprocess.Popen("foo.py", "--filename=file
%i.txt"%x) for x in xrange(10)]

while any(popenobject.statuscode is None for popenobject in
my_popenobjects):
time.sleep(0.25)

If your tasks are more like function calls and less like shell
scripts, then investigate writing your python as an importable module
and use the multiprocessing module, which will do threading/
subprocessing for you.

 

[Chris Rebert]

hi jason....

forgive me... but in your sample:
my_popenobjects = [subprocess.Popen("foo.py", "--filename=file
%i.txt"%x) for x in xrange(10)]
are you spawning 'foo.py' 10 times? that can't be right!

Indeed, it probably ought to be (note the 2nd pair of brackets):
my_popenobjects = [subprocess.Popen(["foo.py",
"--filename=file%i.txt"%x]) for x in xrange(10)]

so just what is "foo.py" used for? what am i missing...

It's the name of the program you want to run. In this case, it happens
to be a Python script.

it looks like the my_popenobjects array is iterated through to check the
statuscode. is the statuscode the value that would be returned from a child
python script via something like "return(2)"....

It's the POSIX exit code of the program (i.e. what int you return from
main() in a C program, or what you pass to sys.exit() in Python)

i've seen mention of os.waitpid(..) does this play into waiting for child
processes to complete, or determine if they've terminated??

Don't know specifically, but that's another, lower-level API. The
`subprocess` module is superior.

Cheers,
Chris

 

[Mark Wooding]

toying with an idea.. trying to figure out a good/best way to spawn
multiple python scripts from a parent python app. i'm trying to figure
out how to determine when all child apps have completed, or to
possibly determine if any of the child processes have died/halted..

You don't say what platform you're using, but you mention os.waitpid so
I'll randomly assume it's Unix-like.

do i iterate through a os.waitpid(pid) for each pid of the child
processes i create?

That will technically work, and give you the information you wanted,
though not necessarily in the most timely fashion. It'll block on each
process in turn, waiting for its exit status -- so it'll finish as soon
as all the children are dead, but if (say) the fifth process dies first,
you won't find out until the first four have also passed on.

If you don't have anything better for your program to do, and you're
really on Unix, you can call

kid, status = os.waitpid(0, 0)

to wait for something to happen to any of your process's children; the
kid is the process-id of the child being reported and the status is what
happened to it.

If you do have other things for your process to be doing, then your best
bet is to establish a signal handler for SIGCHLD and installing a
handler of the form

import signal as S
import os as OS
import errno as E

## Children sometimes die. It's sad.
def sigchld(sig, frame):
try:
while True:
kid, status = OS.waitpid(0, OS.WNOHANG)
if kid == 0:
break
## Handle death of KID here.
except OSError, err:
if err.errno != E.ECHILD:
raise

### ...

## Establish handler.
S.signal(S.SIGCHLD, sigchld)

should work.

If you're running on Windows then these tricks won't work. As a grim
hack, you could start a thread per child process and have each thread
wait for its own child (sending the exit status through a queue or
something). I'm afraid I don't know Windows well enough to offer
slicker solutions; maybe someone else can help.

-- [mdw]