spawning pyhon apps...

Discussion in 'Python' started by bruce, Jan 9, 2009.

  1. bruce

    bruce Guest

    hi...

    toying with an idea.. trying to figure out a good/best way to spawn multiple
    python scripts from a parent python app. i'm trying to figure out how to
    determine when all child apps have completed, or to possibly determine if
    any of the child processes have died/halted..

    parent app
    spawn child1
    spawn child2
    spawn child3
    .
    .
    .
    spawn childn

    do i iterate through a os.waitpid(pid) for each pid of the child processes i
    create?

    is there another approach? code samples/tutorial...??

    i've seen various approaches via google, but not just what i'm looking for..

    thanks
     
    bruce, Jan 9, 2009
    #1
    1. Advertising

  2. On Jan 9, 2:47 pm, "bruce" <> wrote:
    > hi...
    >
    > toying with an idea.. trying to figure out a good/best way to spawn multiple
    > python scripts from a parent python app. i'm trying to figure out how to
    > determine when all child apps have completed, or to possibly determine if
    > any of the child processes have died/halted..
    >
    > parent app
    >  spawn child1
    >  spawn child2
    >  spawn child3
    >  .
    >  .
    >  .
    >  spawn childn
    >
    > do i iterate through a os.waitpid(pid) for each pid of the child processes i
    > create?
    >
    > is there another approach? code samples/tutorial...??
    >
    > i've seen various approaches via google, but not just what i'm looking for..
    >
    > thanks


    Investigate the subprocess module, you probably want Popen objects.
    You can do a poll loop like

    my_popenobjects = [subprocess.Popen("foo.py", "--filename=file
    %i.txt"%x) for x in xrange(10)]

    while any(popenobject.statuscode is None for popenobject in
    my_popenobjects):
    time.sleep(0.25)

    If your tasks are more like function calls and less like shell
    scripts, then investigate writing your python as an importable module
    and use the multiprocessing module, which will do threading/
    subprocessing for you.
     
    Jason Scheirer, Jan 9, 2009
    #2
    1. Advertising

  3. bruce

    Chris Rebert Guest

    On Fri, Jan 9, 2009 at 3:43 PM, bruce <> wrote:
    > hi jason....
    >
    > forgive me... but in your sample:
    > my_popenobjects = [subprocess.Popen("foo.py", "--filename=file
    > %i.txt"%x) for x in xrange(10)]
    > are you spawning 'foo.py' 10 times? that can't be right!


    Indeed, it probably ought to be (note the 2nd pair of brackets):
    my_popenobjects = [subprocess.Popen(["foo.py",
    "--filename=file%i.txt"%x]) for x in xrange(10)]

    > so just what is "foo.py" used for? what am i missing...


    It's the name of the program you want to run. In this case, it happens
    to be a Python script.

    >
    > it looks like the my_popenobjects array is iterated through to check the
    > statuscode. is the statuscode the value that would be returned from a child
    > python script via something like "return(2)"....


    It's the POSIX exit code of the program (i.e. what int you return from
    main() in a C program, or what you pass to sys.exit() in Python)

    > i've seen mention of os.waitpid(..) does this play into waiting for child
    > processes to complete, or determine if they've terminated??


    Don't know specifically, but that's another, lower-level API. The
    `subprocess` module is superior.

    Cheers,
    Chris

    --
    Follow the path of the Iguana...
    http://rebertia.com
     
    Chris Rebert, Jan 9, 2009
    #3
  4. bruce

    Mark Wooding Guest

    bruce <> wrote:

    > toying with an idea.. trying to figure out a good/best way to spawn
    > multiple python scripts from a parent python app. i'm trying to figure
    > out how to determine when all child apps have completed, or to
    > possibly determine if any of the child processes have died/halted..


    You don't say what platform you're using, but you mention os.waitpid so
    I'll randomly assume it's Unix-like.

    > do i iterate through a os.waitpid(pid) for each pid of the child
    > processes i create?


    That will technically work, and give you the information you wanted,
    though not necessarily in the most timely fashion. It'll block on each
    process in turn, waiting for its exit status -- so it'll finish as soon
    as all the children are dead, but if (say) the fifth process dies first,
    you won't find out until the first four have also passed on.

    If you don't have anything better for your program to do, and you're
    really on Unix, you can call

    kid, status = os.waitpid(0, 0)

    to wait for something to happen to any of your process's children; the
    kid is the process-id of the child being reported and the status is what
    happened to it.

    If you do have other things for your process to be doing, then your best
    bet is to establish a signal handler for SIGCHLD and installing a
    handler of the form

    import signal as S
    import os as OS
    import errno as E

    ## Children sometimes die. It's sad.
    def sigchld(sig, frame):
    try:
    while True:
    kid, status = OS.waitpid(0, OS.WNOHANG)
    if kid == 0:
    break
    ## Handle death of KID here.
    except OSError, err:
    if err.errno != E.ECHILD:
    raise

    ### ...

    ## Establish handler.
    S.signal(S.SIGCHLD, sigchld)

    should work.

    If you're running on Windows then these tricks won't work. As a grim
    hack, you could start a thread per child process and have each thread
    wait for its own child (sending the exit status through a queue or
    something). I'm afraid I don't know Windows well enough to offer
    slicker solutions; maybe someone else can help.

    -- [mdw]
     
    Mark Wooding, Jan 10, 2009
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jim
    Replies:
    0
    Views:
    351
  2. Stewart Midwinter

    Simple pyhon-based CMS

    Stewart Midwinter, Apr 14, 2004, in forum: Python
    Replies:
    4
    Views:
    435
    Mike C. Fletcher
    Apr 16, 2004
  3. aph
    Replies:
    4
    Views:
    439
    Peter Hansen
    Jan 14, 2006
  4. Robin Becker

    pyhon 2.5.1 build fails

    Robin Becker, Jun 4, 2007, in forum: Python
    Replies:
    0
    Views:
    302
    Robin Becker
    Jun 4, 2007
  5. Evan
    Replies:
    10
    Views:
    541
Loading...

Share This Page