managing multiple subprocesses

Discussion in 'Python' started by Marcos, Feb 3, 2005.

  1. Marcos

    Marcos Guest

    Hi guys,
    I realise this question has been answered in one form or another many
    times before but I can't quite find the solution I need. I am trying to
    run multiple subprocesses from a python script and then wait until all
    subprocesses have completed before continuing. The subprocesses run on
    other machines ie this is a coarse grained parallel computation task.

    So the commands I am using are:
    for x in range(0, limit):
    os.system("xgrid .... ./someprogram %d &" % x)

    and then to check whether they are all finished I do a very crude:

    joblist = commands.getoutput("xgrid .... -job list")
    while joblist != "{jobArray = (); }":
    time.sleep(1)
    joblist = commands.getoutput("xgrid .... -job list")

    Xgrid is a system for parallel computation on Mac OS X. It is quite
    good, check it out at www.apple.com/acg/xgrid. But anyway the method I
    am using above, while it works, it quite often hangs. I'd like to just
    run all the subprocesses at once and then check within the script,
    rather than an auxilliary, to see if everything has returned. The
    output of the commands is not important as that gets redirected to
    files outside the script. I have tried all sorts of popens / excevs /
    os.systems / commands etc etc. I realise the subprocess module may have
    what I need but I can't get python 2.4 on the Mac so I need a 2.3 based
    solution. Any help is much appreciated. Cheers.
     
    Marcos, Feb 3, 2005
    #1
    1. Advertising

  2. >>>>> "Marcos" == Marcos <> writes:

    Marcos> I have tried all sorts of popens / excevs / os.systems /
    Marcos> commands etc etc.

    I think os.spawn* and os.wait will do what you want. I have trouble with
    os.spawn* myself, so may just fall back to fork/exec + os.wait in the
    parent.

    Skip
     
    Skip Montanaro, Feb 3, 2005
    #2
    1. Advertising

  3. Marcos

    Donn Cave Guest

    Quoth Skip Montanaro <>:
    | >>>>> "Marcos" == Marcos <> writes:
    |
    | Marcos> I have tried all sorts of popens / excevs / os.systems /
    | Marcos> commands etc etc.
    |
    | I think os.spawn* and os.wait will do what you want. I have trouble with
    | os.spawn* myself, so may just fall back to fork/exec + os.wait in the
    | parent.

    That's probably the ticket. There are endless variations on how you
    can use these functions (especially if you include pipe() and the dup
    fcntls), and in a way it may be simpler to write your own variation
    than work with a packaged one that does approximately what you need.
    As long as it doesn't need to run on Windows.

    By the way, I never ever use the *p versions of these functions, and
    always specify the full path of the executable. I don't use the the
    C library versions, and I don't use the Python versions. The latter
    aren't actually C library function wrappers, but rather like I think
    most shells they contrive to look through PATH themselves, and at any
    rate it's difficult to deal with the lookup failure in a useful way
    in the child fork. No doubt there are situations where a path lookup
    is essential, but it just hasn't been happening to me.

    Donn Cave,
     
    Donn Cave, Feb 4, 2005
    #3
  4. Marcos

    Oren Tirosh Guest

    "Marcos" <> wrote in message news:<>...
    > ...
    > os.systems / commands etc etc. I realise the subprocess module may have
    > what I need but I can't get python 2.4 on the Mac so I need a 2.3 based
    > solution. Any help is much appreciated. Cheers.


    The Python 2.4 subprocess module by Peter Astrand is pure python for
    posix systems (a small C extension module is required only for win32
    systems). You can bundle it with your code and use it with older
    Python versions. It even contains backward compatibility code to
    replace True and False if not found so it can run on Python 2.2.

    Oren
     
    Oren Tirosh, Feb 7, 2005
    #4
  5. Oren Tirosh wrote:

    >> os.systems / commands etc etc. I realise the subprocess module may have
    >> what I need but I can't get python 2.4 on the Mac so I need a 2.3 based
    >> solution. Any help is much appreciated. Cheers.

    >
    > The Python 2.4 subprocess module by Peter Astrand is pure python for
    > posix systems (a small C extension module is required only for win32
    > systems). You can bundle it with your code and use it with older
    > Python versions. It even contains backward compatibility code to
    > replace True and False if not found so it can run on Python 2.2.


    footnote: a stand-alone distribution is available here:

    http://www.lysator.liu.se/~astrand/popen5/

    direct link to the latest subprocess.py source:

    http://tinyurl.com/3mmka

    </F>
     
    Fredrik Lundh, Feb 7, 2005
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jane Austine
    Replies:
    4
    Views:
    459
    Colin Brown
    Nov 13, 2003
  2. Fuzzyman
    Replies:
    0
    Views:
    365
    Fuzzyman
    Dec 22, 2003
  3. Marcos

    managing multiple subprocesses

    Marcos, Feb 3, 2005, in forum: Python
    Replies:
    0
    Views:
    270
    Marcos
    Feb 3, 2005
  4. cypher543

    Managing a queue of subprocesses?

    cypher543, Dec 30, 2006, in forum: Python
    Replies:
    3
    Views:
    303
    Tom Plunket
    Jan 1, 2007
  5. JB
    Replies:
    1
    Views:
    223
    Jean-Michel Pichavant
    Jan 27, 2011
Loading...

Share This Page