Executing multiple subprocesses and waiting

Discussion in 'Python' started by JB, Jan 27, 2011.

  1. JB

    JB Guest

    One of my python scripts that takes a bunch of inputs from a tKinter
    gui, generates a set of command line stings, and then threads them off
    to subprocess for calls to other programs like Nuke and our render
    farm has recently started randomly crashing pythonw.exe.

    I'm taking a look at my threading setup and attempting to clean it up.
    I was wondering what a smart way of doing what I describe is? Take a
    list of strings containing command line calls to other programs,
    process them one at a time (waiting for the previous one to complete
    before starting the next) and then finishing elegantly. Currently, the
    gui passes them all to a "workerThread" which loops through each
    string, sending it to a "processThread" which makes a call to
    subprocess to execute it.

    This has worked fine for over a year so the recent crashing is
    mystifying me. I'm wondering if it's increased network stress (we've
    grown) or something similar?

    Any thoughts and suggestions on waiting for threads to complete are
    appreciated.
    JB, Jan 27, 2011
    #1
    1. Advertising

  2. JB wrote:
    > One of my python scripts that takes a bunch of inputs from a tKinter
    > gui, generates a set of command line stings, and then threads them off
    > to subprocess for calls to other programs like Nuke and our render
    > farm has recently started randomly crashing pythonw.exe.
    >
    > I'm taking a look at my threading setup and attempting to clean it up.
    > I was wondering what a smart way of doing what I describe is? Take a
    > list of strings containing command line calls to other programs,
    > process them one at a time (waiting for the previous one to complete
    > before starting the next) and then finishing elegantly. Currently, the
    > gui passes them all to a "workerThread" which loops through each
    > string, sending it to a "processThread" which makes a call to
    > subprocess to execute it.
    >
    > This has worked fine for over a year so the recent crashing is
    > mystifying me. I'm wondering if it's increased network stress (we've
    > grown) or something similar?
    >
    > Any thoughts and suggestions on waiting for threads to complete are
    > appreciated.
    >


    Google 'Python execnet'.

    The main purpose of this module is to execute jobs on a bunch of
    different machines. However by simply declaring your local as unique
    available gateway, execnet will take care of queueing the jobs on your
    local. Look for examples with 'Popen' type gateways.

    I love this module, (and any other that are doing the network stuff for
    you).

    JM
    Jean-Michel Pichavant, Jan 27, 2011
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Marcos

    managing multiple subprocesses

    Marcos, Feb 3, 2005, in forum: Python
    Replies:
    4
    Views:
    593
    Fredrik Lundh
    Feb 7, 2005
  2. Marcos

    managing multiple subprocesses

    Marcos, Feb 3, 2005, in forum: Python
    Replies:
    0
    Views:
    246
    Marcos
    Feb 3, 2005
  3. Jonathan Smith
    Replies:
    0
    Views:
    249
    Jonathan Smith
    Jun 6, 2006
  4. Mitja Trampus
    Replies:
    0
    Views:
    318
    Mitja Trampus
    Jul 5, 2006
  5. Daz
    Replies:
    5
    Views:
    321
Loading...

Share This Page