Managing a queue of subprocesses?

Discussion in 'Python' started by cypher543, Dec 30, 2006.

  1. cypher543

    cypher543 Guest

    My app uses a "queue" of commands which are run one at a time. I am
    using the subprocess module to execute the commands in the queue.
    However, processes always run at the same time. How can I make one
    process run at a time, and then execute the next process when the first
    has terminated? My code is below:

    self.cmdQueue = {}
    self.queue(theProject.directory, "ls", "-l")
    self.queue(theProject.directory, "echo", "hello, world!")

    def consoleLogAddLine(self, text):
    self.consoleLogBuffer.insert(self.consoleLogBuffer.get_end_iter(),
    text)
    self.consoleLog.scroll_to_mark(self.consoleLogBuffer.get_insert(), 0)

    def onGetData(self, fd, cond, *args):
    self.consoleLogAddLine(fd.readline())
    return True

    def queue(self, rootDir, cmd, args = ""):
    count = len(self.cmdQueue) + 1
    self.cmdQueue[count] = [cmd, args, rootDir]

    def runQueue(self):
    for i in self.cmdQueue.values():
    self.execute(i[2], i[0], i[1])

    def execute(self, rootDir, cmd, args = ""):
    os.chdir(rootDir)
    if args == "":
    buildCmd = cmd
    else:
    args = args.split(" ")
    buildCmd = [cmd] + args
    self.buildPID = subprocess.Popen(buildCmd, stdout = subprocess.PIPE,
    stderr = subprocess.STDOUT)
    gobject.io_add_watch(self.buildPID.stdout, gobject.IO_IN,
    self.onGetData)

    As you can see, I add the commands "ls -l" and "echo Hello" to the
    queue. However, "Hello" is always printed inside the output of "ls -l".
    I would like to wait for "ls -l" to terminate and then run "echo
    Hello". But, the output must still print to the consoleLogBuffer
    line-by-line, and my GUI must not hang during execution.

    Is this even possible?
     
    cypher543, Dec 30, 2006
    #1
    1. Advertising

  2. cypher543

    Tom Plunket Guest

    cypher543 wrote:

    > self.buildPID = subprocess.Popen(buildCmd, stdout = subprocess.PIPE, stderr = subprocess.STDOUT)


    Instead of calling it self.buildPID, you might just call it
    self.buildProcess or something. It's actually a Popen object that gets
    returned.

    So yes you can do what you want:

    __init__ self.buildProcess to None. Then, in execute(), if
    (self.buildProcess is None) or (self.buildProcess.poll() is not None)
    start the next process.

    You've got to wait for one process to end before starting the next one,
    it's really that easy. So, don't just go ahead and fire them all
    instantly. Possibly what you want to do instead is have runQueue() do
    the check somehow that there's no active process running.

    What I would do, have runQueue() check to see if self.buildProcess is
    None. If it is None, fire the next process in the queue. If it isn't
    None, then check to see if it's ended. If it has ended, then set
    self.buildProcess to None. Next UI update the next step in the queue
    gets done. Mind that you'll have to modify the queue as you go, e.g.
    self.queue = self.queue[1:].

    Finally, consider piping stderr separately, and direct its output to a
    different window in your GUI. You could even make that window pop open
    on demand, if errors occur.

    good luck,
    -tom!

    --
     
    Tom Plunket, Dec 30, 2006
    #2
    1. Advertising

  3. cypher543

    cypher543 Guest

    That was a very good answer, and it sure sounds like it would work.
    However, I failed at implementing it. :( My updated runQueue() function
    is:

    def runQueue(self):
    self.buildProcess = None
    count = 1 # current position in the queue
    while True:
    if self.buildProcess is None:
    self.execute(self.cmdQueue[count][2], self.cmdQueue[count][0],
    self.cmdQueue[count][1])
    count = count + 1
    else:
    # I'm not really sure what to put here

    I pretty sure I did all of that wrong. ;) Also, how exactly would I
    redirect stderr to another place?

    On Dec 30, 12:22 am, Tom Plunket <> wrote:
    > cypher543 wrote:
    > > self.buildPID = subprocess.Popen(buildCmd, stdout = subprocess.PIPE, stderr = subprocess.STDOUT)Instead of calling it self.buildPID, you might just call it

    > self.buildProcess or something. It's actually a Popen object that gets
    > returned.
    >
    > So yes you can do what you want:
    >
    > __init__ self.buildProcess to None. Then, in execute(), if
    > (self.buildProcess is None) or (self.buildProcess.poll() is not None)
    > start the next process.
    >
    > You've got to wait for one process to end before starting the next one,
    > it's really that easy. So, don't just go ahead and fire them all
    > instantly. Possibly what you want to do instead is have runQueue() do
    > the check somehow that there's no active process running.
    >
    > What I would do, have runQueue() check to see if self.buildProcess is
    > None. If it is None, fire the next process in the queue. If it isn't
    > None, then check to see if it's ended. If it has ended, then set
    > self.buildProcess to None. Next UI update the next step in the queue
    > gets done. Mind that you'll have to modify the queue as you go, e.g.
    > self.queue = self.queue[1:].
    >
    > Finally, consider piping stderr separately, and direct its output to a
    > different window in your GUI. You could even make that window pop open
    > on demand, if errors occur.
    >
    > good luck,
    > -tom!
    >
    > --
     
    cypher543, Dec 30, 2006
    #3
  4. cypher543

    Tom Plunket Guest

    cypher543 wrote:

    > That was a very good answer, and it sure sounds like it would work.
    > However, I failed at implementing it. :( My updated runQueue() function
    > is:
    >
    > def runQueue(self):
    > self.buildProcess = None
    > count = 1 # current position in the queue
    > while True:
    > if self.buildProcess is None:
    > self.execute(self.cmdQueue[count][2], self.cmdQueue[count][0],
    > self.cmdQueue[count][1])
    > count = count + 1
    > else:
    > # I'm not really sure what to put here
    >
    > I pretty sure I did all of that wrong. ;) Also, how exactly would I
    > redirect stderr to another place?


    You're thinking too hard. ;)

    class Whatever:
    def __init__(self):
    self.process = None
    # and other stuff, probably.

    def runQueue(self):
    # check to see if no process has been started, or if
    # the most-recently-started one has finished.
    if not (self.process or (self.process.poll() is None)):
    if self.process:
    previousReturnCode = self.process.returncode

    if len(self.cmdQueue) > 0:
    command = self.cmdQueue.pop(0) # pull off the first command.
    self.execute(command[2], command[0], command[1])
    else:
    self.process = None

    ....then, to prevent your GUI from freezing, you need to just call this
    function in your idle handling. If you don't want to start all of your
    commands at once, you need to not start all your commands at once. ;)


    -tom!

    --
     
    Tom Plunket, Jan 1, 2007
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Marcos

    managing multiple subprocesses

    Marcos, Feb 3, 2005, in forum: Python
    Replies:
    4
    Views:
    619
    Fredrik Lundh
    Feb 7, 2005
  2. Marcos

    managing multiple subprocesses

    Marcos, Feb 3, 2005, in forum: Python
    Replies:
    0
    Views:
    270
    Marcos
    Feb 3, 2005
  3. Paul L. Du Bois

    Queue.Queue-like class without the busy-wait

    Paul L. Du Bois, Mar 24, 2005, in forum: Python
    Replies:
    29
    Views:
    1,110
    Antoon Pardon
    Apr 4, 2005
  4. Russell Warren

    Is Queue.Queue.queue.clear() thread-safe?

    Russell Warren, Jun 22, 2006, in forum: Python
    Replies:
    4
    Views:
    725
    Russell Warren
    Jun 27, 2006
  5. Kris
    Replies:
    0
    Views:
    535
Loading...

Share This Page