Managing a queue of subprocesses?

[cypher543]

My app uses a "queue" of commands which are run one at a time. I am
using the subprocess module to execute the commands in the queue.
However, processes always run at the same time. How can I make one
process run at a time, and then execute the next process when the first
has terminated? My code is below:

self.cmdQueue = {}
self.queue(theProject.directory, "ls", "-l")
self.queue(theProject.directory, "echo", "hello, world!")

def consoleLogAddLine(self, text):
self.consoleLogBuffer.insert(self.consoleLogBuffer.get_end_iter(),
text)
self.consoleLog.scroll_to_mark(self.consoleLogBuffer.get_insert(), 0)

def onGetData(self, fd, cond, *args):
self.consoleLogAddLine(fd.readline())
return True

def queue(self, rootDir, cmd, args = ""):
count = len(self.cmdQueue) + 1
self.cmdQueue[count] = [cmd, args, rootDir]

def runQueue(self):
for i in self.cmdQueue.values():
self.execute(i[2], i[0], i[1])

def execute(self, rootDir, cmd, args = ""):
os.chdir(rootDir)
if args == "":
buildCmd = cmd
else:
args = args.split(" ")
buildCmd = [cmd] + args
self.buildPID = subprocess.Popen(buildCmd, stdout = subprocess.PIPE,
stderr = subprocess.STDOUT)
gobject.io_add_watch(self.buildPID.stdout, gobject.IO_IN,
self.onGetData)

As you can see, I add the commands "ls -l" and "echo Hello" to the
queue. However, "Hello" is always printed inside the output of "ls -l".
I would like to wait for "ls -l" to terminate and then run "echo
Hello". But, the output must still print to the consoleLogBuffer
line-by-line, and my GUI must not hang during execution.

Is this even possible?

 

[Tom Plunket]

self.buildPID = subprocess.Popen(buildCmd, stdout = subprocess.PIPE, stderr = subprocess.STDOUT)

Instead of calling it self.buildPID, you might just call it
self.buildProcess or something. It's actually a Popen object that gets
returned.

So yes you can do what you want:

__init__ self.buildProcess to None. Then, in execute(), if
(self.buildProcess is None) or (self.buildProcess.poll() is not None)
start the next process.

You've got to wait for one process to end before starting the next one,
it's really that easy. So, don't just go ahead and fire them all
instantly. Possibly what you want to do instead is have runQueue() do
the check somehow that there's no active process running.

What I would do, have runQueue() check to see if self.buildProcess is
None. If it is None, fire the next process in the queue. If it isn't
None, then check to see if it's ended. If it has ended, then set
self.buildProcess to None. Next UI update the next step in the queue
gets done. Mind that you'll have to modify the queue as you go, e.g.
self.queue = self.queue[1:].

Finally, consider piping stderr separately, and direct its output to a
different window in your GUI. You could even make that window pop open
on demand, if errors occur.

good luck,
-tom!

--

 

[cypher543]

That was a very good answer, and it sure sounds like it would work.
However, I failed at implementing it. My updated runQueue() function
is:

def runQueue(self):
self.buildProcess = None
count = 1 # current position in the queue
while True:
if self.buildProcess is None:
self.execute(self.cmdQueue[count][2], self.cmdQueue[count][0],
self.cmdQueue[count][1])
count = count + 1
else:
# I'm not really sure what to put here

I pretty sure I did all of that wrong. Also, how exactly would I
redirect stderr to another place?

self.buildPID = subprocess.Popen(buildCmd, stdout = subprocess.PIPE, stderr = subprocess.STDOUT)Instead of calling it self.buildPID, you might just call it

self.buildProcess or something. It's actually a Popen object that gets
returned.

So yes you can do what you want:

__init__ self.buildProcess to None. Then, in execute(), if
(self.buildProcess is None) or (self.buildProcess.poll() is not None)
start the next process.

You've got to wait for one process to end before starting the next one,
it's really that easy. So, don't just go ahead and fire them all
instantly. Possibly what you want to do instead is have runQueue() do
the check somehow that there's no active process running.

What I would do, have runQueue() check to see if self.buildProcess is
None. If it is None, fire the next process in the queue. If it isn't
None, then check to see if it's ended. If it has ended, then set
self.buildProcess to None. Next UI update the next step in the queue
gets done. Mind that you'll have to modify the queue as you go, e.g.
self.queue = self.queue[1:].

Finally, consider piping stderr separately, and direct its output to a
different window in your GUI. You could even make that window pop open
on demand, if errors occur.

good luck,
-tom!

--

 

[Tom Plunket]

That was a very good answer, and it sure sounds like it would work.
However, I failed at implementing it. My updated runQueue() function
is:

def runQueue(self):
self.buildProcess = None
count = 1 # current position in the queue
while True:
if self.buildProcess is None:
self.execute(self.cmdQueue[count][2], self.cmdQueue[count][0],
self.cmdQueue[count][1])
count = count + 1
else:
# I'm not really sure what to put here

I pretty sure I did all of that wrong. Also, how exactly would I
redirect stderr to another place?

You're thinking too hard.

class Whatever:
def __init__(self):
self.process = None
# and other stuff, probably.

def runQueue(self):
# check to see if no process has been started, or if
# the most-recently-started one has finished.
if not (self.process or (self.process.poll() is None)):
if self.process:
previousReturnCode = self.process.returncode

if len(self.cmdQueue) > 0:
command = self.cmdQueue.pop(0) # pull off the first command.
self.execute(command[2], command[0], command[1])
else:
self.process = None

....then, to prevent your GUI from freezing, you need to just call this
function in your idle handling. If you don't want to start all of your
commands at once, you need to not start all your commands at once.


-tom!

--