build flow? SCons? AAP? process creation?

[corey.coughlin]

Hey guys, here's a little question. I'm looking for something like a
build system, where I can set up a bunch of jobs with dependencies, and
have them run in parallel. Ideally, I'd like a system where jobs can
be run in parallel, all the stdout and stderr for each job is kept in a
database of some kind, jobs can be anything that can be run at a
command line, and a failing job won't cause the whole run to fail.
I've been looking into some of the standard stuff (GNU make) and some
of the more pythonic stuff like SCons and AAP. SCons seems to do
parallel execution, which is nice, but I can't figure out from the docs
if it's possible to keep stdout and stderr somewhere. AAP seems to
have a nice interface, but it doesn't do parallel.

So if I do wind up having to write this thing myself, I've been
checking the docs on process creation, and have a couple questions if
anyone can fill me in. It looks like the os.spawn* commands can start
nonblocking sub-processes, but there doesn't seem to be a way to get
stdout and stderr. On the other hand, the popen commands make it easy
to trap stdout and stderr, but I guess I'd have to do the thread setup
and spawning myself. Is there another alternative that I'm missing
here?

Thanks!

 

[Dennis Benzinger]

[...]
So if I do wind up having to write this thing myself, I've been
checking the docs on process creation, and have a couple questions if
anyone can fill me in. It looks like the os.spawn* commands can start
nonblocking sub-processes, but there doesn't seem to be a way to get
stdout and stderr. On the other hand, the popen commands make it easy
to trap stdout and stderr, but I guess I'd have to do the thread setup
and spawning myself. Is there another alternative that I'm missing
here?
[...]

Yes, you are missing the subprocess module which was introduced in
Python 2.4: http://docs.python.org/lib/module-subprocess.html

Bye,
Dennis

 

[corey.coughlin]

good point, I hadn't checked the docs too closely. Shame it's only in
2.4 though, we're still running 2.2 around here, but it would be nice
to have a reason to upgrade, anyway. Thanks for the pointer!

 

[Steve Christensen]

Hey guys, here's a little question. I'm looking for something like a
build system, where I can set up a bunch of jobs with dependencies, and
have them run in parallel. Ideally, I'd like a system where jobs can
be run in parallel, all the stdout and stderr for each job is kept in a
database of some kind, jobs can be anything that can be run at a
command line, and a failing job won't cause the whole run to fail.
I've been looking into some of the standard stuff (GNU make) and some
of the more pythonic stuff like SCons and AAP. SCons seems to do
parallel execution, which is nice, but I can't figure out from the docs
if it's possible to keep stdout and stderr somewhere. AAP seems to
have a nice interface, but it doesn't do parallel.

Here's something from the SCons Wiki:
http://www.scons.org/cgi-bin/wiki/BuildLog



-Steve