build flow? SCons? AAP? process creation?

Discussion in 'Python' started by corey.coughlin@comcast.net, Apr 13, 2005.

  1. Guest

    Hey guys, here's a little question. I'm looking for something like a
    build system, where I can set up a bunch of jobs with dependencies, and
    have them run in parallel. Ideally, I'd like a system where jobs can
    be run in parallel, all the stdout and stderr for each job is kept in a
    database of some kind, jobs can be anything that can be run at a
    command line, and a failing job won't cause the whole run to fail.
    I've been looking into some of the standard stuff (GNU make) and some
    of the more pythonic stuff like SCons and AAP. SCons seems to do
    parallel execution, which is nice, but I can't figure out from the docs
    if it's possible to keep stdout and stderr somewhere. AAP seems to
    have a nice interface, but it doesn't do parallel.

    So if I do wind up having to write this thing myself, I've been
    checking the docs on process creation, and have a couple questions if
    anyone can fill me in. It looks like the os.spawn* commands can start
    nonblocking sub-processes, but there doesn't seem to be a way to get
    stdout and stderr. On the other hand, the popen commands make it easy
    to trap stdout and stderr, but I guess I'd have to do the thread setup
    and spawning myself. Is there another alternative that I'm missing
    here?

    Thanks!
     
    , Apr 13, 2005
    #1
    1. Advertising

  2. schrieb:
    > [...]
    > So if I do wind up having to write this thing myself, I've been
    > checking the docs on process creation, and have a couple questions if
    > anyone can fill me in. It looks like the os.spawn* commands can start
    > nonblocking sub-processes, but there doesn't seem to be a way to get
    > stdout and stderr. On the other hand, the popen commands make it easy
    > to trap stdout and stderr, but I guess I'd have to do the thread setup
    > and spawning myself. Is there another alternative that I'm missing
    > here?
    > [...]


    Yes, you are missing the subprocess module which was introduced in
    Python 2.4: http://docs.python.org/lib/module-subprocess.html

    Bye,
    Dennis
     
    Dennis Benzinger, Apr 13, 2005
    #2
    1. Advertising

  3. Guest

    good point, I hadn't checked the docs too closely. Shame it's only in
    2.4 though, we're still running 2.2 around here, but it would be nice
    to have a reason to upgrade, anyway. Thanks for the pointer!
     
    , Apr 13, 2005
    #3
  4. In article <>, wrote:
    > Hey guys, here's a little question. I'm looking for something like a
    > build system, where I can set up a bunch of jobs with dependencies, and
    > have them run in parallel. Ideally, I'd like a system where jobs can
    > be run in parallel, all the stdout and stderr for each job is kept in a
    > database of some kind, jobs can be anything that can be run at a
    > command line, and a failing job won't cause the whole run to fail.
    > I've been looking into some of the standard stuff (GNU make) and some
    > of the more pythonic stuff like SCons and AAP. SCons seems to do
    > parallel execution, which is nice, but I can't figure out from the docs
    > if it's possible to keep stdout and stderr somewhere. AAP seems to
    > have a nice interface, but it doesn't do parallel.


    Here's something from the SCons Wiki:
    http://www.scons.org/cgi-bin/wiki/BuildLog



    -Steve

    --
    So if you people can't figure out what my birthday is based on subtle
    hints and vague statements made at random intervals then I'm beginning
    to think that you really don't care. -- Dean Lenort, a.r.k.
     
    Steve Christensen, Apr 19, 2005
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Lothar Werzinger
    Replies:
    1
    Views:
    510
    fuzzylollipop
    Dec 27, 2004
  2. ted

    SCons build tool speed

    ted, Feb 12, 2005, in forum: Python
    Replies:
    8
    Views:
    1,449
    Peter Hansen
    Feb 14, 2005
  3. sks
    Replies:
    4
    Views:
    406
    Alf P. Steinbach
    Jul 6, 2006
  4. Jack Dowson
    Replies:
    0
    Views:
    463
    Jack Dowson
    May 7, 2007
  5. Ruby Script
    Replies:
    0
    Views:
    141
    Ruby Script
    Aug 21, 2004
Loading...

Share This Page