A little help with child processes.

Discussion in 'Python' started by Rech, Jun 16, 2004.

  1. Rech

    Rech Guest

    Hi,
    I need a little help here managing child processes in Python. I'm not so
    skilled in system programming so I hope you can give me some good
    suggestions.

    I have a very CPU and memory intensive task that has to be repeated many
    times (with different input parameters). So I've thought to write a
    Python script that create a child process, wait for it to finish and
    then starts again another child with different parameters.

    The children processes will save all the results to the disk (using
    cPickle module), so the parent process has to wait each child only and
    then starts the next one. The problem is that I can't work out a
    solution. Sorry, but system programming is not my job.

    Any suggestions about how to realize that? A skeleton of the script will
    just suffice.


    Thanks in advance,
    Andrea.
     
    Rech, Jun 16, 2004
    #1
    1. Advertising

  2. Rech

    Larry Bates Guest

    Unless you are going to start more than one child
    process in parallel and there is enough I/O to
    make it worthwhile, there's no reason for using
    child processes at all. Just program the application
    as a single loop. You can't speed up CPU bound
    applications with child processes.

    HTH,
    Larry Bates
    Syscon, Inc.
    "Rech" <> wrote in message
    news:...
    > Hi,
    > I need a little help here managing child processes in Python. I'm not so
    > skilled in system programming so I hope you can give me some good
    > suggestions.
    >
    > I have a very CPU and memory intensive task that has to be repeated many
    > times (with different input parameters). So I've thought to write a
    > Python script that create a child process, wait for it to finish and
    > then starts again another child with different parameters.
    >
    > The children processes will save all the results to the disk (using
    > cPickle module), so the parent process has to wait each child only and
    > then starts the next one. The problem is that I can't work out a
    > solution. Sorry, but system programming is not my job.
    >
    > Any suggestions about how to realize that? A skeleton of the script will
    > just suffice.
    >
    >
    > Thanks in advance,
    > Andrea.
     
    Larry Bates, Jun 16, 2004
    #2
    1. Advertising

  3. Rech

    Rech Guest

    Yes I know, but the CPU-intensive tasks eat a lot of memory and for
    weird reasons they don't free it (I know bad programming, but re-writing
    all the code is impratical at this point). Running them in child
    processes solves the problem because when a child dies it frees all the
    allocated memory.

    Andrea.


    In article <>,
    "Larry Bates" <> wrote:

    > Unless you are going to start more than one child
    > process in parallel and there is enough I/O to
    > make it worthwhile, there's no reason for using
    > child processes at all. Just program the application
    > as a single loop. You can't speed up CPU bound
    > applications with child processes.
    >
    > HTH,
    > Larry Bates
    > Syscon, Inc.
     
    Rech, Jun 16, 2004
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jeff Rodriguez
    Replies:
    23
    Views:
    1,180
    David Schwartz
    Dec 9, 2003
  2. ..:: sjf ::..
    Replies:
    0
    Views:
    429
    ..:: sjf ::..
    May 6, 2004
  3. Rob Newman
    Replies:
    0
    Views:
    1,004
    Rob Newman
    Jun 16, 2009
  4. Matt
    Replies:
    2
    Views:
    434
    Mike Kazantsev
    Jun 17, 2009
  5. Marc Heiler
    Replies:
    1
    Views:
    187
    Robert Klemme
    May 24, 2009
Loading...

Share This Page