Re: subprocess.Popen and multiprocessing fails to execute externalprogram

Discussion in 'Python' started by Dave Angel, Jan 10, 2013.

  1. Dave Angel

    Dave Angel Guest

    On 01/09/2013 11:08 PM, Niklas Berliner wrote:
    > I have a pipline that involves processing some data, handing the data to an
    > external program (t_coffee used for sequence alignments in bioinformatics),
    > and postprocessing the result. Since I have a lot of data, I need to run my
    > pipeline in parallel which I implemented using the multiprocessing module
    > following Doug Hellmanns blog (
    > http://blog.doughellmann.com/2009/04/pymotw-multiprocessing-part-1.html).
    >
    > My pipeline works perfectly fine when I run it with the multiprocessing
    > implementation and one consumer, i.e. on one core. If I increase the number
    > of consumers, i.e. that multiple instances of my pipeline run in parallel
    > the external program fails with a core dump.
    >


    Could it be that the external program is not designed to have multiple
    simultaneous instances? There are many such programs, some of which
    check for an existing process before allowing another one to get far.

    When using the multiprocessing module, always make sure your externals
    are well-behaved before looking for problems in your multi-code.

    To put it more strongly, a well-written program cannot easily be crashed
    by the parent that launched it.


    --

    DaveA
    Dave Angel, Jan 10, 2013
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Earl Eiland
    Replies:
    3
    Views:
    512
    Peter Hansen
    May 16, 2005
  2. Replies:
    3
    Views:
    662
  3. Replies:
    8
    Views:
    728
  4. Hseu-Ming Chen
    Replies:
    1
    Views:
    978
    Chris Torek
    Jun 12, 2011
  5. Niklas Berliner
    Replies:
    0
    Views:
    144
    Niklas Berliner
    Jan 10, 2013
Loading...

Share This Page