Re: Python-list Digest, Vol 112, Issue 79

Discussion in 'Python' started by Niklas Berliner, Jan 10, 2013.

  1. >
    >
    >
    >
    > ---------- Weitergeleitete Nachricht ----------
    > From: Dave Angel <>
    > To:
    > Cc:
    > Date: Thu, 10 Jan 2013 00:56:20 -0500
    > Subject: Re: subprocess.Popen and multiprocessing fails to execute
    > external program
    > On 01/09/2013 11:08 PM, Niklas Berliner wrote:
    > > I have a pipline that involves processing some data, handing the data to

    > an
    > > external program (t_coffee used for sequence alignments in

    > bioinformatics),
    > > and postprocessing the result. Since I have a lot of data, I need to run

    > my
    > > pipeline in parallel which I implemented using the multiprocessing module
    > > following Doug Hellmanns blog (
    > > http://blog.doughellmann.com/2009/04/pymotw-multiprocessing-part-1.html

    > ).
    > >
    > > My pipeline works perfectly fine when I run it with the multiprocessing
    > > implementation and one consumer, i.e. on one core. If I increase the

    > number
    > > of consumers, i.e. that multiple instances of my pipeline run in parallel
    > > the external program fails with a core dump.
    > >

    >
    > Could it be that the external program is not designed to have multiple
    > simultaneous instances? There are many such programs, some of which
    > check for an existing process before allowing another one to get far.
    >
    > When using the multiprocessing module, always make sure your externals
    > are well-behaved before looking for problems in your multi-code.
    >
    > To put it more strongly, a well-written program cannot easily be crashed
    > by the parent that launched it.
    >
    >
    > --
    >
    > DaveA
    >
    >


    Hi Dave,

    the developers of the external program said that they are using the program
    with multiple simultaneous instances. Also, when I execute multiple
    simultaneous instances of the external program using a bash wrapper script
    on my machine it works (the same wrapper scripts that fail when executed
    through python).
    Before asking here I have contacted the developers of the external program
    but they couldn't help me any further.

    Cheers,
    Niklas
     
    Niklas Berliner, Jan 10, 2013
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Andrew James

    Re: Python-list Digest, Vol 17, Issue 54

    Andrew James, Feb 3, 2005, in forum: Python
    Replies:
    2
    Views:
    316
    Steve Holden
    Feb 3, 2005
  2. Charles Hartman

    Re: Python-list Digest, Vol 18, Issue 208

    Charles Hartman, Mar 14, 2005, in forum: Python
    Replies:
    10
    Views:
    511
    Roy Smith
    Mar 16, 2005
  3. Charles Hartman

    Re: Python-list Digest, Vol 18, Issue 391

    Charles Hartman, Mar 24, 2005, in forum: Python
    Replies:
    1
    Views:
    319
    Peter Hansen
    Mar 25, 2005
  4. Replies:
    2
    Views:
    284
  5. Levi Nie
    Replies:
    0
    Views:
    152
    Levi Nie
    Jan 16, 2013
Loading...

Share This Page