Python deadlocks when Popen in multithread.

Discussion in 'Python' started by INADA Naoki, Oct 11, 2010.

  1. INADA Naoki

    INADA Naoki Guest

    On Ubuntu 10.10 amd64 and dual-core CPU, following code deadlocks.
    What's wrong on this code?

    #http://codepad.org/GkoXHbik

    #!/usr/bin/env python

    from subprocess import *
    from threading import *

    THREAD_COUNT=50

    def worker():
    p = Popen(["cat"], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
    out = p.communicate("hoge")[0]
    print "%s %s" % (current_thread().name, out)

    threads = []
    for i in range(THREAD_COUNT):
    threads.append(Thread(target=worker, name=str(i)))
    for t in threads: t.start()
    for t in threads: t.join()
     
    INADA Naoki, Oct 11, 2010
    #1
    1. Advertising

  2. INADA Naoki

    Jed Smith Guest

    On Mon, Oct 11, 2010 at 4:19 PM, INADA Naoki <> wrote:

    > def worker():
    >    p = Popen(["cat"], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
    >    out = p.communicate("hoge")[0]
    >    print "%s %s" % (current_thread().name, out)


    If, instead of spawning workers you directly call worker(), does it succeed?

    I suspect the threading is a red herring here.

    --
    Jed Smith
     
    Jed Smith, Oct 11, 2010
    #2
    1. Advertising

  3. INADA Naoki

    INADA Naoki Guest

    On Oct 12, 5:33 am, Jed Smith <> wrote:
    > On Mon, Oct 11, 2010 at 4:19 PM, INADA Naoki <> wrote:
    > > def worker():
    > >    p = Popen(["cat"], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
    > >    out = p.communicate("hoge")[0]
    > >    print "%s %s" % (current_thread().name, out)

    >
    > If, instead of spawning workers you directly call worker(), does it succeed?


    Yes.

    > I suspect the threading is a red herring here.


    But thread & subprocess is a normal way to invoke outer commands in
    parallel.
     
    INADA Naoki, Oct 12, 2010
    #3
  4. INADA Naoki

    INADA Naoki Guest

    I've found the bug http://bugs.python.org/issue1731717.
    It seems wating subprocess can be multithreaded but creating
    subprocess cannot.

    On Oct 12, 9:57 am, INADA Naoki <> wrote:
    > On Oct 12, 5:33 am, Jed Smith <> wrote:
    >
    > > On Mon, Oct 11, 2010 at 4:19 PM, INADA Naoki <> wrote:
    > > > def worker():
    > > >    p = Popen(["cat"], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
    > > >    out = p.communicate("hoge")[0]
    > > >    print "%s %s" % (current_thread().name, out)

    >
    > > If, instead of spawning workers you directly call worker(), does it succeed?

    >
    > Yes.
    >
    > > I suspect the threading is a red herring here.

    >
    > But thread & subprocess is a normal way to invoke outer commands in
    > parallel.
     
    INADA Naoki, Oct 12, 2010
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Martin Blackstone [MVP - Exchange]

    ASP.NET Deadlocks

    Martin Blackstone [MVP - Exchange], Aug 23, 2003, in forum: ASP .Net
    Replies:
    6
    Views:
    765
    Nick Wienholt
    Aug 24, 2003
  2. r_obert@REMOVE_THIS.hotmail.com

    Multithread or Multithread DLL?

    r_obert@REMOVE_THIS.hotmail.com, Nov 27, 2004, in forum: C++
    Replies:
    0
    Views:
    2,241
    r_obert@REMOVE_THIS.hotmail.com
    Nov 27, 2004
  3. kanji
    Replies:
    1
    Views:
    400
    Anders J. Munch
    Sep 29, 2004
  4. eraserix

    subprocess.Popen deadlocks

    eraserix, Nov 10, 2010, in forum: Python
    Replies:
    2
    Views:
    588
    eraserix
    Nov 12, 2010
  5. File.popen/IO.popen

    , May 20, 2006, in forum: Ruby
    Replies:
    1
    Views:
    232
    Robert Klemme
    May 20, 2006
Loading...

Share This Page