subprocess.Popen pipeline bug?

Discussion in 'Python' started by Marko Rauhamaa, Mar 13, 2008.

  1. This tiny program hangs:

    ========================================================================
    #!/usr/bin/env python
    import subprocess
    a = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE,
    stdout = subprocess.PIPE)
    b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout)
    a.stdin.close()
    b.wait() # hangs
    a.wait() # never reached
    ========================================================================

    It shouldn't, should it?

    Environment:
    ========================================================================
    Python 2.5.1 (r251:54863, Jun 20 2007, 12:14:09)
    [GCC 4.1.2 20061115 (prerelease) (SUSE Linux)] on linux2
    ========================================================================


    Marko

    --
    Marko Rauhamaa mailto: http://pacujo.net/marko/
    Marko Rauhamaa, Mar 13, 2008
    #1
    1. Advertising

  2. Marko Rauhamaa

    Guest

    Marko Rauhamaa wrote:
    > This tiny program hangs:
    >
    > ========================================================================
    > #!/usr/bin/env python
    > import subprocess
    > a = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE,
    > stdout = subprocess.PIPE)
    > b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout)
    > a.stdin.close()
    > b.wait() # hangs
    > a.wait() # never reached
    > ========================================================================


    To make it work, add close_fds=True in the Popen that creates b.

    > It shouldn't, should it?


    Not sure. I think what's happening is that the second cat subprocess
    never gets EOF on its stdin, because there are still processes with
    an open file descriptor for the other end of the pipe.

    The Python program closes a.stdin, and let's suppose that's file
    descriptor 4. That's not enough, because the subshell that ran cat and
    the cat process itself inherited the open file descriptor 4 when they
    forked off.

    It looks like Popen is smart enough to close the extraneous
    descriptors for pipes it created in the same Popen call, but that
    one was created in a previous call and passed in.


    --
    --Bryan
    , Mar 13, 2008
    #2
    1. Advertising

  3. In article <>,
    Marko Rauhamaa <> writes:
    >
    > This tiny program hangs:
    >
    > ========================================================================
    > #!/usr/bin/env python
    > import subprocess
    > a = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE,
    > stdout = subprocess.PIPE)
    > b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout)
    > a.stdin.close()
    > b.wait() # hangs
    > a.wait() # never reached
    > ========================================================================
    >
    > It shouldn't, should it?


    Yes, it should.

    This issue is related to the subtleties of creating a pipeline in
    POSIX environments. The problem is that the cat command in
    subprocess a never completes because it never encounters an EOF
    (on a.stdin). Even though you issue a close call (a.stdin.close ()),
    you're not issuing the "last" close. That's because there is still
    at least one file descriptor open in subprocess tree b. That
    happened because it was open when the subprocess module executed
    a POSIX fork call and it got duplicated as part of the fork call.

    I don't see any clean and simple way to actually fix this. (That's
    one of the reasons why POSIX shells are so complicated.) There
    are a couple of work-arounds that you can use:

    1) Force close-on-exec on the specific file descriptor:

    import subprocess
    a = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE,
    stdout = subprocess.PIPE)
    # ********* beginning of changes
    import os, fcntl
    fd = a.stdin.fileno ()
    old = fcntl.fcntl (fd, fcntl.F_GETFD)
    fcntl.fcntl (fd, fcntl.F_SETFD, old | fcntl.FD_CLOEXEC)
    # ********* end of changes
    b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout)
    a.stdin.close()
    b.wait()
    a.wait()

    Or if it happens to not cause undesired side-effects for you, you can
    2) Force close-on-exec *all* non-standard file descriptors by using
    the close_fds argument to Popen:

    import subprocess
    a = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE,
    stdout = subprocess.PIPE)
    # ********* beginning of changes
    # b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout)
    b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout,
    close_fds = True)
    # ********* end of changes
    a.stdin.close()
    b.wait()
    a.wait()

    Good luck.

    - dmw

    --
    .. Douglas Wells . Connection Technologies .
    .. Internet: -sp9804- -at - contek.com- .
    Douglas Wells, Mar 13, 2008
    #3
  4. :

    > Not sure. I think what's happening is that the second cat subprocess
    > never gets EOF on its stdin, because there are still processes with an
    > open file descriptor for the other end of the pipe.


    You are right. However, the close_fds technique seems a bit
    heavy-handed. Well, that's what you get when you try to combine fork and
    exec into a single call.


    Marko

    --
    Marko Rauhamaa mailto: http://pacujo.net/marko/
    Marko Rauhamaa, Mar 13, 2008
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Michele Simionato

    subprocess.Popen

    Michele Simionato, Dec 12, 2004, in forum: Python
    Replies:
    5
    Views:
    17,635
  2. Ivo Woltring
    Replies:
    4
    Views:
    3,531
    Jeff Shannon
    Jan 27, 2005
  3. John Abel
    Replies:
    2
    Views:
    3,272
    John Abel
    May 12, 2005
  4. Jonathan Amsterdam

    bug: subprocess.Popen() hangs

    Jonathan Amsterdam, Oct 25, 2007, in forum: Python
    Replies:
    1
    Views:
    532
    Nick Craig-Wood
    Oct 26, 2007
  5. File.popen/IO.popen

    , May 20, 2006, in forum: Ruby
    Replies:
    1
    Views:
    217
    Robert Klemme
    May 20, 2006
Loading...

Share This Page