subprocess -popen - reading stdout from child - hangs

Discussion in 'Python' started by gregpinero@gmail.com, Sep 23, 2007.

  1. Guest

    Let's say I have this Python file called loop.py:

    import sys
    print 'hi'
    sys.stdout.flush()
    while 1:
    pass

    And I want to call it from another Python process and read the value
    'hi'. How would I do it?

    So far I have tried this:

    >>> proc = subprocess.Popen('python /home/chiefinnovator/loop.py',shell=True,stdin=subprocess.PIPE,stdout=subprocess.PIPE)
    >>> proc.stdout.read()


    But it just hangs at read()

    proc.communicate() also just hangs. What am I doing wrong? Please
    advise.

    Thanks,

    Greg
     
    , Sep 23, 2007
    #1
    1. Advertising

  2. Guest

    On Sep 22, 11:28 pm, "" <>
    wrote:
    > Let's say I have this Python file called loop.py:
    >
    > import sys
    > print 'hi'
    > sys.stdout.flush()
    > while 1:
    > pass
    >
    > And I want to call it from another Python process and read the value
    > 'hi'. How would I do it?
    >
    > So far I have tried this:
    >
    > >>> proc = subprocess.Popen('python /home/chiefinnovator/loop.py',shell=True,stdin=subprocess.PIPE,stdout=subprocess.PIPE)
    > >>> proc.stdout.read()

    >
    > But it just hangs at read()
    >
    > proc.communicate() also just hangs. What am I doing wrong? Please
    > advise.


    Well, using this subclass of subprocess.Popen fixes the problem
    (http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554)

    I don't understand how it works though. Would anyone mind
    explaining? I'm thinking there's something fundamental about Unix
    processes I'm not understanding.

    -Greg
     
    , Sep 23, 2007
    #2
    1. Advertising

  3. wrote:
    > Let's say I have this Python file called loop.py:
    >
    > import sys
    > print 'hi'
    > sys.stdout.flush()
    > while 1:
    > pass
    >
    > And I want to call it from another Python process and read the value
    > 'hi'. How would I do it?
    >
    > So far I have tried this:
    >
    >>>> proc = subprocess.Popen('python /home/chiefinnovator/loop.py',shell=True,stdin=subprocess.PIPE,stdout=subprocess.PIPE)
    >>>> proc.stdout.read()

    >


    From python documentation

    `read([size])'
    Read at most SIZE bytes from the file (less if the read hits `EOF'
    before obtaining SIZE bytes). If the SIZE argument is negative or
    omitted, read all data until `EOF' is reached. The bytes are
    returned as a string object. An empty string is returned when
    `EOF' is encountered immediately. (For certain files, like ttys,
    it makes sense to continue reading after an `EOF' is hit.) Note
    that this method may call the underlying C function `fread()' more
    than once in an effort to acquire as close to SIZE bytes as
    possible. Also note that when in non-blocking mode, less data than
    what was requested may be returned, even if no SIZE parameter was
    given.

    read call in your code is waiting for EOF, since the script never exits
    EOF is not reached.

    Change read code to

    proc.stdout.readline()

    or

    remove while 1 loop from loop.py.

    HTH
    Kolla
     
    Suresh Babu Kolla, Sep 23, 2007
    #3
  4. On Sep 22, 8:28 pm, "" <>
    wrote:
    > Let's say I have this Python file called loop.py:
    >
    > import sys
    > print 'hi'
    > sys.stdout.flush()


    Add sys.stdout.close()

    > while 1:
    > pass
    >
    > And I want to call it from another Python process and read the value
    > 'hi'. How would I do it?
    >
    > So far I have tried this:
    >
    > >>> proc = subprocess.Popen('python /home/chiefinnovator/loop.py',shell=True,stdin=subprocess.PIPE,stdout=subprocess.PIPE)
    > >>> proc.stdout.read()

    >
    > But it just hangs at read()
    >
    > proc.communicate() also just hangs. What am I doing wrong? Please
    > advise.


    Since your loop.py is still alive and hasn't closed its stdout, the
    caller continues to wait for EOF (it doesn't know if loop.py is done
    generating all its output)

    Karthik

    >
    > Thanks,
    >
    > Greg
     
    Karthik Gurusamy, Sep 23, 2007
    #4
  5. In message <>,
    wrote:

    > Let's say I have this Python file called loop.py:
    >
    > import sys
    > print 'hi'
    > sys.stdout.flush()
    > while 1:
    > pass
    >
    > And I want to call it from another Python process and read the value
    > 'hi'. How would I do it?
    >
    > So far I have tried this:
    >
    >>>> proc = subprocess.Popen('python
    >>>> /home/chiefinnovator/loop.py',shell=True,stdin=subprocess.PIPE,stdout=subprocess.PIPE)
    >>>> proc.stdout.read()

    >
    > But it just hangs at read()


    That's because you didn't tell it how much to read, so by default it tries
    to read until EOF <http://docs.python.org/lib/bltin-file-objects.html>. But
    since the subprocess is still running and hasn't closed its stdout, the
    pipe has not reached EOF.
     
    Lawrence D'Oliveiro, Sep 24, 2007
    #5
  6. In message <>,
    wrote:

    > Well, using this subclass of subprocess.Popen fixes the problem
    > (http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554)
    >
    > I don't understand how it works though. Would anyone mind
    > explaining? I'm thinking there's something fundamental about Unix
    > processes I'm not understanding.


    The "select" calls are checking that the subprocess is ready to read before
    writing to its stdin, or that it has something to write before reading from
    its stdout.

    The O_NONBLOCK fcntl simply says not to wait if there is nothing more to be
    read, just return what was already read.
     
    Lawrence D'Oliveiro, Sep 24, 2007
    #6
  7. Guest

    On Sep 23, 2:58 am, Karthik Gurusamy <> wrote:
    > On Sep 22, 8:28 pm, "" <>
    > wrote:
    >
    > > Let's say I have this Python file called loop.py:

    >
    > > import sys
    > > print 'hi'
    > > sys.stdout.flush()

    >
    > Add sys.stdout.close()
    >


    Adding sys.stdout.close() and removing sys.stdout.flush() seems to
    make it work. But can the while loop still use sys.stdout later on?
    Do I have to reopen it?

    Thanks,

    Greg
     
    , Sep 24, 2007
    #7
  8. On Sep 24, 2:22 pm, "" <>
    wrote:
    > On Sep 23, 2:58 am, Karthik Gurusamy <> wrote:
    >
    > > On Sep 22, 8:28 pm, "" <>
    > > wrote:

    >
    > > > Let's say I have this Python file called loop.py:

    >
    > > > import sys
    > > > print 'hi'
    > > > sys.stdout.flush()

    >
    > > Add sys.stdout.close()

    >
    > Adding sys.stdout.close() and removing sys.stdout.flush() seems to
    > make it work. But can the while loop still use sys.stdout later on?
    > Do I have to reopen it?


    Once you close a file-object, you cannot use it. You'll get exception
    if you try.
    I quickly tried the fdopen on id 1 (stdout in unix like OS) and it
    seems to work

    >>> import sys
    >>> sys.stdout.close()
    >>> import os
    >>> print 'hi'

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    ValueError: I/O operation on closed file
    >>> sys.stdout=os.fdopen(1, 'w')
    >>> print 'hi'

    hi
    >>>


    BUT you may want to re-think your design. Note that your caller
    process will anyway stop reading further from your loop.py process
    the moment it sees the "first" EOF. So even if you enable loop.py to
    generate more output (thru' the above fdopen), the caller process is
    not going to see the newly generated data.

    If your requirement is to collect all output from loop.py then you
    can't do it if loop.py has an infinite loop across its data generation
    points (ie it can generate data after the infinite loop -- which
    anyway doesn't sense).

    If all you want is not to get blocked, try one of the select solutions
    or read a small amount at a time (which you can guarantee to be
    available). Yet another solution would be you could set up an alarm
    and get out of the blocking read if the alarm fires.

    Karthik

    >
    > Thanks,
    >
    > Greg
     
    Karthik Gurusamy, Sep 25, 2007
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jonathan Amsterdam

    bug: subprocess.Popen() hangs

    Jonathan Amsterdam, Oct 25, 2007, in forum: Python
    Replies:
    1
    Views:
    538
    Nick Craig-Wood
    Oct 26, 2007
  2. Kenneth McDonald

    subprocess.Popen hangs at times?

    Kenneth McDonald, Sep 10, 2008, in forum: Python
    Replies:
    0
    Views:
    310
    Kenneth McDonald
    Sep 10, 2008
  3. giohappy
    Replies:
    5
    Views:
    1,842
    giohappy
    Apr 15, 2009
  4. Tim Arnold

    subprocess hangs on reading stdout

    Tim Arnold, Oct 14, 2009, in forum: Python
    Replies:
    2
    Views:
    528
    Tim Arnold
    Oct 15, 2009
  5. Tim Johnson

    subprocess.Popen instance hangs

    Tim Johnson, Aug 29, 2013, in forum: Python
    Replies:
    0
    Views:
    83
    Tim Johnson
    Aug 29, 2013
Loading...

Share This Page