Bug/Weak Implementation? popen* routines can't handle simultaneous read/write?

Discussion in 'Python' started by dmoore, Jun 7, 2007.

  1. dmoore

    dmoore Guest

    Hi folks,

    I've seen the following issue come up in multiple posts to this
    mailing list:

    I have a python program that spawns a child process with popen or
    popen2 or popen3 or popen2.popen2 etc.
    the child process is interactive: it asks for input then spits out
    some output, asks for more input then spits out some output. for
    example, consider the trivial child program:

    print "welcome"
    print ">",
    s=raw_input()
    while s!='exit':
    print "you entered:",s
    print ">",
    s=raw_input()

    Now I may be completely wrong about this (I did play with popen for a
    very long time before writing this message), but it appears that none
    of the popen variants allow for a sequence of reads and writes to/from
    this child. that is, if I read from the open pipe's output I will
    never be able to write the input for the child because the parent
    program will block on read until eof (I will have similar blocking
    problems if I start with write - using readline does not seem to
    help).

    the standard proposed remedy I have seen on this list is to use Unix-
    only select/fctl, or otherwise dig into the bowls of the win32 api, or
    download some half-complete sourceforge process control project. All
    well and good, but unsatisfying for writing platform independent code.

    it turns out that there is at least one open source multi-platform
    (read: win32/linux) api that does handle synchronous I/O with the
    child: wxWidgets and wxPython using the class wxProcess. Now the
    wxWidgets implementation is far from perfect, but it at least allows a
    program to test for new input on the child's stdout and read stdout/
    write stdout in a non-blocking way. However, I find it frustrating
    that I have to import wx just to have useable interactive pipes in my
    python scripts when I would expect this to be part of the native
    python implementation.

    Anybody have any thoughts on this? Do I have my story straight? (the
    popen variants can't handle this case and there are no other
    alternatives in the standard python distro) Is there some place I can
    submit this as a feature request? (Python dev?)
     
    dmoore, Jun 7, 2007
    #1
    1. Advertising

  2. dmoore <> wrote:
    > I've seen the following issue come up in multiple posts to this
    > mailing list:
    >
    > I have a python program that spawns a child process with popen or
    > popen2 or popen3 or popen2.popen2 etc.
    > the child process is interactive: it asks for input then spits out
    > some output, asks for more input then spits out some output. for
    > example, consider the trivial child program:
    >
    > print "welcome"
    > print ">",
    > s=raw_input()
    > while s!='exit':
    > print "you entered:",s
    > print ">",
    > s=raw_input()
    >
    > Now I may be completely wrong about this (I did play with popen for a
    > very long time before writing this message), but it appears that none
    > of the popen variants allow for a sequence of reads and writes to/from
    > this child. that is, if I read from the open pipe's output I will
    > never be able to write the input for the child because the parent
    > program will block on read until eof (I will have similar blocking
    > problems if I start with write - using readline does not seem to
    > help).


    You are correct.

    > the standard proposed remedy I have seen on this list is to use Unix-
    > only select/fctl, or otherwise dig into the bowls of the win32 api, or
    > download some half-complete sourceforge process control project.


    If you are referring to pexpect I've found it works well - unix only
    though. I've not noticed it being half complete.

    There is also a recipe for a non-blocking subprocess - see

    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

    > All well and good, but unsatisfying for writing platform independent code.
    >
    > it turns out that there is at least one open source multi-platform
    > (read: win32/linux) api that does handle synchronous I/O with the
    > child: wxWidgets and wxPython using the class wxProcess. Now the
    > wxWidgets implementation is far from perfect, but it at least allows a
    > program to test for new input on the child's stdout and read stdout/
    > write stdout in a non-blocking way.


    Interesting I didn't know about that - I shall try it!

    > However, I find it frustrating
    > that I have to import wx just to have useable interactive pipes in my
    > python scripts when I would expect this to be part of the native
    > python implementation.
    >
    > Anybody have any thoughts on this? Do I have my story straight? (the
    > popen variants can't handle this case and there are no other
    > alternatives in the standard python distro) Is there some place I can
    > submit this as a feature request? (Python dev?)


    The non-blocking subprocess would make a good start for a stdlib
    submission.

    It should really optionally use ptys under unix too otherwise you'll
    never be able to script passwd etc. An interface a bit like pexpect
    wpuld be useful too (ie scan for these regexps or timeout and return a
    match object).

    --
    Nick Craig-Wood <> -- http://www.craig-wood.com/nick
     
    Nick Craig-Wood, Jun 7, 2007
    #2
    1. Advertising

  3. dmoore

    Terry Reedy Guest

    Re: Bug/Weak Implementation? popen* routines can't handlesimultaneousread/write?

    "dmoore" <> wrote in message
    news:...
    | alternatives in the standard python distro) Is there some place I can
    | submit this as a feature request? (Python dev?)

    http://sourceforge.net/tracker/?group_id=5470

    But don't hold your breath for a volunteer to respond. Discussion here is
    good until you have a concrete request and support/rationale.
     
    Terry Reedy, Jun 7, 2007
    #3
  4. dmoore

    Noah Guest

    On Jun 7, 9:01 am, dmoore <> wrote:

    popen and friends will never do what you want it to do. Down that path
    lies bitter disappointment.
    You need pseduo-ttys and non-blocking IO. I don't know how to do this
    on Windows, but I know it's possible.
    Cygwin does it.

    > Anybody have any thoughts on this? Do I have my story straight? (the
    > popen variants can't handle this case and there are no other
    > alternatives in the standard python distro) Is there some place I can
    > submit this as a feature request? (Python dev?)


    Try Pexpect http://pexpect.sourceforge.net/
    It's been around for a long time and is quite complete and stable.

    The catch is that it's UNIX-only. If you want to tease out the magic
    code from wxProcess that
    does non-blocking reads on win32 then I'd be happy to integrate that
    into the current development branch
    of Pexpect. If someone can provide a pure Python drop-in replacement
    for the read_nonblocking() function
    I use for UNIX systems then it would be easy. I have a whole test
    framework that I can run it through to
    see if it performs the same as the UNIX flavor.

    I'm sure this is feasible without any C extensions -- it might require
    some ctypes hacking.
    I know Windows has pretty decent async IO, but I don't know what they
    have as an equivalent for a pty.
    Maybe it isn't necessary. A pty is only necessary on UNIX because the
    standard c library, stdio, behaves
    differently when it's talking to a plain pipe versus a terminal -- it
    switches buffering
    between block and line oriented buffer. You don't want block buffering
    on interactive applications.
    This is why popen eventually breaks down. No, there is no way to
    select this behavior from
    the calling side... unless you can trick it into dynamically linking
    to your specially hacked libc.
    But that's getting ahead because that's what happens on UNIX -- it
    might be a non-issue on Windows.

    The read_nonblocking() function I use has an interface like this:
    <pre>
    def read_nonblocking (self, size = 1, timeout = -1):

    """This reads at most size characters from the child
    application. It
    includes a timeout. If the read does not complete within the
    timeout
    period then a TIMEOUT exception is raised. If the end of file
    is read
    then an EOF exception will be raised. If a log file was set
    using
    setlog() then all data will also be written to the log file.

    If timeout is None then the read may block indefinitely. If
    timeout is -1
    then the self.timeout value is used. If timeout is 0 then the
    child is
    polled and if there was no data immediately ready then this
    will raise
    a TIMEOUT exception.

    The timeout refers only to the amount of time to read at least
    one
    character. This is not effected by the 'size' parameter, so if
    you call
    read_nonblocking(size=100, timeout=30) and only one character
    is
    available right away then one character will be returned
    immediately.
    It will not wait for 30 seconds for another 99 characters to
    come in.

    This is a wrapper around os.read(). It uses select.select() to
    implement the timeout. """
    </pre>

    Yours,
    Noah
     
    Noah, Jun 8, 2007
    #4
  5. dmoore

    dmoore Guest

    thanks for all of your responses. i'll look more closely at pexpect
    (The version I originally saw was much much older). I do want a cross-
    platform solution. I'm pretty busy so getting the wxProcess magic into
    useful shape will take me some time (at least on the order of weeks)
    so other interest parties feel free to step up.

    cheers
    Damien

    On Jun 7, 8:59 pm, Noah <> wrote:
    > On Jun 7, 9:01 am, dmoore <> wrote:
    >
    > popen and friends will never do what you want it to do. Down that path
    > lies bitter disappointment.
    > You need pseduo-ttys and non-blocking IO. I don't know how to do this
    > on Windows, but I know it's possible.
    > Cygwin does it.
    >
    > > Anybody have any thoughts on this? Do I have my story straight? (the
    > > popen variants can't handle this case and there are no other
    > > alternatives in the standard python distro) Is there some place I can
    > > submit this as a feature request? (Python dev?)

    >
    > Try Pexpecthttp://pexpect.sourceforge.net/
    > It's been around for a long time and is quite complete and stable.
    >
    > The catch is that it's UNIX-only. If you want to tease out the magic
    > code from wxProcess that
    > does non-blocking reads on win32 then I'd be happy to integrate that
    > into the current development branch
    > of Pexpect. If someone can provide a pure Python drop-in replacement
    > for the read_nonblocking() function
    > I use for UNIX systems then it would be easy. I have a whole test
    > framework that I can run it through to
    > see if it performs the same as the UNIX flavor.
    >
    > I'm sure this is feasible without any C extensions -- it might require
    > some ctypes hacking.
    > I know Windows has pretty decent async IO, but I don't know what they
    > have as an equivalent for a pty.
    > Maybe it isn't necessary. A pty is only necessary on UNIX because the
    > standard c library, stdio, behaves
    > differently when it's talking to a plain pipe versus a terminal -- it
    > switches buffering
    > between block and line oriented buffer. You don't want block buffering
    > on interactive applications.
    > This is why popen eventually breaks down. No, there is no way to
    > select this behavior from
    > the calling side... unless you can trick it into dynamically linking
    > to your specially hacked libc.
    > But that's getting ahead because that's what happens on UNIX -- it
    > might be a non-issue on Windows.
    >
    > The read_nonblocking() function I use has an interface like this:
    > <pre>
    > def read_nonblocking (self, size = 1, timeout = -1):
    >
    > """This reads at most size characters from the child
    > application. It
    > includes a timeout. If the read does not complete within the
    > timeout
    > period then a TIMEOUT exception is raised. If the end of file
    > is read
    > then an EOF exception will be raised. If a log file was set
    > using
    > setlog() then all data will also be written to the log file.
    >
    > If timeout is None then the read may block indefinitely. If
    > timeout is -1
    > then the self.timeout value is used. If timeout is 0 then the
    > child is
    > polled and if there was no data immediately ready then this
    > will raise
    > a TIMEOUT exception.
    >
    > The timeout refers only to the amount of time to read at least
    > one
    > character. This is not effected by the 'size' parameter, so if
    > you call
    > read_nonblocking(size=100, timeout=30) and only one character
    > is
    > available right away then one character will be returned
    > immediately.
    > It will not wait for 30 seconds for another 99 characters to
    > come in.
    >
    > This is a wrapper around os.read(). It uses select.select() to
    > implement the timeout. """
    > </pre>
    >
    > Yours,
    > Noah
     
    dmoore, Jun 8, 2007
    #5
  6. Re: Bug/Weak Implementation? popen* routines can't handlesimultaneousread/write?

    "dmoore" <> wrote:

    8< --------------- <description of full duplex problem> ------------

    > Anybody have any thoughts on this? Do I have my story straight? (the
    > popen variants can't handle this case and there are no other
    > alternatives in the standard python distro) Is there some place I can
    > submit this as a feature request? (Python dev?)


    I think this is a hassle in the file implementation - I have had it also on
    serial ports, where it acts as if the file "driver" is inherently half duplex.

    It manifests as: "My characters don't come out"

    I don't know if it is python or the underlying stuff.

    I use fcntl to unblock and generally mess around to solve it, but
    I don't think my solution is portable.

    It is a real issue though - you are not imagining the dragon, it is
    right at the door...

    - Hendrik
     
    Hendrik van Rooyen, Jun 8, 2007
    #6
  7. Noah <> wrote:
    > On Jun 7, 9:01 am, dmoore <> wrote:
    >
    > popen and friends will never do what you want it to do. Down that path
    > lies bitter disappointment.
    > You need pseduo-ttys and non-blocking IO. I don't know how to do this
    > on Windows, but I know it's possible.
    > Cygwin does it.
    >
    > > Anybody have any thoughts on this? Do I have my story straight? (the
    > > popen variants can't handle this case and there are no other
    > > alternatives in the standard python distro) Is there some place I can
    > > submit this as a feature request? (Python dev?)

    >
    > Try Pexpect http://pexpect.sourceforge.net/
    > It's been around for a long time and is quite complete and stable.
    >
    > The catch is that it's UNIX-only. If you want to tease out the magic
    > code from wxProcess that
    > does non-blocking reads on win32 then I'd be happy to integrate that
    > into the current development branch
    > of Pexpect.


    Windows has a really strange idea of non-blocking IO - it uses
    something called overlapped io. You or in the FILE_FLAG_OVERLAPPED
    flag when you create the file/pipe. You then pass in overlap buffers
    for reading writing.

    I implemented this for serial ports in C a while ago. If you look at
    the code in pyserial (in serialwin32.py) you'll see a very similar
    implementation of non blocking IO. I assume that the same things will
    work for pipes, but I've only direct experience with serial ports!

    > I'm sure this is feasible without any C extensions -- it might require
    > some ctypes hacking.


    pyserial uses win32file and win32event

    > I know Windows has pretty decent async IO, but I don't know what they
    > have as an equivalent for a pty.
    > Maybe it isn't necessary. A pty is only necessary on UNIX because the
    > standard c library, stdio, behaves
    > differently when it's talking to a plain pipe versus a terminal -- it
    > switches buffering
    > between block and line oriented buffer. You don't want block buffering
    > on interactive applications.


    Pty's probably aren't needed on Windows.

    BTW I'd love to see pexpect working on windows and also in the
    standard library. It is the proper answer to controlling other
    interactive processes IMHO.

    --
    Nick Craig-Wood <> -- http://www.craig-wood.com/nick
     
    Nick Craig-Wood, Jun 8, 2007
    #7
  8. dmoore

    dmoore Guest

    On Jun 8, 12:30 pm, Nick Craig-Wood <> wrote:
    > Windows has a really strange idea of non-blocking IO - it uses
    > something called overlapped io. You or in the FILE_FLAG_OVERLAPPED
    > flag when you create the file/pipe. You then pass in overlap buffers
    > for reading writing.
    >


    the wx guys appear to do it differently (unless FILE_FLAG_OVERLAPPED
    is implicit in the calls they make)

    take a look at:
    http://cvs.wxwidgets.org/viewcvs.cg...rev=1.88&content-type=text/vnd.viewcvs-markup

    a reasonably well-documented wxExecute function handles all the messy
    win32 api calls and wraps the process in a wxProcess object and the
    streams in wxInputStream / wxOutputStream (also see the wxExecuteDDE
    function)
     
    dmoore, Jun 8, 2007
    #8
  9. dmoore <> wrote:
    > On Jun 8, 12:30 pm, Nick Craig-Wood <> wrote:
    > > Windows has a really strange idea of non-blocking IO - it uses
    > > something called overlapped io. You or in the FILE_FLAG_OVERLAPPED
    > > flag when you create the file/pipe. You then pass in overlap buffers
    > > for reading writing.
    > >

    >
    > the wx guys appear to do it differently (unless FILE_FLAG_OVERLAPPED
    > is implicit in the calls they make)
    >
    > take a look at:
    > http://cvs.wxwidgets.org/viewcvs.cg...rev=1.88&content-type=text/vnd.viewcvs-markup
    >
    > a reasonably well-documented wxExecute function handles all the messy
    > win32 api calls and wraps the process in a wxProcess object and the
    > streams in wxInputStream / wxOutputStream (also see the wxExecuteDDE
    > function)


    You are right, named pipes seem to have a non blocking mode.

    Here is the lowdown from
    http://msdn2.microsoft.com/en-US/library/aa365605.aspx

    Both pipe clients and pipe servers can change a pipe handle's wait
    mode by specifying either PIPE_WAIT or PIPE_NOWAIT in a call to the
    SetNamedPipeHandleState function.

    Note The nonblocking-wait mode is supported for compatibility with
    Microsoft® LAN Manager version 2.0. This mode should not be used to
    achieve overlapped input and output (I/O) with named
    pipes. Overlapped I/O should be used instead, because it enables
    time-consuming operations to run in the background after the
    function returns. For more information about overlapped I/O, see
    Synchronous and Overlapped Input and Output.

    So it has a nonblocking mode but you shouldn't use it!

    --
    Nick Craig-Wood <> -- http://www.craig-wood.com/nick
     
    Nick Craig-Wood, Jun 10, 2007
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kuan Zhou
    Replies:
    1
    Views:
    5,087
    Paul Uiterlinden
    Jan 24, 2005
  2. Piotr Jakubowski

    Threads and simultaneous read and write

    Piotr Jakubowski, Jan 14, 2011, in forum: C++
    Replies:
    2
    Views:
    498
    Joshua Maurice
    Jan 14, 2011
  3. Novice Coder
    Replies:
    27
    Views:
    648
    BCFD36
    Jan 14, 2012
  4. Steve D
    Replies:
    0
    Views:
    186
    Steve D
    Sep 4, 2003
  5. Replies:
    1
    Views:
    145
    Brian McCauley
    Jul 2, 2005
Loading...

Share This Page