non-blocking PIPE read on Windows

Discussion in 'Python' started by placid, Jul 28, 2006.

  1. placid

    placid Guest

    Hi all,

    I have been looking into non-blocking read (readline) operations on
    PIPES on windows XP and there seems to be no way of doing this. Ive
    read that you could use a Thread to read from the pipe, but if you
    still use readline() wouldnt the Thread block too?

    What i need to do is, create a process using subprocess.Popen, where
    the subprocess outputs information on one line (but the info
    continuesly changes and its always on the same line) and read this
    information without blocking, so i can retrieve other data from the
    line i read in then put this in a GUI interface.


    readline() blocks until the newline character is read, but when i use
    read(X) where X is a number of bytes then it doesnt block(expected
    functionality) but i dont know how many bytes the line will be and its
    not constant so i cant use this too.

    Any ideas of solving this problem?


    Cheers
     
    placid, Jul 28, 2006
    #1
    1. Advertising

  2. placid

    Simon Forman Guest

    placid wrote:
    > Hi all,
    >
    > I have been looking into non-blocking read (readline) operations on
    > PIPES on windows XP and there seems to be no way of doing this. Ive
    > read that you could use a Thread to read from the pipe, but if you
    > still use readline() wouldnt the Thread block too?


    Yes it will, but that's ok. In this case that's what it's for. While
    the thread waits for the readline(), the rest of your program continues
    to carry on.

    >
    > What i need to do is, create a process using subprocess.Popen, where
    > the subprocess outputs information on one line (but the info
    > continuesly changes and its always on the same line) and read this
    > information without blocking, so i can retrieve other data from the
    > line i read in then put this in a GUI interface.
    >
    >
    > readline() blocks until the newline character is read, but when i use
    > read(X) where X is a number of bytes then it doesnt block(expected
    > functionality) but i dont know how many bytes the line will be and its
    > not constant so i cant use this too.
    >
    > Any ideas of solving this problem?
    >
    >
    > Cheers
     
    Simon Forman, Jul 28, 2006
    #2
    1. Advertising

  3. On 27 Jul 2006 22:26:25 -0700, "placid" <> declaimed the
    following in comp.lang.python:

    >
    > readline() blocks until the newline character is read, but when i use
    > read(X) where X is a number of bytes then it doesnt block(expected
    > functionality) but i dont know how many bytes the line will be and its
    > not constant so i cant use this too.
    >
    > Any ideas of solving this problem?
    >

    Use a thread that reads one character at a time; when it sees
    whatever signals "end of line" (it sounds like you're reading a progress
    bar implemented via <cr>overwrite). Combine the characters into a
    string, return the string to the main program via a queue.

    If there is no such "end of line" character, but there IS a
    noticeable delay between "writes", a more complex method might suffice
    -- in which one thread does the byte reads, setting a time value on each
    read; a related thread then does a sleep() loop, checking the "last read
    time" against the pause length -- if close enough to the pause duration,
    combine and return...

    Alternatively, take a good old style terminal keyboard (a VT100
    Tempest-rated model should be ideal), and use it to beat Bill Gates over
    the head until he agrees to push a high-priority upgrade to the command
    line I/O system... or makes files work with select() (so you can combine
    the time-out with the byte read)

    --
    Wulfraed Dennis Lee Bieber KD6MOG

    HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff: )
    HTTP://www.bestiaria.com/
     
    Dennis Lee Bieber, Jul 28, 2006
    #3
  4. "placid" <> wrote in message
    news:

    > Hi all,
    >
    > I have been looking into non-blocking read (readline) operations on
    > PIPES on windows XP and there seems to be no way of doing this. Ive
    > read that you could use a Thread to read from the pipe, but if you
    > still use readline() wouldnt the Thread block too?


    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

    > What i need to do is, create a process using subprocess.Popen, where
    > the subprocess outputs information on one line (but the info
    > continuesly changes and its always on the same line) and read this
    > information without blocking, so i can retrieve other data from the
    > line i read in then put this in a GUI interface.
    >
    >
    > readline() blocks until the newline character is read, but when i use
    > read(X) where X is a number of bytes then it doesnt block(expected
    > functionality) but i dont know how many bytes the line will be and its
    > not constant so i cant use this too.
    >
    > Any ideas of solving this problem?
    >
    >
    > Cheers


    I realized something very similar to what you described in

    http://sourceforge.net/projects/bestgui

    - the subprocess2.py module realizes the non blocking I/O
    - the outputparser.py module processes the output from the controlled
    process and updates the progress-bar, the status-bar and the log
    messages in the GUI. Incomplete lines are stored in a buffer and
    processed at the next read.

    ciao

    --
    Antonio Valentino



    --
    Posted via Mailgate.ORG Server - http://www.Mailgate.ORG
     
    Antonio Valentino, Jul 29, 2006
    #4
  5. placid

    placid Guest

    Dennis Lee Bieber wrote:
    > On 27 Jul 2006 22:26:25 -0700, "placid" <> declaimed the
    > following in comp.lang.python:
    >
    > >
    > > readline() blocks until the newline character is read, but when i use
    > > read(X) where X is a number of bytes then it doesnt block(expected
    > > functionality) but i dont know how many bytes the line will be and its
    > > not constant so i cant use this too.
    > >
    > > Any ideas of solving this problem?
    > >

    > Use a thread that reads one character at a time; when it sees
    > whatever signals "end of line" (it sounds like you're reading a progress
    > bar implemented via <cr>overwrite). Combine the characters into a
    > string, return the string to the main program via a queue.
    >


    Yes it is a progress bar implemented via <cr> overwrite. I will try
    this.

    > If there is no such "end of line" character, but there IS a
    > noticeable delay between "writes", a more complex method might suffice
    > -- in which one thread does the byte reads, setting a time value on each
    > read; a related thread then does a sleep() loop, checking the "last read
    > time" against the pause length -- if close enough to the pause duration,
    > combine and return...


    i dont think there is a noticeable delay between "writes".


    > Alternatively, take a good old style terminal keyboard (a VT100
    > Tempest-rated model should be ideal), and use it to beat Bill Gates over
    > the head until he agrees to push a high-priority upgrade to the command
    > line I/O system... or makes files work with select() (so you can combine
    > the time-out with the byte read)


    ;) Tsk Tsk
     
    placid, Jul 31, 2006
    #5
  6. On 30 Jul 2006 16:22:34 -0700, "placid" <> declaimed the
    following in comp.lang.python:

    >
    > ;) Tsk Tsk


    Have you ever seen a Tempest VT-100? Lead shielding on the monitor
    FACE... Turn the brightness all the way up and it still looked dim.
    Fiber optic to the computer. And a shield keyboard about three inches
    thick and weighing 5 lbs.

    All to keep "them" from using radio equipment to pick up key strokes
    or the display scan line radiation (use an external vertical/horizontal
    sweep generator to an oscilloscope, and a high-gain amplifier to detect
    the brightness variation of the CRT).
    --
    Wulfraed Dennis Lee Bieber KD6MOG

    HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff: )
    HTTP://www.bestiaria.com/
     
    Dennis Lee Bieber, Jul 31, 2006
    #6
  7. placid

    placid Guest

    Dennis Lee Bieber wrote:
    > On 30 Jul 2006 16:22:34 -0700, "placid" <> declaimed the
    > following in comp.lang.python:
    >
    > >
    > > ;) Tsk Tsk

    >
    > Have you ever seen a Tempest VT-100? Lead shielding on the monitor
    > FACE... Turn the brightness all the way up and it still looked dim.
    > Fiber optic to the computer. And a shield keyboard about three inches
    > thick and weighing 5 lbs.
    >
    > All to keep "them" from using radio equipment to pick up key strokes
    > or the display scan line radiation (use an external vertical/horizontal
    > sweep generator to an oscilloscope, and a high-gain amplifier to detect
    > the brightness variation of the CRT).



    Nope, i have not seen one.
     
    placid, Jul 31, 2006
    #7
  8. placid

    Paul Du Bois Guest

    placid wrote:
    > What i need to do is, create a process using subprocess.Popen, where
    > the subprocess outputs information on one line (but the info
    > continuesly changes and its always on the same line) and read this
    > information without blocking, so i can retrieve other data from the
    > line i read in then put this in a GUI interface.


    > readline() blocks until the newline character is read, but when i use
    > read(X) where X is a number of bytes then it doesnt block(expected
    > functionality) but i dont know how many bytes the line will be and its
    > not constant so i cant use this too.


    I wrote something for this the other day. The class has a getline()
    method, which either returns a line or raises an exception instead of
    blocking. It also raises an exception instead of returning EOF.

    My use case had me reading from multiple processes at once; since
    select() doesn't work on files in win32, I had to get a little cheesy.
    I've appended the function that implements that use case, for
    reference.

    The central idea is to use PeekNamedPipe to figure out what's in the
    pipe. You can then read that data without fear of blocking. I wrote it
    quickly, therefore the code is a little embarassing, but... hopefully
    useful all the same.

    .. class NoLineError(Exception): pass
    .. class NoMoreLineError(Exception): pass
    .. class liner(object):
    .. """Helper class for multi_readlines."""
    .. def __init__(self, f):
    .. self.fd = f.fileno()
    .. self.osf = msvcrt.get_osfhandle(self.fd)
    .. self.buf = ''
    ..
    .. def getline(self):
    .. """Returns a line of text, or raises NoLineError, or
    NoMoreLineError."""
    .. try:
    .. data, avail, _ = win32pipe.PeekNamedPipe(self.osf, 0)
    .. except pywintypes.error:
    .. # Pipe closed: give up what we have, then that's it
    .. if self.buf:
    .. ret, self.buf = self.buf, None
    .. return ret
    .. else:
    .. raise NoMoreLineError
    .. if avail:
    .. self.buf += os.read(self.fd, avail)
    ..
    .. idx = self.buf.find('\n')
    .. if idx >= 0:
    .. ret, self.buf = self.buf[:idx+1], self.buf[idx+1:]
    .. return ret
    .. else:
    .. raise NoLineError
    ..
    ..
    .. def multi_readlines(fs):
    .. """Read lines from |fs|, a list of file objects.
    .. The lines come out in arbitrary order, depending on which files
    .. have output available first."""
    .. if type(fs) not in (list, tuple):
    .. raise Exception("argument must be a list.")
    .. objs = [liner(f) for f in fs]
    .. for i,obj in enumerate(objs): obj._index = i
    .. while objs:
    .. for i,obj in enumerate(objs):
    .. try:
    .. yield (obj._index, obj.getline())
    .. except NoLineError:
    .. pass
    .. except NoMoreLineError:
    .. del objs
    .. break # Because we mutated the array
     
    Paul Du Bois, Aug 4, 2006
    #8
  9. placid

    Durumdara Guest

    Hi !

    Sorry, but I want to share my experiences. I hope this help to you.

    I think that specialized MSWindows based services too complicated. They have
    to many bug possibilites.
    So I trying with normal, "in python accessable" pipes. I see that with
    flush(), and some of the bintotext tricks I can use the
    subprocess/masterprocess communication.
    Ok, this is not asynchronous. I can send some job to sp(s), and I can
    receive the report from sp(s).
    But with threading I can create non-blocking communication.

    You can see in the example: the PipeBPPThr define a pipe based process-pool
    thread.
    This can communicate with a subprocess, can send/receive jobs, etc.

    If you collect these threads, and write a process pool object, you can
    handle all of the communications with one object.

    I hope to these examples can help to you.

    If not, you can try with wm_copydata messages in Windows.
    http://msdn.microsoft.com/library/d...rface/dataexchange/datacopy/usingdatacopy.asp
    This way of data exchanging is based on the message handling/sending.

    dd
     
    Durumdara, Aug 4, 2006
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hendra Gunawan
    Replies:
    1
    Views:
    13,065
    Allan Herriman
    Apr 8, 2004
  2. Andre Kelmanson

    blocking i/o vs. non blocking i/o (performance)

    Andre Kelmanson, Oct 10, 2003, in forum: C Programming
    Replies:
    3
    Views:
    996
    Valentin Tihomirov
    Oct 12, 2003
  3. Serge Savoie
    Replies:
    4
    Views:
    292
    Serge Savoie
    Oct 1, 2008
  4. Stuart Moore

    pipe - non blocking read? (fork/Win32)

    Stuart Moore, Jul 2, 2003, in forum: Perl Misc
    Replies:
    6
    Views:
    356
    Bryan Castillo
    Jul 7, 2003
  5. Kevin Walzer
    Replies:
    2
    Views:
    146
    Terry Reedy
    Dec 22, 2012
Loading...

Share This Page