Re: read stdout/stderr without blocking

Discussion in 'Python' started by Adriaan Renting, Sep 12, 2005.

  1. I was not aware you were using Windows, you might need to find something similar to select and pty that works in Windows or maybe go though Cygwin, I don't know. I'm on Linux, the only help I can offer is showing you my working code, that's a mix of Pexpect, subProcess and Parseltongue.
    I'm not sure if this is 100% correct, it just happens to work and might help you in solving a similar problem:

    ---- in spawn()
    (self._errorpipe_end, self._errorpipe_front) = os.pipe() ## need to handle stderr separate from stdout
    try:
    (self._pid, self._child_fd) = pty.fork()
    except OSError, e:
    raise Exception ('fork failed')
    if self._pid == 0: ## the new client
    try:
    os.dup2(self._errorpipe_front, 2) ## we hardcoded assume stderr of the pty has fd 2
    os.close(self._errorpipe_end)
    os.close(self._errorpipe_front) ## close what we don't need
    os.execvp(self.task, self.inputs)
    except:
    sys.stderr.write('Process could not be started: ' + self.task)
    os._exit(1)
    else: ## the parent
    os.close(self._errorpipe_front) ## close what we don't need
    fcntl.fcntl(self._child_fd, fcntl.F_SETFL, os.O_NONBLOCK)

    ---- in handle_messages()
    tocheck=[]
    if not self._fd_eof:
    tocheck.append(self._child_fd)
    if not self._pipe_eof:
    tocheck.append(self._errorpipe_end)
    ready = select.select(tocheck, [], [], 0.25) ##continues after 0.25s
    for file in ready[0]:
    try:
    text = os.read(file, 1024)
    except: ## probalby Input/Output error because the child died
    text = ''
    if text:
    for x in self._expect:
    if x[0] in text: ## we need to do something if we see this text
    returntext = x[1](text)
    if returntext:
    os.write(file, returntext)
    self.handle_text(text)
    else:
    if file == self._child_fd:
    self._fd_eof = 1
    elif file == self._errorpipe_end:
    self._pipe_eof = 1
    return 1
    if self._fd_eof or self._pipe_eof: # should be an and not an or, but python 2.3.5 doesn't like it
    return 0
    if len(ready[0]) == 0: ## no data in 0.25 second timeout
    return 1
    return 0

    ---- in finish()
    (pid, status) = os.waitpid(self._pid, os.WNOHANG) ## clean up the zombie
    assert(pid == self._pid)
    if os.WIFEXITED(status) or os.WIFSIGNALED(status):
    self._pid = 0
    self.exitstatus = status
    assert(self.finished())
    del self._pid
    os.close(self._child_fd)
    os.close(self._errorpipe_end)


    |>>>Jacek Pop*awski <> 09/12/05 1:14 pm >>>
    |Adriaan Renting wrote:
    |>Check out the select module, for an example on how to use it:
    |>pexpect.sourceforge.net
    |
    |Two problems:
    |- it won't work on Windows (Cygwin)
    |- how much data should I read after select? 1 character? Can it block if
    |I read 2 characters?
    |--
    |http://mail.python.org/mailman/listinfo/python-list
     
    Adriaan Renting, Sep 12, 2005
    #1
    1. Advertising

  2. > ready = select.select(tocheck, [], [], 0.25) ##continues after 0.25s
    > for file in ready[0]:
    > try:
    > text = os.read(file, 1024)


    How do you know here, that you should read 1024 characters?
    What will happen when output is shorter?
     
    =?UTF-8?B?SmFjZWsgUG9wxYJhd3NraQ==?=, Sep 12, 2005
    #2
    1. Advertising

  3. On 2005-09-12, Jacek Pop?awski <> wrote:
    >> ready = select.select(tocheck, [], [], 0.25) ##continues after 0.25s
    >> for file in ready[0]:
    >> try:
    >> text = os.read(file, 1024)

    >
    > How do you know here, that you should read 1024 characters?
    > What will happen when output is shorter?


    It will return however much data is available.

    --
    Grant Edwards grante Yow! I'm a fuschia bowling
    at ball somewhere in Brittany
    visi.com
     
    Grant Edwards, Sep 12, 2005
    #3
  4. Grant Edwards wrote:
    > On 2005-09-12, Jacek Pop?awski <> wrote:
    >
    >>> ready = select.select(tocheck, [], [], 0.25) ##continues after 0.25s
    >>> for file in ready[0]:
    >>> try:
    >>> text = os.read(file, 1024)

    >>
    >>How do you know here, that you should read 1024 characters?
    >>What will happen when output is shorter?

    >
    >
    > It will return however much data is available.


    My tests showed, that it will block.
     
    =?ISO-8859-2?Q?Jacek_Pop=B3awski?=, Sep 13, 2005
    #4
  5. Adriaan Renting

    Peter Hansen Guest

    Jacek Pop³awski wrote:
    > Grant Edwards wrote:
    >
    >> On 2005-09-12, Jacek Pop?awski <> wrote:
    >>
    >>>> ready = select.select(tocheck, [], [], 0.25) ##continues
    >>>> after 0.25s
    >>>> for file in ready[0]:
    >>>> try:
    >>>> text = os.read(file, 1024)
    >>>
    >>>
    >>> How do you know here, that you should read 1024 characters?
    >>> What will happen when output is shorter?

    >>
    >> It will return however much data is available.

    >
    > My tests showed, that it will block.


    Not if you use non-blocking sockets, as I believe you are expected to
    when using select().

    -Peter
     
    Peter Hansen, Sep 13, 2005
    #5
  6. On 2005-09-13, Jacek Pop³awski <> wrote:

    >>>> ready = select.select(tocheck, [], [], 0.25) ##continues after 0.25s
    >>>> for file in ready[0]:
    >>>> try:
    >>>> text = os.read(file, 1024)
    >>>
    >>>How do you know here, that you should read 1024 characters?
    >>>What will happen when output is shorter?

    >>
    >> It will return however much data is available.

    >
    > My tests showed, that it will block.


    You're right. I must have been remembering the behavior of a
    network socket. Apparently, you're supposed to read a single
    byte and then call select() again. That seems pretty lame.

    --
    Grant Edwards grante Yow! Psychoanalysis?? I
    at thought this was a nude
    visi.com rap session!!!
     
    Grant Edwards, Sep 13, 2005
    #6
  7. Grant Edwards wrote:
    > You're right. I must have been remembering the behavior of a
    > network socket. Apparently, you're supposed to read a single
    > byte and then call select() again. That seems pretty lame.


    I created another thread with single read(), it works, as long as I have
    only one PIPE (i.e. stderr is redirected into stdout).
    I wonder is it Python limitation or systems one (I need portable solution)?
     
    =?UTF-8?B?SmFjZWsgUG9wxYJhd3NraQ==?=, Sep 13, 2005
    #7
  8. On 2005-09-13, Jacek Pop?awski <> wrote:
    > Grant Edwards wrote:
    >> You're right. I must have been remembering the behavior of a
    >> network socket. Apparently, you're supposed to read a single
    >> byte and then call select() again. That seems pretty lame.

    >
    > I created another thread with single read(), it works, as long
    > as I have only one PIPE (i.e. stderr is redirected into
    > stdout). I wonder is it Python limitation or systems one (I
    > need portable solution)?


    Not sure what you mean. Here is my test program that blocks on
    the read(1024) call:

    #!/usr/bin/python
    import os,select

    p = os.popen("while sleep 2; do date; done","r")
    print p

    while 1:
    r,w,e = select.select([p],[],[],1)
    if r:
    d = r[0].read(1024)
    print len(d),repr(d)
    else:
    print "timeout"

    It also blocks if the call is changed to read(). This seems
    pretty counter-intuitive, since that's not the way read()
    usually works on pipes.

    Here's the corresponding C program that works as I
    expected (read(1024) returns available data):

    #include <unistd.h>
    #include <stdlib.h>
    #include <stdio.h>
    #include <sys/select.h>

    unsigned char buffer[1024];
    int main(void)
    {
    fd_set readfds, writefds, exceptfds;
    struct timeval tv;
    FILE *fp;
    int fd;

    fp = popen("while sleep 2; do date; done","r");
    if (!fp)
    {
    perror("popen");
    exit(1);
    }
    fd = fileno(fp);

    FD_ZERO(&readfds);
    FD_ZERO(&writefds);
    FD_ZERO(&exceptfds);

    while (1)
    {
    int s;
    FD_SET(fd,&readfds);
    tv.tv_sec = 1;
    tv.tv_usec = 0;
    s = select(fd+1,&readfds,&writefds,&exceptfds,&tv);
    if (s==0)
    printf("timeout\n");
    else if (s<0)
    {
    perror("select");
    exit(2);
    }
    else
    {
    if FD_ISSET(fd,&readfds)
    {
    int n = read(fd,buffer,(sizeof buffer)-1);
    buffer[n] = '\0';
    printf("read %d: '%s'\n",n,buffer);
    }
    }
    }
    }


    --
    Grant Edwards grante Yow! Does that mean
    at I'm not a well-adjusted
    visi.com person??
     
    Grant Edwards, Sep 13, 2005
    #8
  9. Adriaan Renting

    Donn Cave Guest

    In article <>,
    Peter Hansen <> wrote:

    > Jacek Pop³awski wrote:
    > > Grant Edwards wrote:
    > >
    > >> On 2005-09-12, Jacek Pop?awski <> wrote:
    > >>
    > >>>> ready = select.select(tocheck, [], [], 0.25) ##continues
    > >>>> after 0.25s
    > >>>> for file in ready[0]:
    > >>>> try:
    > >>>> text = os.read(file, 1024)
    > >>>
    > >>>
    > >>> How do you know here, that you should read 1024 characters?
    > >>> What will happen when output is shorter?
    > >>
    > >> It will return however much data is available.

    > >
    > > My tests showed, that it will block.

    >
    > Not if you use non-blocking sockets, as I believe you are expected to
    > when using select().


    On the contrary, you need non-blocking sockets only if
    you don't use select. select waits until a read [write]
    would not block - it's like "if dict.has_key(x):" instead of
    "try: val = dict[x] ; except KeyError:". I suppose you
    knew that, but have read some obscure line of reasoning
    that makes non-blocking out to be necessary anyway.
    Who knows, but it certainly isn't in this case.

    I don't recall the beginning of this thread, so I'm not sure
    if this is the usual wretched exercise of trying to make this
    work on both UNIX and Windows, but there are strong signs
    of the usual confusion over os.read (a.k.a. posix.read), and
    file object read. Let's hopefully forget about Windows for
    the moment.

    The above program looks fine to me, but it will not work
    reliably if file object read() is substituted for os.read().
    In this case, C library buffering will read more than 1024
    bytes if it can, and then that data will not be visible to
    select(), so there's no guarantee it will return in a timely
    manner even though the next read() would return right
    away. Reading one byte at a time won't resolve this problem,
    obviously it will only make it worse. The only reason to
    read one byte at a time is for data-terminated read semantics,
    specifically readline(), in an unbuffered file. That's what
    happens -- at the system call level, where it's expensive --
    when you turn off stdio buffering and then call readline().

    In the C vs. Python example, read() is os.read(), and file
    object read() is fread(); so of course, C read() works
    where file object read() doesn't.

    Use select, and os.read (and UNIX) and you can avoid blocking
    on a pipe. That's essential if as I am reading it there are supposed
    to be two separate pipes from the same process, since if one is
    allowed to fill up, that process will block, causing a deadlock if
    the reading process blocks on the other pipe.

    Hope I'm not missing anything here. I just follow this group
    to answer this question over and over, so after a while it
    gets sort of automatic.

    Donn Cave,
     
    Donn Cave, Sep 15, 2005
    #9
  10. Adriaan Renting

    Peter Hansen Guest

    Donn Cave wrote:
    > Peter Hansen <> wrote:
    >>Jacek Pop³awski wrote:
    >>>My tests showed, that it will block.

    >>
    >>Not if you use non-blocking sockets, as I believe you are expected to
    >>when using select().

    >
    > On the contrary, you need non-blocking sockets only if
    > you don't use select. select waits until a read [write]
    > would not block - it's like "if dict.has_key(x):" instead of
    > "try: val = dict[x] ; except KeyError:". I suppose you
    > knew that, but have read some obscure line of reasoning
    > that makes non-blocking out to be necessary anyway.


    No, no, I suspect I was just plain wrong. I think I felt a twinge of
    suspicion even as I wrote it, but I went ahead and hit Send anyway
    perhaps because I'd had two nights with little sleep (if I need an
    excuse to be wrong, that is, which I don't :) ).

    > Who knows, but it certainly isn't in this case.


    Thanks for straightening things out for the record.

    -Peter
     
    Peter Hansen, Sep 16, 2005
    #10
  11. Donn Cave wrote:
    > I don't recall the beginning of this thread, so I'm not sure
    > if this is the usual wretched exercise of trying to make this
    > work on both UNIX and Windows,


    It is used in "test framework" which runs on Linux, Windows (Cygwin) and
    QNX. I can't forget about Windows.
     
    =?ISO-8859-2?Q?Jacek_Pop=B3awski?=, Sep 16, 2005
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?ISO-8859-2?Q?Jacek_Pop=B3awski?=

    read stdout/stderr without blocking

    =?ISO-8859-2?Q?Jacek_Pop=B3awski?=, Sep 12, 2005, in forum: Python
    Replies:
    1
    Views:
    3,996
    =?ISO-8859-2?Q?Jacek_Pop=B3awski?=
    Sep 13, 2005
  2. Adriaan Renting

    Re: read stdout/stderr without blocking

    Adriaan Renting, Sep 12, 2005, in forum: Python
    Replies:
    1
    Views:
    339
    =?UTF-8?B?SmFjZWsgUG9wxYJhd3NraQ==?=
    Sep 12, 2005
  3. Adriaan Renting

    Re: read stdout/stderr without blocking

    Adriaan Renting, Sep 14, 2005, in forum: Python
    Replies:
    0
    Views:
    429
    Adriaan Renting
    Sep 14, 2005
  4. Adriaan Renting

    Re: read stdout/stderr without blocking

    Adriaan Renting, Sep 16, 2005, in forum: Python
    Replies:
    0
    Views:
    385
    Adriaan Renting
    Sep 16, 2005
  5. George C. Demetros

    BLOCKING and STDOUT/STDERR

    George C. Demetros, Jan 6, 2004, in forum: Perl Misc
    Replies:
    1
    Views:
    102
    Ben Morrow
    Jan 6, 2004
Loading...

Share This Page