Re: read stdout/stderr without blocking

Discussion in 'Python' started by Adriaan Renting, Sep 16, 2005.

  1. Great reply,

    I had just mixed Pexpect and subProcess code until I'd got something that worked, you can actually explain my code better a I can myself. I find it quite cumbersome to read stdout/strerr separately, and to be able to write to stdin in reaction to either of them, but at least on Linux you can get it to work. My NON_BLOCKing command might be unnecesary, I'll try without it.

    The OP seemed interested on how to do this on Windows, but I've yet to see an answer on that one I think.

    Thank you for the reply.

    Adriaan Renting

    |>>>Donn Cave <> 09/15/05 6:19 pm >>>
    |In article <>,
    |Peter Hansen <> wrote:
    |
    |>Jacek Pop³awski wrote:
    |>>Grant Edwards wrote:
    |>>
    |>>>On 2005-09-12, Jacek Pop?awski <> wrote:
    |>>>
    |>>>>> ready = select.select(tocheck, [], [], 0.25) ##continues
    |>>>>>after 0.25s
    |>>>>> for file in ready[0]:
    |>>>>> try:
    |>>>>> text = os.read(file, 1024)
    |>>>>
    |>>>>
    |>>>>How do you know here, that you should read 1024 characters?
    |>>>>What will happen when output is shorter?
    |>>>
    |>>>It will return however much data is available.
    |>>
    |>>My tests showed, that it will block.
    |>
    |>Not if you use non-blocking sockets, as I believe you are expected to
    |>when using select().
    |
    |On the contrary, you need non-blocking sockets only if
    |you don't use select. select waits until a read [write]
    |would not block - it's like "if dict.has_key(x):" instead of
    |"try: val = dict[x] ; except KeyError:". I suppose you
    |knew that, but have read some obscure line of reasoning
    |that makes non-blocking out to be necessary anyway.
    |Who knows, but it certainly isn't in this case.
    |
    |I don't recall the beginning of this thread, so I'm not sure
    |if this is the usual wretched exercise of trying to make this
    |work on both UNIX and Windows, but there are strong signs
    |of the usual confusion over os.read (a.k.a. posix.read), and
    |file object read. Let's hopefully forget about Windows for
    |the moment.
    |
    |The above program looks fine to me, but it will not work
    |reliably if file object read() is substituted for os.read().
    |In this case, C library buffering will read more than 1024
    |bytes if it can, and then that data will not be visible to
    |select(), so there's no guarantee it will return in a timely
    |manner even though the next read() would return right
    |away. Reading one byte at a time won't resolve this problem,
    |obviously it will only make it worse. The only reason to
    |read one byte at a time is for data-terminated read semantics,
    |specifically readline(), in an unbuffered file. That's what
    |happens -- at the system call level, where it's expensive --
    |when you turn off stdio buffering and then call readline().
    |
    |In the C vs. Python example, read() is os.read(), and file
    |object read() is fread(); so of course, C read() works
    |where file object read() doesn't.
    |
    |Use select, and os.read (and UNIX) and you can avoid blocking
    |on a pipe. That's essential if as I am reading it there are supposed
    |to be two separate pipes from the same process, since if one is
    |allowed to fill up, that process will block, causing a deadlock if
    |the reading process blocks on the other pipe.
    |
    |Hope I'm not missing anything here. I just follow this group
    |to answer this question over and over, so after a while it
    |gets sort of automatic.
    |
    | Donn Cave,
    Adriaan Renting, Sep 16, 2005
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?ISO-8859-2?Q?Jacek_Pop=B3awski?=

    read stdout/stderr without blocking

    =?ISO-8859-2?Q?Jacek_Pop=B3awski?=, Sep 12, 2005, in forum: Python
    Replies:
    1
    Views:
    3,985
    =?ISO-8859-2?Q?Jacek_Pop=B3awski?=
    Sep 13, 2005
  2. Adriaan Renting

    Re: read stdout/stderr without blocking

    Adriaan Renting, Sep 12, 2005, in forum: Python
    Replies:
    1
    Views:
    330
    =?UTF-8?B?SmFjZWsgUG9wxYJhd3NraQ==?=
    Sep 12, 2005
  3. Adriaan Renting

    Re: read stdout/stderr without blocking

    Adriaan Renting, Sep 12, 2005, in forum: Python
    Replies:
    10
    Views:
    671
    =?ISO-8859-2?Q?Jacek_Pop=B3awski?=
    Sep 16, 2005
  4. Adriaan Renting

    Re: read stdout/stderr without blocking

    Adriaan Renting, Sep 14, 2005, in forum: Python
    Replies:
    0
    Views:
    416
    Adriaan Renting
    Sep 14, 2005
  5. George C. Demetros

    BLOCKING and STDOUT/STDERR

    George C. Demetros, Jan 6, 2004, in forum: Perl Misc
    Replies:
    1
    Views:
    94
    Ben Morrow
    Jan 6, 2004
Loading...

Share This Page