Nonblocking IO read

Discussion in 'Ruby' started by srobertjames@gmail.com, Oct 31, 2006.

  1. Guest

    How can I perform a nonblocking IO read? That is, read whatever is
    there to read and return. If there's nothing yet, just return.

    Failing that, is there a way to manually check if there's anything to
    read or the read will block?

    (Note that I'm developing for both 'nix and Windows.)
    , Oct 31, 2006
    #1
    1. Advertising

  2. Guest

    On Wed, 1 Nov 2006 wrote:

    > How can I perform a nonblocking IO read? That is, read whatever is
    > there to read and return. If there's nothing yet, just return.
    >
    > Failing that, is there a way to manually check if there's anything to
    > read or the read will block?
    >
    > (Note that I'm developing for both 'nix and Windows.)


    it can't be done. search the archives, this sort of thing almost always
    indicates a design flaw. for instance - what will your program do if there is
    no input?

    regards.

    -a
    --
    my religion is very simple. my religion is kindness. -- the dalai lama
    , Oct 31, 2006
    #2
    1. Advertising

  3. Tom Pollard Guest

    On Oct 31, 2006, at 6:23 PM, wrote:
    > On Wed, 1 Nov 2006 wrote:
    >> How can I perform a nonblocking IO read? That is, read whatever
    >> is there to read and return. If there's nothing yet, just return.


    That's what the Unix read() function does ("man 2 read"), available
    in Ruby as IO#sysread(). The ri documentation doesn't explain much
    about how to use Ruby's sysread, unfortunately.

    >> Failing that, is there a way to manually check if there's anything
    >> to read or the read will block?
    >>
    >> (Note that I'm developing for both 'nix and Windows.)


    The select() function (Kernel#select, in Ruby) is the standard way to
    do IO multiplexing under Unix. You give it a list of file handles on
    which you're waiting to be able to read or write, and it returns when
    one or more of them is ready. Windows provides a select() function,
    too, but it only supports sockets (not pipes or ordinary file IO.)
    The Win32 API generally doesn't provide non-blocking IO methods,
    because it's assumed you'll use threads if you want to do something
    else while waiting for IO.

    > it can't be done. search the archives, this sort of thing almost
    > always
    > indicates a design flaw. for instance - what will your program do
    > if there is
    > no input?


    Check other IO channels, wait for other sorts of events, do work -
    whatever. Where do come from asserting that the use of non-blocking
    IO represents a "design flaw"?

    Tom
    Tom Pollard, Nov 1, 2006
    #3
  4. Guest

    wrote:
    > On Wed, 1 Nov 2006 wrote:
    >
    > > How can I perform a nonblocking IO read? That is, read whatever is
    > > there to read and return. If there's nothing yet, just return.
    > >
    > > Failing that, is there a way to manually check if there's anything to
    > > read or the read will block?
    > >
    > > (Note that I'm developing for both 'nix and Windows.)

    >
    > it can't be done. search the archives, this sort of thing almost always
    > indicates a design flaw. for instance - what will your program do if there is
    > no input?


    I'm running an external command (using popen4 - yes, it is available on
    Windows also). I capture stdout and stderr. "what will your program do
    if there is no input?" - I assume you mean output - Well, I hope there
    is no stderr output. If there is, I'd like to capture it and feed it
    to Logger.

    Likewise, if there is a problem, and there's no stdout, then I don't
    want to wait for ever - I want to throw an exception.
    , Nov 1, 2006
    #4
  5. Guest

    In the interests of promoting creativity, I'll ask the question without
    any prior assumptions:

    I'm using popen4 (great gem, btw) to run an external command and
    capture stdout and stderr. If one (or both) of those is empty, I don't
    want to hang. Just keep on moving. In fact, stderr will normally be
    empty.

    How could I do this?

    (I'm supporting 'nix and Windows)
    , Nov 1, 2006
    #5
  6. Guest

    On Wed, 1 Nov 2006, Tom Pollard wrote:

    >
    > On Oct 31, 2006, at 6:23 PM, wrote:
    >> On Wed, 1 Nov 2006 wrote:
    >>> How can I perform a nonblocking IO read? That is, read whatever is there
    >>> to read and return. If there's nothing yet, just return.

    >
    > That's what the Unix read() function does ("man 2 read"), available in Ruby
    > as IO#sysread(). The ri documentation doesn't explain much about how to use
    > Ruby's sysread, unfortunately.


    no, that blocks. try this:

    harp:~ > ruby -e 'STDIN.sysread(1)'

    it blocks. the OP wants a non-blocking call. eg

    harp:~ > ruby -r io/nonblock -e 'STDIN.nonblock{ STDIN.sysread(1) }'
    -e:1:in `sysread': Resource temporarily unavailable (Errno::EAGAIN)
    from -e:1
    from /home/ahoward//lib/ruby/1.8/io/nonblock.rb:19:in `nonblock'
    from -e:1


    > The select() function (Kernel#select, in Ruby) is the standard way to do IO
    > multiplexing under Unix. You give it a list of file handles on which you're
    > waiting to be able to read or write, and it returns when one or more of them
    > is ready. Windows provides a select() function, too, but it only supports
    > sockets (not pipes or ordinary file IO.) The Win32 API generally doesn't
    > provide non-blocking IO methods, because it's assumed you'll use threads if
    > you want to do something else while waiting for IO.


    right. but ruby's thread are green and, on windows, block the entire process
    when waiting on IO so not an option here.

    > Check other IO channels, wait for other sorts of events, do work - whatever.
    > Where do come from asserting that the use of non-blocking IO represents a
    > "design flaw"?


    because nine times out of ten code has nothing to do if no input is available.
    if it does that's another story, but then there are basically two options:

    - use select or an event abstraction in one giant loop. if you do this you are
    going to block anyhow, only based on all inputs, not just one.

    - use threads, only you can't if you're wanting ruby code to run nb in windows.


    i'm not saying there is never a case for nbio - i use it myself - just that,
    unless a developer has a good reason to be doing something else it's not
    required and, as a search of ruby-talk will show, that is often the case.

    regards.

    -a
    --
    my religion is very simple. my religion is kindness. -- the dalai lama
    , Nov 1, 2006
    #6
  7. Guest

    On Wed, 1 Nov 2006 wrote:

    > I'm running an external command (using popen4 - yes, it is available on
    > Windows also). I capture stdout and stderr. "what will your program do if
    > there is no input?" - I assume you mean output - Well, I hope there is no
    > stderr output. If there is, I'd like to capture it and feed it to Logger.
    >
    > Likewise, if there is a problem, and there's no stdout, then I don't want to
    > wait for ever - I want to throw an exception.


    so, basically you want to timeout on a read right? in unix you can just use

    require 'timeout'

    Timeout::timeout(n) do
    Open4.popen4 ...
    ...
    ...
    end

    but, this is implimented via a thread so i think the read is going to block
    your entire process in windows... you might try it.

    -a
    --
    my religion is very simple. my religion is kindness. -- the dalai lama
    , Nov 1, 2006
    #7
  8. Not very attractive...

    Back to the original question: there is no simple way to read
    nonblocking, or see if any output is available ?!?!


    wrote:
    > On Wed, 1 Nov 2006 wrote:
    >
    > > I'm running an external command (using popen4 - yes, it is available on
    > > Windows also). I capture stdout and stderr. "what will your program do if
    > > there is no input?" - I assume you mean output - Well, I hope there is no
    > > stderr output. If there is, I'd like to capture it and feed it to Logger.
    > >
    > > Likewise, if there is a problem, and there's no stdout, then I don't want to
    > > wait for ever - I want to throw an exception.

    >
    > so, basically you want to timeout on a read right? in unix you can just use
    >
    > require 'timeout'
    >
    > Timeout::timeout(n) do
    > Open4.popen4 ...
    > ...
    > ...
    > end
    >
    > but, this is implimented via a thread so i think the read is going to block
    > your entire process in windows... you might try it.
    >
    > -a
    > --
    > my religion is very simple. my religion is kindness. -- the dalai lama
    S. Robert James, Nov 1, 2006
    #8
  9. Ryan Davis Guest

    On Oct 31, 2006, at 5:30 PM, S. Robert James wrote:

    > Not very attractive...
    >
    > Back to the original question: there is no simple way to read
    > nonblocking, or see if any output is available ?!?!


    in io/wait:

    --- IO#ready?
    returns non-nil if input available without blocking, or nil.

    also: ri IO.read_nonblock
    Ryan Davis, Nov 1, 2006
    #9
  10. Bjorn Borud Guest

    []
    |
    | it can't be done. search the archives, this sort of thing almost
    | always indicates a design flaw. for instance - what will your
    | program do if there is no input?

    nonblocking IO is not a design-flaw; it is something that is in common
    use and is supported by most languages with a decent socket API. on
    some OS'es this is even available for file IO (although widespread
    language support seems to lag behind).

    Java for example got proper nonblocking socket IO in 1.4 (see
    JSR-000051 "New I/O APIs for the JavaTM Platform").

    for examples of use you might want to check out the Reactor pattern
    and other patterns for concurrent programming.

    -Bjørn
    Bjorn Borud, Nov 1, 2006
    #10
  11. Tom Pollard Guest

    On Oct 31, 2006, at 7:20 PM, wrote:
    > In the interests of promoting creativity, I'll ask the question
    > without
    > any prior assumptions:
    >
    > I'm using popen4 (great gem, btw) to run an external command and
    > capture stdout and stderr. If one (or both) of those is empty, I
    > don't
    > want to hang. Just keep on moving. In fact, stderr will normally be
    > empty.
    >
    > How could I do this?


    Have you tried simply reading from them? When the command
    terminates, that should close those pipes on the command's end. At
    that point, a simple read on the stdout handle, for instance, will
    return whatever output there was and then eof. If there's no output,
    the read should return immediately. Did you try that yet? Did you
    try the (Windows-oriented) example from the 'ri Open4' page? If
    you're working in Unix you could try

    status = POpen4::poen4('cat -') do |stdout, stderr, stdin|
    stdin.puts 'hello world'
    stdin.close
    puts "stdout: #{stdout.read.strip}"
    puts "stderr: #{stderr.read.strip}"
    end

    Anyway, you would only need nonblocking IO if you wanted to read bits
    of the stderr stream before the command exited, but that doesn't
    sound like what you're want.

    Tom
    Tom Pollard, Nov 1, 2006
    #11
  12. Bjorn Borud Guest

    [Tom Pollard <>]
    |
    | The select() function (Kernel#select, in Ruby) is the standard way to
    | do IO multiplexing under Unix. You give it a list of file handles on
    | which you're waiting to be able to read or write, and it returns when
    | one or more of them is ready. Windows provides a select() function,
    | too, but it only supports sockets (not pipes or ordinary file IO.)

    if memory serves, a select(2) on a file descriptors representing a
    (disk) file only had defined semantics when used to wait on mandatory
    locking and is not used for ensuring that a read(2) or write(2) will
    not block. (I might be in error since it has been a while since I
    looked into it).

    -Bjørn
    Bjorn Borud, Nov 1, 2006
    #12
  13. Tom Pollard Guest

    On Oct 31, 2006, at 9:37 PM, Tom Pollard wrote:
    > If you're working in Unix you could try
    >
    > status = POpen4::poen4('cat -') do |stdout, stderr, stdin|


    D'oh! (typo) That should have been

    status = POpen4::popen4('cat -') do |stdout, stderr, stdin|
    stdin.puts 'hello world'
    stdin.close
    puts "stdout: #{stdout.read.strip}"
    puts "stderr: #{stderr.read.strip}"
    end


    Tom
    Tom Pollard, Nov 1, 2006
    #13
  14. John Carter Guest

    On Wed, 1 Nov 2006, Tom Pollard wrote:

    > The select() function (Kernel#select, in Ruby) is the standard way to do IO
    > multiplexing under Unix.


    At a very deep level (last time I looked) thread animation and IO met
    and became the same thing in a giant select lurking in the core of the
    ruby Interpreter.

    ie. Use select / spin another thread amounted cosmically to the same
    thing. Choose whatever is easiest for you.




    John Carter Phone : (64)(3) 358 6639
    Tait Electronics Fax : (64)(3) 359 4632
    PO Box 1645 Christchurch Email :
    New Zealand

    "We have more to fear from
    The Bungling of the Incompetent
    Than from the Machinations of the Wicked." (source unknown)
    John Carter, Nov 1, 2006
    #14
  15. Guest

    On Wed, 1 Nov 2006, Bjorn Borud wrote:

    > []
    > |
    > | it can't be done. search the archives, this sort of thing almost
    > | always indicates a design flaw. for instance - what will your
    > | program do if there is no input?
    >
    > nonblocking IO is not a design-flaw; it is something that is in common
    > use and is supported by most languages with a decent socket API. on
    > some OS'es this is even available for file IO (although widespread
    > language support seems to lag behind).
    >
    > Java for example got proper nonblocking socket IO in 1.4 (see
    > JSR-000051 "New I/O APIs for the JavaTM Platform").


    but it's archaic compared to event driven frameworks like

    http://www.monkey.org/~provos/libevent/
    http://liboop.ofb.net/

    in addition it's a hack in the c lib of many oses.

    > for examples of use you might want to check out the Reactor pattern
    > and other patterns for concurrent programming.


    afaik the reactor pattern is a synchronous pattern

    http://www.artima.com/articles/io_design_patterns2.html

    not unlike the model of libevent and liboop - which are both synchronous...
    am i missing something?

    -a
    --
    my religion is very simple. my religion is kindness. -- the dalai lama
    , Nov 1, 2006
    #15
  16. Guest

    On Wed, 1 Nov 2006, Francis Cianfrocca wrote:

    > Ara, I'm curious to know why you think the usage of nbio is archaic or
    > indicative of a design flaw. I wrote the EventMachine library in order to
    > make certain kinds of Ruby programs easier to write, and it uses nbio
    > pervasively. I'm fairly sure libevent does, also. EM works with almost any
    > kind of descriptor, except for certain things on Windows (like _pipe()
    > descriptors, which are nonselectable, and some of the native Windows objects
    > that don't support a socket-like API).
    >
    > Up until recently (late May 2006), Ruby didn't have complete support for
    > nbio on all descriptor types, which is one of the reasons that EventMachine
    > includes a compiled C++ extension. EM interoperates perfectly well with Ruby
    > threads,* which is another important reason to use nbio.


    hi francis-

    it's exactly things like eventmachine that make me say using nbio is archaic -
    i don't need to handle the complexities of nbio when powerful abstractions
    like it exist!

    for instance, consider the OP's orignial question: basically he wanted to
    timeout if stdout was not produced in a certain amount of time. i certainly
    wouldn't delve into the complexities of nbio to handle this, but instaed might
    do something along the lines of

    harrp:~ > cat a.rb
    require 'open4' and require 'timeout'

    def might_take_too_long cmd, stdin, timeout = 42
    stdout, stderr = '', ''
    Timeout::timeout(timeout) do
    open4.popen4(cmd) do |cid,i,o,e|
    i.write stdin and i.close
    o.each{|line| stdout << line} and e.each{|line| stderr << line}
    end
    end
    [ stdout, stderr ]
    end

    p might_take_too_long('ruby', 'sleep 1 and p 42', 2)
    p might_take_too_long('ruby', 'sleep 42 and p 42', 2)


    harp:~ > ruby a.rb
    ["42\n", ""]
    /home/ahoward//lib/ruby/1.8/timeout.rb:43:in `might_take_too_long': execution expired (Timeout::Error)
    <snip>


    now, timeout might use any manner of select and/or nbio - i don't care. my
    point is just that, for 90-99% of the problems seem to want to solve with
    nbio, it is way too low level a mechanism to skin the cat and prefer to stand
    on the shoulders of powerful abstractions, like eventmachine or threads or
    whatever, and avoid the painful details of EAGAIN, race conditions (it wasn't
    ready so i... oops, now it is ready), and stdio buffers being lost.

    for the record, i have code which uses nbio, io/nonblock, etc. it's simply my
    observation that a majority of the posts to this list about nbio have two
    issues:

    - nothing else can really be done while waiting for io. eg. nbio isn't
    needed for the problem at hand

    - something else could simoultaneously be done, but the problem could be
    more elegantly solved with threads, queues, readpartial, or something even
    more abstract

    the rest, of course, really do need nbio. a quick search of archives though
    is an eye opener into the difficulties of mixing nbio and stdio, check out
    these threads, specifically the posts by tanaka akira, who knows about 10
    billion times more about nbio and ruby than i do

    http://groups-beta.google.com/group...gst&q=non blocking io&rnum=1#23e22cc9e56a3200

    http://groups-beta.google.com/group...gst&q=non blocking io&rnum=6#775af5d9900aa716

    http://groups-beta.google.com/group...gst&q=non blocking io&rnum=6#775af5d9900aa716

    http://groups-beta.google.com/group...gst&q=non blocking io&rnum=6#775af5d9900aa716


    his insight is enough to raise 'too low-level' warnings in my mind! at least
    until using nbio from a ruby script doesn't require such deep understanding of
    ruby's internals

    kind regards.

    -a
    --
    my religion is very simple. my religion is kindness. -- the dalai lama
    , Nov 1, 2006
    #16
  17. Tom Pollard wrote:
    > Anyway, you would only need nonblocking IO if you wanted to read bits of
    > the stderr stream before the command exited, but that doesn't sound like
    > what you're want.


    Actually this is not correct: if there is a lot written to stderr then
    you need to read that concurrently. If you do not do that then the
    process will block on some stderr write operation that fills up the pipe
    and you get a deadlock because your code waits for process termination.

    Kind regards

    robert
    Robert Klemme, Nov 1, 2006
    #17
  18. Bill Kelly Guest

    From: <>
    >
    > In the interests of promoting creativity, I'll ask the question without
    > any prior assumptions:
    >
    > I'm using popen4 (great gem, btw) to run an external command and
    > capture stdout and stderr. If one (or both) of those is empty, I don't
    > want to hang. Just keep on moving. In fact, stderr will normally be
    > empty.
    >
    > How could I do this?
    >
    > (I'm supporting 'nix and Windows)


    Windows will be a problem. Admittedly, I haven't tried
    ruby 1.8.5 yet, which has new nonblock_* methods. However,
    my expectation is that you'll only get nonblocking behavior
    on windows from sockets, not from pipes.

    On Windows, calling select() on a pipe, always returns
    immediately with "data ready to read", regardless if there's
    any data there or not.

    This has been the bane of my existence on Windows ruby for
    5 or 6 years. I do IPC on Windows ruby using TCP over
    loopback, instead of pipes, in order to get nonblocking
    semantics. (That still doesn't help for reading from the
    console, though... (search the archives for 'kbhit' for a
    partial solution there...))

    One of these years, I'd like to chat with a Windows guru
    and ask how he/she would recommend making a select() that
    works on both sockets and pipes on Windows. Ruby could
    *really* use one.


    Regards,

    Bill
    Bill Kelly, Nov 1, 2006
    #18
  19. Il giorno 01/nov/06, alle ore 01:33, ha scritto:

    >> The select() function (Kernel#select, in Ruby) is the standard way
    >> to do IO multiplexing under Unix. You give it a list of file
    >> handles on which you're waiting to be able to read or write, and
    >> it returns when one or more of them is ready. Windows provides a
    >> select() function, too, but it only supports sockets (not pipes or
    >> ordinary file IO.) The Win32 API generally doesn't provide non-
    >> blocking IO methods, because it's assumed you'll use threads if
    >> you want to do something else while waiting for IO.

    >
    > right. but ruby's thread are green and, on windows, block the
    > entire process
    > when waiting on IO so not an option here.


    I thought Ruby internally uses non-blocking I/O in order to avoid
    that a green thread reading something blocks every other thread: am I
    wrong?
    Or is this true just under unix?
    Gabriele Marrone, Nov 1, 2006
    #19
  20. Bill Kelly Guest

    From: "Gabriele Marrone" <>
    >
    > Il giorno 01/nov/06, alle ore 01:33, ha scritto:
    >
    >>> The select() function (Kernel#select, in Ruby) is the standard way
    >>> to do IO multiplexing under Unix. You give it a list of file
    >>> handles on which you're waiting to be able to read or write, and
    >>> it returns when one or more of them is ready. Windows provides a
    >>> select() function, too, but it only supports sockets (not pipes or
    >>> ordinary file IO.) The Win32 API generally doesn't provide non-
    >>> blocking IO methods, because it's assumed you'll use threads if
    >>> you want to do something else while waiting for IO.

    >>
    >> right. but ruby's thread are green and, on windows, block the
    >> entire process when waiting on IO so not an option here.

    >
    > I thought Ruby internally uses non-blocking I/O in order to avoid
    > that a green thread reading something blocks every other thread: am I
    > wrong?
    > Or is this true just under unix?


    Ruby uses select() internally, and Windows doesn't support
    select() on pipes, just sockets.


    Regards,

    Bill
    Bill Kelly, Nov 1, 2006
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Peter Ammon

    nonblocking read()

    Peter Ammon, Nov 16, 2004, in forum: Python
    Replies:
    3
    Views:
    556
    Donn Cave
    Nov 17, 2004
  2. Replies:
    1
    Views:
    266
    Paul Boddie
    Feb 6, 2008
  3. Replies:
    3
    Views:
    320
    sturlamolden
    Jun 4, 2008
  4. Mohammed Gad
    Replies:
    0
    Views:
    87
    Mohammed Gad
    Oct 31, 2008
  5. sonet
    Replies:
    2
    Views:
    513
    Brian McCauley
    Dec 7, 2006
Loading...

Share This Page