sysread and buffered I/O

Discussion in 'Ruby' started by Hal Fulton, Jul 21, 2004.

  1. Hal Fulton

    Hal Fulton Guest

    I've been playing with telnet (and ssh) and I've been
    pondering something.

    Mixing sysread with buffered I/O is a Bad Thing (TM).

    In fact, when I try to treat a telnet object as a socket
    (which it is) and read from it using gets(), it crashes.
    Not surprising.

    So I was wondering: Would it make sense to have something like
    "class SysIO < IO" where SysIO would do its own internal buffering
    (and do everything in terms of sysread/-write at the lowest level)?

    It would be a little like wheel-reinventing, but it would provide
    an IO object on which these operations could be mixed.

    Does this make sense or not?


    Hal
    Hal Fulton, Jul 21, 2004
    #1
    1. Advertising

  2. Hal Fulton

    Tanaka Akira Guest

    In article <>,
    Hal Fulton <> writes:

    > I've been playing with telnet (and ssh) and I've been
    > pondering something.
    >
    > Mixing sysread with buffered I/O is a Bad Thing (TM).
    >
    > In fact, when I try to treat a telnet object as a socket
    > (which it is) and read from it using gets(), it crashes.
    > Not surprising.


    I proposed stdio friendly sysread like method: readpartial.
    [ruby-dev:23247] [ruby-talk:96220]

    However it is not incorporated into ruby just because the name is not
    good enough. Currently I think readchunk is better, though.

    If net/telnet use readchunk instead of sysread, gets and other stdio
    methods can be used as usual without buffering problem, maybe.

    > So I was wondering: Would it make sense to have something like
    > "class SysIO < IO" where SysIO would do its own internal buffering
    > (and do everything in terms of sysread/-write at the lowest level)?
    >
    > It would be a little like wheel-reinventing, but it would provide
    > an IO object on which these operations could be mixed.
    >
    > Does this make sense or not?


    The new class SysIO is not required.
    I think matz will accept stdio-less implementation for IO.
    --
    Tanaka Akira
    Tanaka Akira, Jul 21, 2004
    #2
    1. Advertising

  3. Hal Fulton

    Hal Fulton Guest

    Tanaka Akira wrote:
    > I proposed stdio friendly sysread like method: readpartial.
    > [ruby-dev:23247] [ruby-talk:96220]


    Very interesting, I did not notice that. Thank you.

    > However it is not incorporated into ruby just because the name is not
    > good enough. Currently I think readchunk is better, though.


    I am not sure I like readchunk. I think I prefer an underscore
    at least. Other ideas might be:
    read_partial # looks better with underscore?
    read_part
    read_any
    read_all (?)
    read_waiting (?)
    read_avail # meaning "available"
    read_bytes
    readbytes # looks ok without underscore?

    > The new class SysIO is not required.
    > I think matz will accept stdio-less implementation for IO.


    OK, I see. The code is already written? Are we just seeking
    a name?


    Hal
    Hal Fulton, Jul 21, 2004
    #3
  4. Hal Fulton

    Tanaka Akira Guest

    In article <>,
    Hal Fulton <> writes:

    > I am not sure I like readchunk. I think I prefer an underscore
    > at least. Other ideas might be:
    > read_partial # looks better with underscore?
    > read_part
    > read_any
    > read_all (?)
    > read_waiting (?)
    > read_avail # meaning "available"
    > read_bytes
    > readbytes # looks ok without underscore?


    An underscore is inconsistent with other IO read??? methods: readchar,
    readline, readlines.

    Also, readbytes is already used by lib/readbytes.rb.

    > OK, I see. The code is already written? Are we just seeking
    > a name?


    Currently Ruby uses stdio because no one write it.
    --
    Tanaka Akira
    Tanaka Akira, Jul 21, 2004
    #4
  5. Hal Fulton

    Hal Fulton Guest

    Tanaka Akira wrote:

    >>I am not sure I like readchunk. I think I prefer an underscore
    >>at least. Other ideas might be:
    >> read_partial # looks better with underscore?
    >> read_part
    >> read_any
    >> read_all (?)
    >> read_waiting (?)
    >> read_avail # meaning "available"
    >> read_bytes
    >> readbytes # looks ok without underscore?

    >
    > An underscore is inconsistent with other IO read??? methods: readchar,
    > readline, readlines.


    I see. Well, maybe:
    readchars # with an s
    readstr # could be 'string' or 'stream'
    readsys # reminds us of sysread

    But I do not really care much, readchunk is ok if it works. :)


    > Currently Ruby uses stdio because no one write it.


    I do not know much about low-level I/O. How hard would this
    be to do?


    Hal
    Hal Fulton, Jul 21, 2004
    #5
  6. Hal Fulton

    Tanaka Akira Guest

    In article <>,
    Hal Fulton <> writes:

    >>>I am not sure I like readchunk. I think I prefer an underscore
    >>>at least. Other ideas might be:
    >>> read_partial # looks better with underscore?
    >>> read_part
    >>> read_any
    >>> read_all (?)
    >>> read_waiting (?)
    >>> read_avail # meaning "available"
    >>> read_bytes
    >>> readbytes # looks ok without underscore?

    >>
    >> An underscore is inconsistent with other IO read??? methods: readchar,
    >> readline, readlines.

    >
    > I see. Well, maybe:
    > readchars # with an s
    > readstr # could be 'string' or 'stream'
    > readsys # reminds us of sysread


    any: doesn't represent readpartial's behavior.
    all: readpartial doesn't return all data from a stream.
    waiting: readpartial may return non-waiting data if buffer is empty.
    avail: readpartial may return data which is not available when readpartial is called.
    bytes: doesn't represent the difference from IO#read.
    chars: readpartial doesn't treat characters and encodings.
    str: doesn't represent the difference from IO#read.
    sys: readpartial is not system call.

    > But I do not really care much, readchunk is ok if it works. :)


    Good name is necessary to incorporate a method to ruby.
    --
    Tanaka Akira
    Tanaka Akira, Jul 21, 2004
    #6
  7. Hal Fulton

    Hal Fulton Guest

    Tanaka Akira wrote:
    >
    > Good name is necessary to incorporate a method to ruby.


    Certainly, I agree fully.

    How much work is it to implement this method? I may not be
    knowledgeable enough to do it myself, but I would be willing
    to assist someone if I can.


    Hal
    Hal Fulton, Jul 21, 2004
    #7
  8. Hal Fulton

    Tanaka Akira Guest

    In article <>,
    Hal Fulton <> writes:

    > How much work is it to implement this method? I may not be
    > knowledgeable enough to do it myself, but I would be willing
    > to assist someone if I can.


    See [ruby-dev:23247] and [ruby-dev:23248].

    However now I think readpartial should raise EOFError on EOF.
    The implementation in [ruby-dev:23247] returns nil on EOF.
    --
    Tanaka Akira
    Tanaka Akira, Jul 21, 2004
    #8
  9. ruby postgresql question

    Does anyone know of a good way to basically seek through a large query
    record by record (like we used to do with the SEEK command in Clipper?) ...
    I am converting the Ruby BBS to use postgres. Basically, I have real
    message numbers and the message numbers displayed by the system (so the
    messages can be 1-x) but the real message numbers (which are unique and
    assending) can be used to keep track of the last read pointer.

    I could make an array of message numbers, each time a user trys to pull a
    message up, but this seens wasteful.

    Thanks.

    Mark

    "But Schindler is bueno! Senior Burns is El Diablo!"

    --------------------------------------------------------------
    Website - http://www.retrobbs.org
    Tradewars - telnet tradewars.retrobbs.org
    BBS - http://bbs.retrobbs.org:8000
    IRC - irc.retrobbs.org #main
    WIKI - http://www.tpoh.org/cgi-bin/tpoh-wiki
    Mark Firestone, Jul 21, 2004
    #9
  10. On Wednesday, July 21, 2004, 6:22:19 PM, Tanaka wrote:

    >> But I do not really care much, readchunk is ok if it works. :)


    > Good name is necessary to incorporate a method to ruby.


    Here's the relevant bit from the relevant ruby-dev summary:

    TANAKA Akira suggested a new method IO#readpartial, which is
    a better sysread() implementation supporting stdio buffer and
    avoiding troubles due to non-blocking I/O.

    Tanaka, can you please give a brief description of the method (assume
    I know nothing about advanced I/O), and I'll suggest a name. Of
    course, other people will too.

    Gavin
    Gavin Sinclair, Jul 21, 2004
    #10
  11. Hal Fulton

    Tanaka Akira Guest

    In article <>,
    Gavin Sinclair <> writes:

    > Tanaka, can you please give a brief description of the method (assume
    > I know nothing about advanced I/O), and I'll suggest a name. Of
    > course, other people will too.


    [ruby-dev:23247] contains a patch including RDoc comment in English.

    /*
    * call-seq:
    * ios.readpartial(integer [, buffer]) => string, buffer, or nil
    *
    * Reads at most <i>integer</i> bytes from the I/O stream but
    * it blocks only if <em>ios</em> has no data immediately available.
    * If the optional <i>buffer</i> argument is present,
    * it must reference a String, which will receive the data.
    * It raises <code>EOFError</code> on end of file.
    *
    * STDIN.readpartial(4096) #=> "Data immediately available"
    */

    (The behavior on EOF is modified.)
    --
    Tanaka Akira
    Tanaka Akira, Jul 21, 2004
    #11
  12. Hal Fulton

    Ara.T.Howard Guest

    Re: ruby postgresql question

    On Wed, 21 Jul 2004, Mark Firestone wrote:

    > Does anyone know of a good way to basically seek through a large query
    > record by record (like we used to do with the SEEK command in Clipper?) ...
    > I am converting the Ruby BBS to use postgres. Basically, I have real
    > message numbers and the message numbers displayed by the system (so the
    > messages can be 1-x) but the real message numbers (which are unique and
    > assending) can be used to keep track of the last read pointer.
    >
    > I could make an array of message numbers, each time a user trys to pull a
    > message up, but this seens wasteful.
    >
    > Thanks.
    >
    > Mark


    are you asking about cursors?

    -a
    --
    ===============================================================================
    | EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
    | PHONE :: 303.497.6469
    | A flower falls, even though we love it;
    | and a weed grows, even though we do not love it.
    | --Dogen
    ===============================================================================
    Ara.T.Howard, Jul 21, 2004
    #12
  13. Re: ruby postgresql question

    That is entirely possible (;

    I'm not sure if that is the way to do it or not. My SQL is not the
    greatest.

    "But Schindler is bueno! Senior Burns is El Diablo!"

    --------------------------------------------------------------
    Website - http://www.retrobbs.org
    Tradewars - telnet tradewars.retrobbs.org
    BBS - http://bbs.retrobbs.org:8000
    IRC - irc.retrobbs.org #main
    WIKI - http://www.tpoh.org/cgi-bin/tpoh-wiki

    ----- Original Message -----
    From: "Ara.T.Howard" <>
    Newsgroups: comp.lang.ruby
    To: "ruby-talk ML" <>
    Sent: Wednesday, July 21, 2004 1:32 PM
    Subject: Re: ruby postgresql question


    > On Wed, 21 Jul 2004, Mark Firestone wrote:
    >
    > > Does anyone know of a good way to basically seek through a large query
    > > record by record (like we used to do with the SEEK command in Clipper?)

    ...
    > > I am converting the Ruby BBS to use postgres. Basically, I have real
    > > message numbers and the message numbers displayed by the system (so the
    > > messages can be 1-x) but the real message numbers (which are unique and
    > > assending) can be used to keep track of the last read pointer.
    > >
    > > I could make an array of message numbers, each time a user trys to pull

    a
    > > message up, but this seens wasteful.
    > >
    > > Thanks.
    > >
    > > Mark

    >
    > are you asking about cursors?
    >
    > -a
    > --
    >

    ============================================================================
    ===
    > | EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
    > | PHONE :: 303.497.6469
    > | A flower falls, even though we love it;
    > | and a weed grows, even though we do not love it.
    > | --Dogen
    >

    ============================================================================
    ===
    >
    >
    Mark Firestone, Jul 21, 2004
    #13
  14. Hal Fulton

    John Feezell Guest

    On Wed, 21 Jul 2004 18:56:01 +0900, Tanaka Akira <> wrote:

    > In article <>,
    > Gavin Sinclair <> writes:
    >
    >> Tanaka, can you please give a brief description of the method (assume
    >> I know nothing about advanced I/O), and I'll suggest a name. Of
    >> course, other people will too.

    >
    > [ruby-dev:23247] contains a patch including RDoc comment in English.
    >
    > /*
    > * call-seq:
    > * ios.readpartial(integer [, buffer]) => string, buffer, or nil
    > *
    > * Reads at most <i>integer</i> bytes from the I/O stream but
    > * it blocks only if <em>ios</em> has no data immediately available.
    > * If the optional <i>buffer</i> argument is present,
    > * it must reference a String, which will receive the data.
    > * It raises <code>EOFError</code> on end of file.
    > *
    > * STDIN.readpartial(4096) #=> "Data immediately available"
    > */
    >
    > (The behavior on EOF is modified.)


    Here are some other possible names that seems to fit with readchar,
    readline,
    and readlines and yet catch the idea of "readpartial."

    readportion
    readparcel
    readbundle
    readatmost

    Personally, "readatmost" would fit closes in my mind to the above
    description.
    John Feezell, Jul 21, 2004
    #14
  15. Hal Fulton

    Ara.T.Howard Guest

    On Wed, 21 Jul 2004, Tanaka Akira wrote:

    > In article <>,
    > Gavin Sinclair <> writes:
    >
    >> Tanaka, can you please give a brief description of the method (assume
    >> I know nothing about advanced I/O), and I'll suggest a name. Of
    >> course, other people will too.

    >
    > [ruby-dev:23247] contains a patch including RDoc comment in English.
    >
    > /*
    > * call-seq:
    > * ios.readpartial(integer [, buffer]) => string, buffer, or nil
    > *
    > * Reads at most <i>integer</i> bytes from the I/O stream but
    > * it blocks only if <em>ios</em> has no data immediately available.
    > * If the optional <i>buffer</i> argument is present,
    > * it must reference a String, which will receive the data.
    > * It raises <code>EOFError</code> on end of file.
    > *
    > * STDIN.readpartial(4096) #=> "Data immediately available"
    > */
    >
    > (The behavior on EOF is modified.)


    why not

    IO#receive

    as in

    STDIN.receive(4096) #=> "we are receiving data as it becomes available"

    -a
    --
    ===============================================================================
    | EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
    | PHONE :: 303.497.6469
    | A flower falls, even though we love it;
    | and a weed grows, even though we do not love it.
    | --Dogen
    ===============================================================================
    Ara.T.Howard, Jul 21, 2004
    #15
  16. Re: ruby postgresql question

    Mark Firestone wrote:
    > That is entirely possible (;
    >
    > I'm not sure if that is the way to do it or not. My SQL is not the
    > greatest.


    Can you explain again, what exactly is your problem.
    Are there performance problems with getting the whole query result set
    at once?

    Postgres is a good choice ;-)

    Regards,

    Michael
    Michael Neumann, Jul 21, 2004
    #16
  17. Re: ruby postgresql question

    Well, I've written re-written an old BBS program. Right now it holds it's
    messages in individual text files (really marshaled object files).

    I can't begin to tell you how much this sucks.

    So, I'm moving it all to sql. I have a table which is a message board. the
    messages are records. each message has a unique number used as a new
    message index. each user has a pointer to that record.

    I need to be able to go up and down the sql table, reading individual
    messages in each direction, and be able to jump to a message by number, etc.
    Right now I am making a array of the message numbers, and looking up the
    messages that way.

    So if I want message 4 in the list, which is really message 3219, because
    the ones before it have been deleted, then I ask the array...

    index_tbl[message] = whatever the number I should pull out of the table is.

    I got the index_tbl by doing a select on the board table, ordered by the
    message number.

    Anyway, before each operation, I have to rebuild the table, because the
    whole thing is multi-user... and someone else might have done something
    like add or delete a message. it works fine when there are a few messages,
    but I'm not sure how it will perform when it hits 1000's of messages.

    My friend pointed me at postresql... it seems lean and fast.

    Thanks for your help!

    mark



    "But Schindler is bueno! Senior Burns is El Diablo!"

    --------------------------------------------------------------
    Website - http://www.retrobbs.org
    Tradewars - telnet tradewars.retrobbs.org
    BBS - http://bbs.retrobbs.org:8000
    IRC - irc.retrobbs.org #main
    WIKI - http://www.tpoh.org/cgi-bin/tpoh-wiki

    ----- Original Message -----
    From: "Michael Neumann" <>
    To: "ruby-talk ML" <>
    Sent: Wednesday, July 21, 2004 2:31 PM
    Subject: Re: ruby postgresql question


    > Mark Firestone wrote:
    > > That is entirely possible (;
    > >
    > > I'm not sure if that is the way to do it or not. My SQL is not the
    > > greatest.

    >
    > Can you explain again, what exactly is your problem.
    > Are there performance problems with getting the whole query result set
    > at once?
    >
    > Postgres is a good choice ;-)
    >
    > Regards,
    >
    > Michael
    >
    >
    Mark Firestone, Jul 21, 2004
    #17
  18. Re: ruby postgresql question

    Mark Firestone wrote:
    > Well, I've written re-written an old BBS program. Right now it holds it's
    > messages in individual text files (really marshaled object files).
    >
    > I can't begin to tell you how much this sucks.
    >
    > So, I'm moving it all to sql. I have a table which is a message board. the
    > messages are records. each message has a unique number used as a new
    > message index. each user has a pointer to that record.


    can you describe the table layout in sql?

    > I need to be able to go up and down the sql table, reading individual
    > messages in each direction, and be able to jump to a message by number, etc.


    what does "reading individual messages in _each direction_" mean? Has
    this something to do with the hierarchy of the messages? (In-response-To
    etc.)?

    > Right now I am making a array of the message numbers, and looking up the
    > messages that way.
    >
    > So if I want message 4 in the list, which is really message 3219, because
    > the ones before it have been deleted, then I ask the array...


    I don't really understand :)

    > index_tbl[message] = whatever the number I should pull out of the table is.
    >
    > I got the index_tbl by doing a select on the board table, ordered by the
    > message number.
    >
    > Anyway, before each operation, I have to rebuild the table, because the
    > whole thing is multi-user... and someone else might have done something
    > like add or delete a message. it works fine when there are a few messages,
    > but I'm not sure how it will perform when it hits 1000's of messages.


    hm, maybe it's better to use transactions and don't cache the messages
    on the client side.

    > My friend pointed me at postresql... it seems lean and fast.


    yes, but maybe SQlite is even better suited for your purposes (easier to
    setup, single database file, no server, faster).

    I probably don't understand what a BBS is, but isn't that something like
    a "forum" where you can post messages and respond to others' messages?

    Here's how I would design the messages table:

    create table messages (
    id integer primary key,
    original_id integer, /* to be backwards compatible with old
    messages */
    parent integer null references messages (id),
    .... /* other attributes */
    body text
    );


    Regards,

    Michael
    Michael Neumann, Jul 21, 2004
    #18
  19. On Wednesday, July 21, 2004, 7:56:01 PM, Tanaka wrote:

    > In article <>,
    > Gavin Sinclair <> writes:


    >> Tanaka, can you please give a brief description of the method (assume
    >> I know nothing about advanced I/O), and I'll suggest a name. Of
    >> course, other people will too.


    > [ruby-dev:23247] contains a patch including RDoc comment in English.


    > /*
    > * call-seq:
    > * ios.readpartial(integer [, buffer]) => string, buffer, or nil
    > *
    > * Reads at most <i>integer</i> bytes from the I/O stream but
    > * it blocks only if <em>ios</em> has no data immediately available.
    > * If the optional <i>buffer</i> argument is present,
    > * it must reference a String, which will receive the data.
    > * It raises <code>EOFError</code> on end of file.
    > *
    > * STDIN.readpartial(4096) #=> "Data immediately available"
    > */


    > (The behavior on EOF is modified.)


    What about this?

    1) read(n):
    current method; no problem if less than n bytes read,
    and no blocking

    2) read(n, :noeof):
    as (1), but raises EOFError on end of file

    3) read(n, :noeof, :pblock):
    as (2), but *partially* blocks; i.e. blocks only if no data
    is immediately available

    4) read(n, :exact):
    as (1), raise some error if less than n bytes read
    (implies :noeof)

    5) read(n, :exact, :block):
    as (4), but block until n bytes are available

    6) read(n, :exact, :pblock):
    as (4), but *partially* block, as per (3)

    All methods can accept a String parameter which acts as the receiving
    buffer. The 'n' parameter must come first; the order of the rest
    doesn't matter.

    The naming of "pblock" could definitely be better...

    Gavin
    Gavin Sinclair, Jul 21, 2004
    #19
  20. Gavin Sinclair wrote:
    > [...]
    >
    > What about this?
    >
    > 1) read(n):
    > current method; no problem if less than n bytes read,
    > and no blocking
    >
    > 2) read(n, :noeof):
    > as (1), but raises EOFError on end of file


    :raise_on_eof reads better for me. but may be too long.

    Regards,

    Michael
    Michael Neumann, Jul 21, 2004
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    9
    Views:
    660
    Michael Wojcik
    Aug 23, 2005
  2. Daniel Berger
    Replies:
    4
    Views:
    207
    Daniel Berger
    Sep 12, 2008
  3. john
    Replies:
    7
    Views:
    232
    Brian McCauley
    Mar 4, 2005
  4. James Marshall

    Buffered socket I/O with sysread-style blocking?

    James Marshall, Mar 28, 2006, in forum: Perl Misc
    Replies:
    21
    Views:
    335
  5. Yohan N. Leder

    Read socket using both <> and sysread()

    Yohan N. Leder, Aug 8, 2006, in forum: Perl Misc
    Replies:
    13
    Views:
    304
    Yohan N. Leder
    Aug 12, 2006
Loading...

Share This Page