sysread and buffered I/O

H

Hal Fulton

I've been playing with telnet (and ssh) and I've been
pondering something.

Mixing sysread with buffered I/O is a Bad Thing (TM).

In fact, when I try to treat a telnet object as a socket
(which it is) and read from it using gets(), it crashes.
Not surprising.

So I was wondering: Would it make sense to have something like
"class SysIO < IO" where SysIO would do its own internal buffering
(and do everything in terms of sysread/-write at the lowest level)?

It would be a little like wheel-reinventing, but it would provide
an IO object on which these operations could be mixed.

Does this make sense or not?


Hal
 
T

Tanaka Akira

Hal Fulton said:
I've been playing with telnet (and ssh) and I've been
pondering something.

Mixing sysread with buffered I/O is a Bad Thing (TM).

In fact, when I try to treat a telnet object as a socket
(which it is) and read from it using gets(), it crashes.
Not surprising.

I proposed stdio friendly sysread like method: readpartial.
[ruby-dev:23247] [ruby-talk:96220]

However it is not incorporated into ruby just because the name is not
good enough. Currently I think readchunk is better, though.

If net/telnet use readchunk instead of sysread, gets and other stdio
methods can be used as usual without buffering problem, maybe.
So I was wondering: Would it make sense to have something like
"class SysIO < IO" where SysIO would do its own internal buffering
(and do everything in terms of sysread/-write at the lowest level)?

It would be a little like wheel-reinventing, but it would provide
an IO object on which these operations could be mixed.

Does this make sense or not?

The new class SysIO is not required.
I think matz will accept stdio-less implementation for IO.
 
H

Hal Fulton

Tanaka said:
I proposed stdio friendly sysread like method: readpartial.
[ruby-dev:23247] [ruby-talk:96220]

Very interesting, I did not notice that. Thank you.
However it is not incorporated into ruby just because the name is not
good enough. Currently I think readchunk is better, though.

I am not sure I like readchunk. I think I prefer an underscore
at least. Other ideas might be:
read_partial # looks better with underscore?
read_part
read_any
read_all (?)
read_waiting (?)
read_avail # meaning "available"
read_bytes
readbytes # looks ok without underscore?
The new class SysIO is not required.
I think matz will accept stdio-less implementation for IO.

OK, I see. The code is already written? Are we just seeking
a name?


Hal
 
T

Tanaka Akira

Hal Fulton said:
I am not sure I like readchunk. I think I prefer an underscore
at least. Other ideas might be:
read_partial # looks better with underscore?
read_part
read_any
read_all (?)
read_waiting (?)
read_avail # meaning "available"
read_bytes
readbytes # looks ok without underscore?

An underscore is inconsistent with other IO read??? methods: readchar,
readline, readlines.

Also, readbytes is already used by lib/readbytes.rb.
OK, I see. The code is already written? Are we just seeking
a name?

Currently Ruby uses stdio because no one write it.
 
H

Hal Fulton

Tanaka said:
An underscore is inconsistent with other IO read??? methods: readchar,
readline, readlines.

I see. Well, maybe:
readchars # with an s
readstr # could be 'string' or 'stream'
readsys # reminds us of sysread

But I do not really care much, readchunk is ok if it works. :)

Currently Ruby uses stdio because no one write it.

I do not know much about low-level I/O. How hard would this
be to do?


Hal
 
T

Tanaka Akira

An underscore is inconsistent with other IO read??? methods: readchar,
readline, readlines.

I see. Well, maybe:
readchars # with an s
readstr # could be 'string' or 'stream'
readsys # reminds us of sysread[/QUOTE]

any: doesn't represent readpartial's behavior.
all: readpartial doesn't return all data from a stream.
waiting: readpartial may return non-waiting data if buffer is empty.
avail: readpartial may return data which is not available when readpartial is called.
bytes: doesn't represent the difference from IO#read.
chars: readpartial doesn't treat characters and encodings.
str: doesn't represent the difference from IO#read.
sys: readpartial is not system call.
But I do not really care much, readchunk is ok if it works. :)

Good name is necessary to incorporate a method to ruby.
 
H

Hal Fulton

Tanaka said:
Good name is necessary to incorporate a method to ruby.

Certainly, I agree fully.

How much work is it to implement this method? I may not be
knowledgeable enough to do it myself, but I would be willing
to assist someone if I can.


Hal
 
T

Tanaka Akira

Hal Fulton said:
How much work is it to implement this method? I may not be
knowledgeable enough to do it myself, but I would be willing
to assist someone if I can.

See [ruby-dev:23247] and [ruby-dev:23248].

However now I think readpartial should raise EOFError on EOF.
The implementation in [ruby-dev:23247] returns nil on EOF.
 
M

Mark Firestone

Does anyone know of a good way to basically seek through a large query
record by record (like we used to do with the SEEK command in Clipper?) ...
I am converting the Ruby BBS to use postgres. Basically, I have real
message numbers and the message numbers displayed by the system (so the
messages can be 1-x) but the real message numbers (which are unique and
assending) can be used to keep track of the last read pointer.

I could make an array of message numbers, each time a user trys to pull a
message up, but this seens wasteful.

Thanks.

Mark

"But Schindler is bueno! Senior Burns is El Diablo!"

--------------------------------------------------------------
Website - http://www.retrobbs.org
Tradewars - telnet tradewars.retrobbs.org
BBS - http://bbs.retrobbs.org:8000
IRC - irc.retrobbs.org #main
WIKI - http://www.tpoh.org/cgi-bin/tpoh-wiki
 
G

Gavin Sinclair

Good name is necessary to incorporate a method to ruby.

Here's the relevant bit from the relevant ruby-dev summary:

TANAKA Akira suggested a new method IO#readpartial, which is
a better sysread() implementation supporting stdio buffer and
avoiding troubles due to non-blocking I/O.

Tanaka, can you please give a brief description of the method (assume
I know nothing about advanced I/O), and I'll suggest a name. Of
course, other people will too.

Gavin
 
T

Tanaka Akira

Gavin Sinclair said:
Tanaka, can you please give a brief description of the method (assume
I know nothing about advanced I/O), and I'll suggest a name. Of
course, other people will too.

[ruby-dev:23247] contains a patch including RDoc comment in English.

/*
* call-seq:
* ios.readpartial(integer [, buffer]) => string, buffer, or nil
*
* Reads at most <i>integer</i> bytes from the I/O stream but
* it blocks only if <em>ios</em> has no data immediately available.
* If the optional <i>buffer</i> argument is present,
* it must reference a String, which will receive the data.
* It raises <code>EOFError</code> on end of file.
*
* STDIN.readpartial(4096) #=> "Data immediately available"
*/

(The behavior on EOF is modified.)
 
A

Ara.T.Howard

Does anyone know of a good way to basically seek through a large query
record by record (like we used to do with the SEEK command in Clipper?) ...
I am converting the Ruby BBS to use postgres. Basically, I have real
message numbers and the message numbers displayed by the system (so the
messages can be 1-x) but the real message numbers (which are unique and
assending) can be used to keep track of the last read pointer.

I could make an array of message numbers, each time a user trys to pull a
message up, but this seens wasteful.

Thanks.

Mark

are you asking about cursors?

-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it.
| --Dogen
===============================================================================
 
M

Mark Firestone

That is entirely possible (;

I'm not sure if that is the way to do it or not. My SQL is not the
greatest.

"But Schindler is bueno! Senior Burns is El Diablo!"

--------------------------------------------------------------
Website - http://www.retrobbs.org
Tradewars - telnet tradewars.retrobbs.org
BBS - http://bbs.retrobbs.org:8000
IRC - irc.retrobbs.org #main
WIKI - http://www.tpoh.org/cgi-bin/tpoh-wiki

----- Original Message -----
From: "Ara.T.Howard" <[email protected]>
Newsgroups: comp.lang.ruby
To: "ruby-talk ML" <[email protected]>
Sent: Wednesday, July 21, 2004 1:32 PM
Subject: Re: ruby postgresql question

Does anyone know of a good way to basically seek through a large query
record by record (like we used to do with the SEEK command in Clipper?) ...
I am converting the Ruby BBS to use postgres. Basically, I have real
message numbers and the message numbers displayed by the system (so the
messages can be 1-x) but the real message numbers (which are unique and
assending) can be used to keep track of the last read pointer.

I could make an array of message numbers, each time a user trys to pull a
message up, but this seens wasteful.

Thanks.

Mark

are you asking about cursors?

-a
--
============================================================================
===
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it.
| --Dogen
============================================================================
===
 
J

John Feezell

Gavin Sinclair said:
Tanaka, can you please give a brief description of the method (assume
I know nothing about advanced I/O), and I'll suggest a name. Of
course, other people will too.

[ruby-dev:23247] contains a patch including RDoc comment in English.

/*
* call-seq:
* ios.readpartial(integer [, buffer]) => string, buffer, or nil
*
* Reads at most <i>integer</i> bytes from the I/O stream but
* it blocks only if <em>ios</em> has no data immediately available.
* If the optional <i>buffer</i> argument is present,
* it must reference a String, which will receive the data.
* It raises <code>EOFError</code> on end of file.
*
* STDIN.readpartial(4096) #=> "Data immediately available"
*/

(The behavior on EOF is modified.)

Here are some other possible names that seems to fit with readchar,
readline,
and readlines and yet catch the idea of "readpartial."

readportion
readparcel
readbundle
readatmost

Personally, "readatmost" would fit closes in my mind to the above
description.
 
A

Ara.T.Howard

Gavin Sinclair said:
Tanaka, can you please give a brief description of the method (assume
I know nothing about advanced I/O), and I'll suggest a name. Of
course, other people will too.

[ruby-dev:23247] contains a patch including RDoc comment in English.

/*
* call-seq:
* ios.readpartial(integer [, buffer]) => string, buffer, or nil
*
* Reads at most <i>integer</i> bytes from the I/O stream but
* it blocks only if <em>ios</em> has no data immediately available.
* If the optional <i>buffer</i> argument is present,
* it must reference a String, which will receive the data.
* It raises <code>EOFError</code> on end of file.
*
* STDIN.readpartial(4096) #=> "Data immediately available"
*/

(The behavior on EOF is modified.)

why not

IO#receive

as in

STDIN.receive(4096) #=> "we are receiving data as it becomes available"

-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it.
| --Dogen
===============================================================================
 
M

Michael Neumann

Mark said:
That is entirely possible (;

I'm not sure if that is the way to do it or not. My SQL is not the
greatest.

Can you explain again, what exactly is your problem.
Are there performance problems with getting the whole query result set
at once?

Postgres is a good choice ;-)

Regards,

Michael
 
M

Mark Firestone

Well, I've written re-written an old BBS program. Right now it holds it's
messages in individual text files (really marshaled object files).

I can't begin to tell you how much this sucks.

So, I'm moving it all to sql. I have a table which is a message board. the
messages are records. each message has a unique number used as a new
message index. each user has a pointer to that record.

I need to be able to go up and down the sql table, reading individual
messages in each direction, and be able to jump to a message by number, etc.
Right now I am making a array of the message numbers, and looking up the
messages that way.

So if I want message 4 in the list, which is really message 3219, because
the ones before it have been deleted, then I ask the array...

index_tbl[message] = whatever the number I should pull out of the table is.

I got the index_tbl by doing a select on the board table, ordered by the
message number.

Anyway, before each operation, I have to rebuild the table, because the
whole thing is multi-user... and someone else might have done something
like add or delete a message. it works fine when there are a few messages,
but I'm not sure how it will perform when it hits 1000's of messages.

My friend pointed me at postresql... it seems lean and fast.

Thanks for your help!

mark



"But Schindler is bueno! Senior Burns is El Diablo!"

--------------------------------------------------------------
Website - http://www.retrobbs.org
Tradewars - telnet tradewars.retrobbs.org
BBS - http://bbs.retrobbs.org:8000
IRC - irc.retrobbs.org #main
WIKI - http://www.tpoh.org/cgi-bin/tpoh-wiki

----- Original Message -----
From: "Michael Neumann" <[email protected]>
To: "ruby-talk ML" <[email protected]>
Sent: Wednesday, July 21, 2004 2:31 PM
Subject: Re: ruby postgresql question
 
M

Michael Neumann

Mark said:
Well, I've written re-written an old BBS program. Right now it holds it's
messages in individual text files (really marshaled object files).

I can't begin to tell you how much this sucks.

So, I'm moving it all to sql. I have a table which is a message board. the
messages are records. each message has a unique number used as a new
message index. each user has a pointer to that record.

can you describe the table layout in sql?
I need to be able to go up and down the sql table, reading individual
messages in each direction, and be able to jump to a message by number, etc.

what does "reading individual messages in _each direction_" mean? Has
this something to do with the hierarchy of the messages? (In-response-To
etc.)?
Right now I am making a array of the message numbers, and looking up the
messages that way.

So if I want message 4 in the list, which is really message 3219, because
the ones before it have been deleted, then I ask the array...

I don't really understand :)
index_tbl[message] = whatever the number I should pull out of the table is.

I got the index_tbl by doing a select on the board table, ordered by the
message number.

Anyway, before each operation, I have to rebuild the table, because the
whole thing is multi-user... and someone else might have done something
like add or delete a message. it works fine when there are a few messages,
but I'm not sure how it will perform when it hits 1000's of messages.

hm, maybe it's better to use transactions and don't cache the messages
on the client side.
My friend pointed me at postresql... it seems lean and fast.

yes, but maybe SQlite is even better suited for your purposes (easier to
setup, single database file, no server, faster).

I probably don't understand what a BBS is, but isn't that something like
a "forum" where you can post messages and respond to others' messages?

Here's how I would design the messages table:

create table messages (
id integer primary key,
original_id integer, /* to be backwards compatible with old
messages */
parent integer null references messages (id),
.... /* other attributes */
body text
);


Regards,

Michael
 
G

Gavin Sinclair

[ruby-dev:23247] contains a patch including RDoc comment in English.
/*
* call-seq:
* ios.readpartial(integer [, buffer]) => string, buffer, or nil
*
* Reads at most <i>integer</i> bytes from the I/O stream but
* it blocks only if <em>ios</em> has no data immediately available.
* If the optional <i>buffer</i> argument is present,
* it must reference a String, which will receive the data.
* It raises <code>EOFError</code> on end of file.
*
* STDIN.readpartial(4096) #=> "Data immediately available"
*/
(The behavior on EOF is modified.)

What about this?

1) read(n):
current method; no problem if less than n bytes read,
and no blocking

2) read(n, :noeof):
as (1), but raises EOFError on end of file

3) read(n, :noeof, :pblock):
as (2), but *partially* blocks; i.e. blocks only if no data
is immediately available

4) read(n, :exact):
as (1), raise some error if less than n bytes read
(implies :noeof)

5) read(n, :exact, :block):
as (4), but block until n bytes are available

6) read(n, :exact, :pblock):
as (4), but *partially* block, as per (3)

All methods can accept a String parameter which acts as the receiving
buffer. The 'n' parameter must come first; the order of the rest
doesn't matter.

The naming of "pblock" could definitely be better...

Gavin
 
M

Michael Neumann

Gavin said:
[...]

What about this?

1) read(n):
current method; no problem if less than n bytes read,
and no blocking

2) read(n, :noeof):
as (1), but raises EOFError on end of file

:raise_on_eof reads better for me. but may be too long.

Regards,

Michael
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top