sysread and buffered I/O

[Mark Firestone]

----- Original Message -----
From: "Michael Neumann" <[email protected]>
To: "ruby-talk ML" <[email protected]>
Sent: Wednesday, July 21, 2004 3:42 PM
Subject: Re: ruby postgresql question

can you describe the table layout in sql?

Sure. Here is the sql statement for a message board table

@db.exec("CREATE TABLE #{table} (delete boolean DEFAULT false, \
locked boolean DEFAULT false, number int PRIMARY KEY, \
m_to varchar(40), \
m_from varchar(40), msg_date timestamp, subject varchar(40),\
msg_text text, exported boolean DEFAULT false)")

what does "reading individual messages in _each direction_" mean? Has
this something to do with the hierarchy of the messages? (In-response-To
etc.)?

Ok. It's an old text (telnet) based Bulletin Board System. You go to look
at messages
post since you last read them. The system find this point for you. Then
you press enter at the prompt to get the new messages, one at a time. Or
you can jump to a particular message by typing it's number, or you can go
backwards, if you like.

I don't really understand [about making the table...]

Well, messages get deleted, but the message numbers in the table don't
change. The message numbers (when you are reading them) *do* change because
they are always 1..<highest message> so the system needs to know which
actual message to pull up when you type it's number in. They won't be the
same, after one message is deleted. So I am doing a ...


for row in @db.query("SELECT number FROM #{table} ORDER BY number")
hash << row[0].to_i
end

and making an array, full of message numbers. A lookup table. So I can
look up which message to pull up.

hm, maybe it's better to use transactions and don't cache the messages
on the client side.

I'm not caching the messages, just making the lookup table. I'm concerned
about future performace problems.

yes, but maybe SQlite is even better suited for your purposes (easier to
setup, single database file, no server, faster).

I'll look into that. I've already got this thing working though... Plus, I
want to write a web front end for this one day...

I probably don't understand what a BBS is, but isn't that something like
a "forum" where you can post messages and respond to others' messages?

Yep. But text based. Have a look. Telnet to: bbs.retrobbs.org 2323 and
have a look, if you like.

Here's how I would design the messages table:

create table messages (
id integer primary key,
original_id integer, /* to be backwards compatible with old
messages */
parent integer null references messages (id),
.... /* other attributes */
body text
);

that's pretty similar to what I've done. Thanks again!

Mark

"But Schindler is bueno! Senior Burns is El Diablo!"

--------------------------------------------------------------
Website - http://www.retrobbs.org
Tradewars - telnet tradewars.retrobbs.org
BBS - http://bbs.retrobbs.org:8000
IRC - irc.retrobbs.org #main
WIKI - http://www.tpoh.org/cgi-bin/tpoh-wiki

 

[Michael Neumann]

----- Original Message -----
From: "Michael Neumann" <[email protected]>
To: "ruby-talk ML" <[email protected]>
Sent: Wednesday, July 21, 2004 3:42 PM
Subject: Re: ruby postgresql question



Sure. Here is the sql statement for a message board table

@db.exec("CREATE TABLE #{table} (delete boolean DEFAULT false, \
locked boolean DEFAULT false, number int PRIMARY KEY, \
m_to varchar(40), \
m_from varchar(40), msg_date timestamp, subject varchar(40),\
msg_text text, exported boolean DEFAULT false)")




Ok. It's an old text (telnet) based Bulletin Board System. You go to look
at messages
post since you last read them. The system find this point for you. Then
you press enter at the prompt to get the new messages, one at a time. Or
you can jump to a particular message by typing it's number, or you can go
backwards, if you like.

aha, now I understand

I don't really understand [about making the table...]

Well, messages get deleted, but the message numbers in the table don't
change. The message numbers (when you are reading them) *do* change because
they are always 1..<highest message> so the system needs to know which
actual message to pull up when you type it's number in. They won't be the
same, after one message is deleted. So I am doing a ...


for row in @db.query("SELECT number FROM #{table} ORDER BY number")
hash << row[0].to_i
end

Okay, you can use Cursors if you want:

BEGIN;
DECLARE CURSOR c SCROLL CURSOR FOR
SELECT number FROM table ORDER BY number;

FETCH c;

FETCH BACKWARD 1 IN c;

CLOSE c;
END;

Look for FETCH and DECLARE CURSOR in the Postgres docs.

Regards,

Michael

 

[Carlos]

Well, messages get deleted, but the message numbers in the table don't
change. The message numbers (when you are reading them) *do* change because
they are always 1..<highest message> so the system needs to know which
actual message to pull up when you type it's number in. They won't be the
same, after one message is deleted. So I am doing a ...


for row in @db.query("SELECT number FROM #{table} ORDER BY number")
hash << row[0].to_i
end

and making an array, full of message numbers. A lookup table. So I can
look up which message to pull up.

In postgresql you can use the OFFSET and LIMIT modifiers. For example
SELECT * FROM table ORDER BY number OFFSET #{nth_row} LIMIT 1
retrieves the nth row.

--

 

[Tanaka Akira]

Here are some other possible names that seems to fit with readchar,
readline,
and readlines and yet catch the idea of "readpartial."

readportion
readparcel
readbundle
readatmost

Personally, "readatmost" would fit closes in my mind to the above
description.

"at most" doesn't represent the difference between IO#readpartial and
IO#read. IO#read also reads at most <i>integer</i> bytes from the I/O
stream.

The difference is the blocking behaviour.

% (sleep 1; echo abc; sleep 1; echo def) | ./ruby -e 't1 = Time.now; p STDIN.readpartial(4096); t2 = Time.now; p t2-t1'
"abc\n"
1.001541
% (sleep 1; echo abc; sleep 1; echo def) | ./ruby -e 't1 = Time.now; p STDIN.read(4096); t2 = Time.now; p t2-t1'
"abc\ndef\n"
2.011527

Both IO#readpartial and IO#read blocks until some data are available.
But after some data are avalable, IO#readpartial just return the data
and doesn't block anymore. IO#read blocks until EOF or specified
length, though.

 

[Tanaka Akira]

What about this?

1) read(n):
current method; no problem if less than n bytes read,
and no blocking

2) read(n, :noeof):
as (1), but raises EOFError on end of file

3) read(n, :noeof, block):
as (2), but *partially* blocks; i.e. blocks only if no data
is immediately available

4) read(n, :exact):
as (1), raise some error if less than n bytes read
(implies :noeof)

5) read(n, :exact, :block):
as (4), but block until n bytes are available

6) read(n, :exact, block):
as (4), but *partially* block, as per (3)

All methods can accept a String parameter which acts as the receiving
buffer. The 'n' parameter must come first; the order of the rest
doesn't matter.

The naming of "pblock" could definitely be better...

I don't see a difference between 4th and 5th.
I can't understand 6th.

Do you have usecase for them?

I think such symbols based behavior specification tends to allow
specifications no one need. So it increase the possibility which an
user select an unsuitable combination by mistake. Also the
implementation should define meanings of all combination of symbols
and implement them, even if a combination is never used. So it waists
developper resources.

 

[Tanaka Akira]

why not

IO#receive

It doesn't represent the difference from IO#read.

 

[John Feezell]

"at most" doesn't represent the difference between IO#readpartial and
IO#read. IO#read also reads at most <i>integer</i> bytes from the I/O
stream.

The difference is the blocking behaviour.

% (sleep 1; echo abc; sleep 1; echo def) | ./ruby -e 't1 = Time.now; p
STDIN.readpartial(4096); t2 = Time.now; p t2-t1'
"abc\n"
1.001541
% (sleep 1; echo abc; sleep 1; echo def) | ./ruby -e 't1 = Time.now; p
STDIN.read(4096); t2 = Time.now; p t2-t1'
"abc\ndef\n"
2.011527

Both IO#readpartial and IO#read blocks until some data are available.
But after some data are avalable, IO#readpartial just return the data
and doesn't block anymore. IO#read blocks until EOF or specified
length, though.

Thanks. That makes the difference clearer.
Well then, how about,

readforedata #suggest that only the "first" or "some" data is read
readandhalt #suggest reading is only partial

 

[Ara.T.Howard]

It doesn't represent the difference from IO#read.

consider


IO.read

actively takes characters from IO until EOF is found

IO.receive

passively receives any waiting characters from IO

??

-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it.
| --Dogen
===============================================================================

 

[Nicholas Van Weerdenburg]

Hi All,

JRuby list is quiet, so I was wondering if anyone here might be able to
help with the following question (below).

Thanks,
Nick

----------------------------------------------------------------------------------------------
Hi,

I was asking about the syntax of java package includes a while back and
gave this example:

require "java"
module Java
include_package "java.util"
end
v=Java::Vector.new


In wanting to not have to type Java:: for each class, I tried a Ruby
idiom I recently learned (I'm fairly new to it):

require "java"
module Java
include_package "java.util"
end
include Java
v=Vector.new

Which doesn't work. Is this a current limitation in Jruby? Does it have
something to do with the lazy-loading I heard mentioned before? Or am I
missing something. I figure if this worked, it would be 10 to 20 lines
of code or so to implement my own "import" method that works just like
Java or Jython syntax, and that would be cool.

On a related topic, I tried to use Class.forname("...") Java idiom to
load a JDBC driver (with the appropriate Module include_package and
prefixes), and it failed. Is there a known limitation here?

Final note- any advocacy tips/issues for recommending JRuby over Jython?

Thanks,
Nick

 

[Tanaka Akira]

[/QUOTE]

readforedata #suggest that only the "first" or "some" data is read
readandhalt #suggest reading is only partial

Matz, is there a name good enough?

 

[Tanaka Akira]

IO.read

actively takes characters from IO until EOF is found

IO.receive

passively receives any waiting characters from IO

I see.

However, another problem is it is confusing with BasicSocket#recv.

 

[Ara.T.Howard]

I see.

However, another problem is it is confusing with BasicSocket#recv.

i wouldn't chose the word 'confusing' - reading the man page of recv makes it
sound VERY close to what you want. however, using exactly 'recv' WOULD be
confusing... checking on dictionary.com for receive :

Entry: receive
Function: verb
Definition: accept
Synonyms: accept, acquire, admit, apprehend, appropriate, arrogate, assume, be given, be informed, be told, catch, collect, come by, come into, cop, corral, derive, draw, earn, gain, gather, get, get from, grab, hear, hold, inherit, make, obtain, perceive, pick up, pocket, procure, pull, pull down, reap, redeem, secure, seize, snag, take, take in, take possession, win
Concept: taking
Source: Roget's New Millenniumâ„¢ Thesaurus, First Edition (v 1.0.5)
Copyright © 2004 by Lexico Publishing Group, LLC. All rights reserved.

Entry: receive

so i'd suggest

receive
accept
reap

i like the sound of

IO#reap

alot, eg. :

#
# read all chars available on pipe
#
buf = pipe.reap

i think this method is much needed and no name exists for something like it.
if we pick one that is memorable people will use it and remember it.

cheers.

-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it.
| --Dogen
===============================================================================

 

[daz]

I am not sure I like readchunk. I think I prefer an underscore
at least. Other ideas might be:
[snip]
read_avail # meaning "available"
[snip]

An underscore is inconsistent with other IO read??? methods: readchar,
readline, readlines.

[snip]
avail: readpartial may return data which is not available when readpartial is called.
[snip]

But I do not really care much, readchunk is ok if it works.

Good name is necessary to incorporate a method to ruby.

I thought Hal's read_avail (perhaps readavail) was a fair description, but ...

"readpartial may return data which is not available when readpartial is called"

Then 'readpartial' is magic ?-) (Returns unavailable data)
I know ... I misunderstand

/*
* call-seq:
* ios.readpartial(integer [, buffer]) => string, buffer, or nil
*
* Reads at most <integer> bytes from the I/O stream but
* it blocks only if <ios> has no data immediately available.
* If the optional <buffer> argument is present,
* it must reference a String, which will receive the data.
* It raises <EOFError> on end of file.
*
* STDIN.readpartial(4096) #=> "Data immediately available"
*/

read_immed ?
(readimmed is *very bad* without the underscore -- weird ! )

read_direct ? (probably not good)

read_instant, readinst, readsnap ? (at this instant in time)


I think the problem with the name 'readpartial' may be that if *all*
data is available, and 'readpartial' reads it all, then it has "failed"
because it has performed 'read', *not* 'readpartial'. :->

'readchunk' gives too much emphasis to the chunk, IMHO.
readchunk(256) looks too predictable.
If I wanted to read from <ios> in chunks, I would try that before
reading the docs.
Users of 'read_avail' or 'read_immed' would, one hopes, want to
refer to the docs before using. I think the docs of *other*
methods could say: "... if you need to reduce the risk of blocking,
'readpartial' may be more appropriate, here".

Errr, sorry to be of no help whatsoever.


daz


Two machine translations of [ruby-dev:23247] (*not* recommended reading

Excite (Japan): ===============================================

sysread thing which takes into consideration the buffer of stdio considered
since before since mind was suitable at last readpartial was mounted.
Demand of wanting to take in the data which has arrived although it does not
know how much data arrival is carried out from the pipe or the socket, if
there are such methods in the settled unit nonblock It can fill without using
sysread. Here, I do not want to use nonblock because nonblock is under a trouble.
Moreover, not wanting to use sysread is everything but IO. (the buffer of stdio
is treated) It is because it becomes impossible to use a method. nonblock is
avoidable, permitting using other methods of IO, if there is readpartial.
attaching dividing and coming out and saying [ to say ] like this

[PATCH follows]


Babelfish: ===============================================

Finally, because the air faced, it tried mounting the sysread thing readpartial
which you thought from the time before, considers the buffer of stdio.
When there is such method, you do not understand about some data it has arrived
from the pipe and the socket, but it is, the data is to take in at the large
unit to be, with the request which is said without using nonblock and sysread,
it is possible to fill up. Therefore here, as for we would not like to using
nonblock, as for nonblock the origin of trouble is. In addition, because we
would not like to using sysread handles the buffer of other (stdio of IO)
becomes unable to use method is. If there is readpartial, while allowing the fact
that the other method of IO is used, it can avoid nonblock. With being the case
that it is said, the fact that you attach such how probably will be?

[PATCH follows]

 

[Tanaka Akira]

I thought Hal's read_avail (perhaps readavail) was a fair description, but ...

"readpartial may return data which is not available when readpartial is called"

Then 'readpartial' is magic ?-) (Returns unavailable data)
I know ... I misunderstand

readpartial blocks in such case.

% (sleep 1; echo abc; sleep 1; echo def) | ./ruby -e 't1 = Time.now; p STDIN.readpartial(4096); t2 = Time.now; p t2-t1'
"abc\n"
1.001787

In this case, readpartial blocks because there are no data at first.
After 1 second, "abc" is sent over the pipe and readpartial reads and
returns it.

So, the name read_avail is not accurate.

read_immed ?
(readimmed is *very bad* without the underscore -- weird ! )

read_direct ? (probably not good)

read_instant, readinst, readsnap ? (at this instant in time)

How about them, matz?

I think the problem with the name 'readpartial' may be that if *all*
data is available, and 'readpartial' reads it all, then it has "failed"
because it has performed 'read', *not* 'readpartial'. :->
Yes.

'readchunk' gives too much emphasis to the chunk, IMHO.
readchunk(256) looks too predictable.
If I wanted to read from <ios> in chunks, I would try that before
reading the docs.

Also, several formats such as PNG defines "chunk" in their spec.

 

[Tanaka Akira]

i wouldn't chose the word 'confusing' - reading the man page of recv makes it
sound VERY close to what you want.

Yes. It is not surprised that BasicSocket#recv and IO#sysread is
similar because recv is a system call which is similar to read system call.

Also BasicSocket#recv is stdio unfriendly similar to IO#sysread:

% ruby -rsocket -e 'TCPSocket.open("www.ruby-lang.org", 80) {|s|
s.print "GET / HTTP/1.0\r\n\r\n"
p s.gets
p s.recv(100)
}'
"HTTP/1.1 302 Found\r\n"
-e:4:in `recv': recv for buffered IO (IOError)
from -e:4
from -e:1:in `open'
from -e:1

I think it is confusing that assign different meaning for abbreviation
and non-abbreviation of single word: "recv" and "receive". Their
meaning doesn't contain stdio friendly/unfriendly property.

so i'd suggest

receive
accept
reap

Since there is a system call "accept" and TCPServer#accept, IO#accept is
not an option.

i like the sound of

IO#reap

alot, eg. :

#
# read all chars available on pipe
#
buf = pipe.reap

i think this method is much needed and no name exists for something like it.
if we pick one that is memorable people will use it and remember it.

I like a method name longer than 4bytes ("read".length). Because I
encourage IO#read over IO#readpartial in usual case.

 

[Yohanes Santoso]

/*
* call-seq:
* ios.readpartial(integer [, buffer]) => string, buffer, or nil
*
* Reads at most <i>integer</i> bytes from the I/O stream but
* it blocks only if <em>ios</em> has no data immediately available.
* If the optional <i>buffer</i> argument is present,
* it must reference a String, which will receive the data.
* It raises <code>EOFError</code> on end of file.
*
* STDIN.readpartial(4096) #=> "Data immediately available"
*/

How about: STDIN.readposixly(4096) or STDIN.posix_read(4096)


Why posix?

The behaviour described above is similar to the behaviour of posix's
read function. posix's read function does a short read[1] if there is
not enough data available to satisfy the request, but perform a full
read if there is enough data. it will also block until there is some
data to be read.

I am actually leaning towards readposixly than posix_read. I read
'readposixly' as: do a read in a manner described in the posix
standard.

Meanwhile, 'posix_read' could be misinterpreted to be: read using a
posix function, which people on non-posix systems might interpret as
not being available on their systems.

Thanks,
YS.

Footnotes:
[1] reading less than requested. "short read" is a
well-established phrase. google for: ' "short read" posix '

 

[Ara.T.Howard]

I like a method name longer than 4bytes ("read".length). Because I
encourage IO#read over IO#readpartial in usual case.

then perhaps using a short name, but it should be an extension to force
require statement :

require 'io/reap'

buf = pipe.reap

this would have the additional advantage that IO#reap is not normally defined.


-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it.
| --Dogen
===============================================================================

 

[Hunter Kelly]

read_next_avail?

H

readpartial blocks in such case.

% (sleep 1; echo abc; sleep 1; echo def) | ./ruby -e 't1 = Time.now; p STDIN.readpartial(4096); t2 = Time.now; p t2-t1'
"abc\n"
1.001787

In this case, readpartial blocks because there are no data at first.
After 1 second, "abc" is sent over the pipe and readpartial reads and
returns it.

So, the name read_avail is not accurate.


How about them, matz?


Also, several formats such as PNG defines "chunk" in their spec.

 

[Tanaka Akira]

The behaviour described above is similar to the behaviour of posix's
read function. posix's read function does a short read[1] if there is
not enough data available to satisfy the request, but perform a full
read if there is enough data. it will also block until there is some
data to be read.

It is not wonder. readpartial is a stdio friendly sysread. sysread
calls posix's read function.

I'm not sure that the name represent "stdio friendly" property well.

 

[Tanaka Akira]

then perhaps using a short name, but it should be an extension to force
require statement :

require 'io/reap'

buf = pipe.reap

this would have the additional advantage that IO#reap is not normally defined.

I don't like such small extension library.