nonblocking read()

[Peter Ammon]

I would like to read from a pipe in which data may arrive slowly. From
experimenting, it looks like os.read() will block until it returns the
maximum amount of data you asked for, in the second parameter to read(),
or until it hits EOF. I cannot find a way to return only the data that
the file object has immediately available.

If no data is available, blocking is OK.

The only workaround I can think of is to call select() in a loop,
reading and storing one byte each time, and then returning them when
select() indicates that the pipe is not ready for reading.

Are there better approaches? Thanks,

-Peter

 

[Donn Cave]

I would like to read from a pipe in which data may arrive slowly. From
experimenting, it looks like os.read() will block until it returns the
maximum amount of data you asked for, in the second parameter to read(),
or until it hits EOF. I cannot find a way to return only the data that
the file object has immediately available.

If no data is available, blocking is OK.

The only workaround I can think of is to call select() in a loop,
reading and storing one byte each time, and then returning them when
select() indicates that the pipe is not ready for reading.

I don't think that will work either!

But os.read (a.k.a. posix.read) does just what you want.
Use it with a file descriptor -

fp = os.popen(cmd, 'r')
fd = fp.fileno()
while 1:
data = os.read(fd, 4096)
if not data:
break
dispose_of(data)

Don't use the file object read or readline methods at all,
because they could leave data in their own input buffer,
where posix.read can't see it (nor select, which is why
I doubt your work-around would work.)

Donn Cave, (e-mail address removed)

 

[paul koelle]

Are there better approaches? Thanks,

What is wrong with:

fp = open('foo', 'r')

while 1:
line = fp.readline()
if not line:
time.sleep(0.3)
continue
process(line)

??
Paul

 

[Donn Cave]

What is wrong with:

fp = open('foo', 'r')

while 1:
line = fp.readline()
if not line:
time.sleep(0.3)
continue
process(line)

??

A couple of things are wrong with it.

The way I remember it, the question was about a
pipe, created by popen(), not open().

When fp.readline() returns '', that means that
the write end of the pipe has closed, and no
more data is forthcoming. There's no use for
the sleep or continue above, you may simply
break from the loop at that point.

Once those issues are taken care of, of course
it's true, you can read lines from a pipe one by
one with readline(). However, the question didn't
mention lines, and instead asked about the read()
method, so you have to wonder if perhaps the data
in question is actually not line structured. If
its structure is something other than lines (which
readline() supports), or fixed length blocks (which
read(n) supports), then I don't think you want to
use file object methods to read it.

Donn Cave, (e-mail address removed)