reading from stdin via pipe, buffering?

[Rudy Gevaert]

Hi,

I have written a perl program that reads from stdin:
while(<STDIN>)
{
chomp
do_it($_);
}
Data is fed to it via a pipe:

cat myfile | ./myprogram

When running the program, the program doesn't always read the whole
line. I'm guessing this has something to do with the stdin buffer.

I would like to know how I can make that perl gets the whole line. As
it is clearly failing from time to time.

Thanks in advance,

Rudy

 

[John W. Krahn]

I have written a perl program that reads from stdin:
while(<STDIN>)
{
chomp
do_it($_);
}
Data is fed to it via a pipe:

cat myfile | ./myprogram

When running the program, the program doesn't always read the whole
line. I'm guessing this has something to do with the stdin buffer.

I would like to know how I can make that perl gets the whole line. As
it is clearly failing from time to time.

It is not your program's STDIN that is the problem, it is the STDOUT
through the pipe that is the problem. If possible, try opening 'myfile'
in your program instead, possibly using sysopen instead of open.

perldoc -f open
perldoc perlopentut
perldoc -f sysopen


John

 

[Rudy Gevaert]

It is not your program's STDIN that is the problem, it is the STDOUT
through the pipe that is the problem. If possible, try opening 'myfile'
in your program instead, possibly using sysopen instead of open.

Unfortunately it needs to work with a pipe. What is the 'perl way' to
handle input coming from a pipe?

Thanks in advance,

Rudy

 

[xhoster]

Hi,

I have written a perl program that reads from stdin:
while(<STDIN>)
{
chomp
do_it($_);
}
Data is fed to it via a pipe:

cat myfile | ./myprogram

When running the program, the program doesn't always read the whole
line. I'm guessing this has something to do with the stdin buffer.

My guess is that you are misinterpreting something. Maybe you have
cross OS line ending problems.

I would like to know how I can make that perl gets the whole line. As
it is clearly failing from time to time.

That is far from clear to me. Can you produce an example we can
run to show this problem?

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

 

[Rudy Gevaert]

My guess is that you are misinterpreting something. Maybe you have
cross OS line ending problems.

I have checked the file and it only has \n on the end of all the lines.

That is far from clear to me. Can you produce an example we can
run to show this problem?

I have looked at my problem a bit more and changed the program so I call
open(FILE, "-") || die("Can't read from stdin: $!");
while(<FILE>){ ...}

The following happens:

1) I run my command: cat bigfile | ./myprogram
2) I do an strace on 'myprogram' and on 'cat'. I observe:
a) cat is reading the whole file!
b) cat finishes reading the whole and 'myprogram' happily carries on
c) cat reads from/ writes to the file in blocks of 4096 bytes:
read(3, "\[email protected]\[email protected]"..., 4096) = 4096
write(1, "\[email protected]\[email protected]"..., 4096) = 4096
d) 'myprogram' reads from stdin in blocks of 4096 bytes:
read(0, "e\[email protected]\nsofie."..., 4096) = 4096
e) But then, 'myprogram' reads:
read(0, "", 4096) = 0
But that is not at the end of the file!

I'll try to come up with a program. But reading a lot of lines and just
printing them doesn't trigger the error.

Thanks in advance,

Rudy