sysread

[Larry]

I have just read up on sysread and I was struck by the following:

"*Attempts* to read LENGTH characters of data into variable SCALAR from
the specified FILEHANDLE, using the system call read(2)."

It made me wonder. What does that mean by "attemps" ?

I'm writing up a script to get binary data from <STDIN> (the script is
run on a normal web server and the data is sent to it by using http's
POST method)

I have a binary header on top of the raw data so here's what I'm doing:

# Get the header size: (4 bytes, int32)
sysread(\*STDIN, $buf, 4);

# Get the header:
sysread(\*STDIN, $header, unpack("N", $buf) );

# Get the raw data:
while( sysread(\*STDIN, $raw, 2048) )
{
...raw data...
}

close STDIN;

__END__;

Suppose sysread was to read a long header data like 65000 bytes, will
sysread actually be able to read all the data and store it on $header?
Do you think I should deal with that another way?

any help will be apreciated,

thanks

 

[Tad J McClellan]

I have just read up on sysread and I was struck by the following:

"*Attempts* to read LENGTH characters of data into variable SCALAR from
the specified FILEHANDLE, using the system call read(2)."

It made me wonder. What does that mean by "attemps" ?

It may fail for the reasons given later in that same paragraph of the docs.

 

[xhoster]

I have just read up on sysread and I was struck by the following:

"*Attempts* to read LENGTH characters of data into variable SCALAR from
the specified FILEHANDLE, using the system call read(2)."

It made me wonder. What does that mean by "attemps" ?

If an error occurs, then it may not read the requested number of bytes.
If eof is reached, then it may not read the requested number of bytes.

If there is more than zero but less than the requested number of bytes
ready to be read, then the sysread will read just the number of bytes
that are currently ready, and not wait for the full requested number to
become ready. (If O_NONBLOCK is in effect, then strike "more than zero
but" from the above)

There maybe other instances where the attempt fails, like if the call gets
interrupted by a signal or something. It would be highly system dependent.

I'm writing up a script to get binary data from <STDIN> (the script is
run on a normal web server and the data is sent to it by using http's
POST method)

I have a binary header on top of the raw data so here's what I'm doing:

# Get the header size: (4 bytes, int32)
sysread(\*STDIN, $buf, 4);

It might be possible, but extremely unlikely, that this could return early
after reading less than 4 bytes, even in the absence of an error condition.
If I had to use sysread, I'd probably take that chance, myself. But why
not just use read? That will restart as needed until it gets 4 bytes,
unless there are errors.

# Get the header:
sysread(\*STDIN, $header, unpack("N", $buf) );

# Get the raw data:
while( sysread(\*STDIN, $raw, 2048) )
{
...raw data...
}

close STDIN;

__END__;

Suppose sysread was to read a long header data like 65000 bytes, will
sysread actually be able to read all the data and store it on $header?

It will be able to, but it might not do so reliably.

Do you think I should deal with that another way?

I'd probably just use "read"?

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

 

[Larry]

It might be possible, but extremely unlikely, that this could return early
after reading less than 4 bytes, even in the absence of an error condition.
If I had to use sysread, I'd probably take that chance, myself. But why
not just use read? That will restart as needed until it gets 4 bytes,
unless there are errors.

this is something I forgot to tell you about:

"sysread bypasses buffered IO, so mixing this with other kinds of reads,
print, write, seek, tell, or eof can cause confusion because the perlio
or stdio layers usually buffers data."

what is buffered IO ?? how size is this buffer ??

May I just use read() for getting important data (like the header) then
use sysread to get the raw data??

thanks

 

[Willem]

Larry wrote:
) this is something I forgot to tell you about:
)
) "sysread bypasses buffered IO, so mixing this with other kinds of reads,
) print, write, seek, tell, or eof can cause confusion because the perlio
) or stdio layers usually buffers data."
)
) what is buffered IO ?? how size is this buffer ??
)
) May I just use read() for getting important data (like the header) then
) use sysread to get the raw data??

Why do you not want to use read() to get the raw data ?


SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT

 

[Larry]

"sysread bypasses buffered IO, so mixing this with other kinds of reads,
print, write, seek, tell, or eof can cause confusion because the perlio
or stdio layers usually buffers data."

what is buffered IO ?? how size is this buffer ??

Ok, I have found this:

use IO::Handle '_IOLBF';

$io->setvbuf($buffer_var, _IOLBF, 1024);

(still I don't know I need to set _IOLBF or _IOFBF)

Will it set input buffer also? I would like to keep it as small as
possible.

And yes, I'll be reading from STDIN by using read, not sysread:
(hopefully the following will work, untested yet)

use IO::Handle;

$io = new IO::Handle;
$io->setvbuf($buffer_var, _IOLBF, 1024);

if ( $io->fdopen(fileno(STDIN),"r") )
{
$io->read($buf, 4);
$io->read($header, unpack("N", $buf));

while( $io->read($raw, 1024) )
{
... raw data
}

}

 

[Larry]

And yes, I'll be reading from STDIN by using read, not sysread:
(hopefully the following will work, untested yet)

I tried the following and it worked great (although I don't know how
much '_IONBF' affected the script)

Do you think I should read the header in small chunks or I can read it
all on the fly with just one read() call ??

If I set the internal buffer to be as small as possible, will I be able
to read the header all at once with just one reda() call ??

Still, I don't know what size the internal buffer is...

#!/usr/bin/perl

use strict;
use warnings;
use IO::Handle '_IONBF';

my $io = new IO::Handle;
my $buf;
my $len;
my $header;

if( $ENV{"REQUEST_METHOD"} eq 'GET' || $ENV{"REQUEST_METHOD"} eq 'HEAD' )
{
close(STDIN);
print "Content-type: text/plain\n\n";
print "Hello World!\n";
exit;
}

if( $ENV{"REQUEST_METHOD"} ne 'POST' )
{
close(STDIN);
print "Content-type: text/plain\n\n";
print "What are you trying to do?\n";
exit;
}

open my $fh1, ">", "rawdata.txt" or die "$!\n";
binmode $fh1;

if ( $io->fdopen(fileno(STDIN),"r") )
{

$io->read($len, 4);
$io->read($header, unpack("N", $len));

{open my $fh2, ">", "header.txt" or die "$!\n";
binmode $fh2;
print $fh2 $header;
close $fh2;}

while( $io->read($buf, 1024) )
{
print $fh1 $buf;
}

$io->close;
}

close $fh1;

print "Content-type: text/plain\n\n";

 

[Tad J McClellan]

what is buffered IO ??

Suffering from Buffering?

http://perl.plover.com/FAQs/Buffering.html

 

[xhoster]

this is something I forgot to tell you about:

"sysread bypasses buffered IO, so mixing this with other kinds of reads,
print, write, seek, tell, or eof can cause confusion because the perlio
or stdio layers usually buffers data."

what is buffered IO ??

IO that is buffered by perl or by C on behalf of perl.

how size is this buffer ??

That depends on your system. Why do you care? The point of using
high-level languages is that you usually don't need to worry about such
things.

May I just use read() for getting important data (like the header) then
use sysread to get the raw data??

That would be mixing sysread with "other kinds of reads", which as you
just quoted, is a bad idea.

Why are you bound and determined to shoot yourself in the foot?

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

 

[RedGrittyBrick]

I have just read up on sysread and I was struck by the following: ....
I'm writing up a script to get binary data from <STDIN> (the script is
run on a normal web server and the data is sent to it by using http's
POST method) ....
any help will be apreciated,

If I was having trouble reinventing a wheel, I'd consider using the free
wheel.

According to the docs, CGI.pm handles binary uploads of this type
without requiring you to use sysread in your code.

Just my ¤0.02 worth.

 

[Larry]

http://perl.plover.com/FAQs/Buffering.html

Thanks, I read that. Too bad my system (Mac OS X) doesn't support:
setvbuf();

 

[Ben Morrow]

Thanks, I read that. Too bad my system (Mac OS X) doesn't support:
setvbuf();

Modern perls (as of 5.8) use PerlIO instead of stdio, so what your
system supports is irrelevant.

Ben

 

[sln]

IO that is buffered by perl or by C on behalf of perl.

I would think that Perl calls level II i/o functions on its own, does it not?

That depends on your system.

Does it? Do you have an example?

Why do you care? The point of using
high-level languages is that you usually don't need to worry about such
things.

Freedom of information. I have the freedom to ask if you know what your talking about.

sln

 

[xhoster]

I would think that Perl calls level II i/o functions on its own, does it
not?

I have no idea what a level II i/o function is. The man pages don't seem
to break them down into that classification. Is this a standard usage
or are you just making crap up?

Does it? Do you have an example?

perl -e 'print "asdfsfsfdsldfjlskdfjlsjflsadjflsdfsadfsf" foreach
(1..3e5)'\
| truss perl -le '<STDIN>'

shows "read" system calls of size 5120 on SunOS 5.9. The same
thing but with strace rather than truss gives ones of 4096 bytes on
Linux 2.6.16.54-0.2.5-smp. An old Irix system I no longer have access
to used 10240 bytes. At least Perl for Windows uses 512 bytes, but since
it doesn't have a "strace" equivalent I don't remember how I deduced that.

Freedom of information. I have the freedom to ask if you know what your
talking about.

Coming from you, that is rather rich.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

 

[Hans Mulder]

Modern perls (as of 5.8) use PerlIO instead of stdio, so what your
system supports is irrelevant.

The stdio on my Mac OS X system supports setvbuf(), but perlIO does not.
For example, if I do:

use IO::Handle '_IOFBF';
open FH, "</etc/motd" or die $!;
FH->setvbuf($buffer_var, _IOFBF, 8151);

, I get:

IO::Handle::setvbuf not implemented on this architecture at t.pl line 3.

This is using the perl 5.8.6 that came bundled with Mac OS X 10.4.2

-- HansM

 

[Ben Morrow]

The stdio on my Mac OS X system supports setvbuf(), but perlIO does not.
For example, if I do:

use IO::Handle '_IOFBF';
open FH, "</etc/motd" or die $!;
FH->setvbuf($buffer_var, _IOFBF, 8151);

, I get:

IO::Handle::setvbuf not implemented on this architecture at t.pl line 3.

Yes. From perldoc IO::Handle:

WARNING: The IO::Handle::setvbuf() is not available by default on
Perls 5.8.0 and later because setvbuf() is rather specific to using
the stdio library, while Perl prefers the new perlio subsystem
instead.

If you need to use your own buffer, push :unix (or just use sysread) and
do it by hand.

Ben

 

[Larry]

If you need to use your own buffer, push :unix (or just use sysread) and
do it by hand.

Can it actually be done? I've been wondering how to do it for a long time

 

[Larry]

If you need to use your own buffer, push :unix (or just use sysread) and
do it by hand.

[/QUOTE]

I have been using the following to read from STDIN:

use IO::Handle;

$io = new IO::Handle;
if ($io->fdopen(fileno(STDIN),"r")) {
print $io->getline;
$io->close;
}

How can I use :unix with that?

thanks

 

[Ben Morrow]

I have been using the following to read from STDIN:

use IO::Handle;

$io = new IO::Handle;
if ($io->fdopen(fileno(STDIN),"r")) {[/QUOTE]

binmode $io, ":unix";

print $io->getline;
$io->close;
}

How can I use :unix with that?

....but that almost certainly isn't what you want. Why do you think you
need to do your own buffering in any case? readline (<>,
IO::Handle->getline) depend on perl's buffering to work; while they will
work with a :unix filehandle, it's incredibly inefficient.

What are you actually trying to do?

Ben

 

[Larry]

...but that almost certainly isn't what you want. Why do you think you
need to do your own buffering in any case? readline (<>,
IO::Handle->getline) depend on perl's buffering to work; while they will
work with a :unix filehandle, it's incredibly inefficient.

What are you actually trying to do?

I'm sorry I copied and pasted the code above. Actually what I am trying
to do is:

if ( $io->fdopen(fileno(STDIN),"r") )
{
while($io->read(my $buf, 1024))
{
# ...deal with $buf...
}
$io->close;
}

thanks