sysread

Discussion in 'Perl Misc' started by Larry, Oct 5, 2008.

  1. Larry

    Larry Guest

    I have just read up on sysread and I was struck by the following:

    "*Attempts* to read LENGTH characters of data into variable SCALAR from
    the specified FILEHANDLE, using the system call read(2)."

    It made me wonder. What does that mean by "attemps" ?

    I'm writing up a script to get binary data from <STDIN> (the script is
    run on a normal web server and the data is sent to it by using http's
    POST method)

    I have a binary header on top of the raw data so here's what I'm doing:

    # Get the header size: (4 bytes, int32)
    sysread(\*STDIN, $buf, 4);

    # Get the header:
    sysread(\*STDIN, $header, unpack("N", $buf) );

    # Get the raw data:
    while( sysread(\*STDIN, $raw, 2048) )
    {
    ...raw data...
    }

    close STDIN;

    __END__;

    Suppose sysread was to read a long header data like 65000 bytes, will
    sysread actually be able to read all the data and store it on $header?
    Do you think I should deal with that another way?

    any help will be apreciated,

    thanks
    Larry, Oct 5, 2008
    #1
    1. Advertising

  2. Larry <> wrote:
    > I have just read up on sysread and I was struck by the following:
    >
    > "*Attempts* to read LENGTH characters of data into variable SCALAR from
    > the specified FILEHANDLE, using the system call read(2)."
    >
    > It made me wonder. What does that mean by "attemps" ?



    It may fail for the reasons given later in that same paragraph of the docs.


    --
    Tad McClellan
    email: perl -le "print scalar reverse qq/moc.noitatibaher\100cmdat/"
    Tad J McClellan, Oct 5, 2008
    #2
    1. Advertising

  3. Larry

    Guest

    Larry <> wrote:
    > I have just read up on sysread and I was struck by the following:
    >
    > "*Attempts* to read LENGTH characters of data into variable SCALAR from
    > the specified FILEHANDLE, using the system call read(2)."
    >
    > It made me wonder. What does that mean by "attemps" ?


    If an error occurs, then it may not read the requested number of bytes.
    If eof is reached, then it may not read the requested number of bytes.

    If there is more than zero but less than the requested number of bytes
    ready to be read, then the sysread will read just the number of bytes
    that are currently ready, and not wait for the full requested number to
    become ready. (If O_NONBLOCK is in effect, then strike "more than zero
    but" from the above)

    There maybe other instances where the attempt fails, like if the call gets
    interrupted by a signal or something. It would be highly system dependent.

    > I'm writing up a script to get binary data from <STDIN> (the script is
    > run on a normal web server and the data is sent to it by using http's
    > POST method)
    >
    > I have a binary header on top of the raw data so here's what I'm doing:
    >
    > # Get the header size: (4 bytes, int32)
    > sysread(\*STDIN, $buf, 4);


    It might be possible, but extremely unlikely, that this could return early
    after reading less than 4 bytes, even in the absence of an error condition.
    If I had to use sysread, I'd probably take that chance, myself. But why
    not just use read? That will restart as needed until it gets 4 bytes,
    unless there are errors.


    >
    > # Get the header:
    > sysread(\*STDIN, $header, unpack("N", $buf) );
    >
    > # Get the raw data:
    > while( sysread(\*STDIN, $raw, 2048) )
    > {
    > ...raw data...
    > }
    >
    > close STDIN;
    >
    > __END__;
    >
    > Suppose sysread was to read a long header data like 65000 bytes, will
    > sysread actually be able to read all the data and store it on $header?


    It will be able to, but it might not do so reliably.


    > Do you think I should deal with that another way?


    I'd probably just use "read"?

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
    , Oct 6, 2008
    #3
  4. Larry

    Larry Guest

    In article <20081005192912.762$>,
    wrote:

    > It might be possible, but extremely unlikely, that this could return early
    > after reading less than 4 bytes, even in the absence of an error condition.
    > If I had to use sysread, I'd probably take that chance, myself. But why
    > not just use read? That will restart as needed until it gets 4 bytes,
    > unless there are errors.


    this is something I forgot to tell you about:

    "sysread bypasses buffered IO, so mixing this with other kinds of reads,
    print, write, seek, tell, or eof can cause confusion because the perlio
    or stdio layers usually buffers data."

    what is buffered IO ?? how size is this buffer ??

    May I just use read() for getting important data (like the header) then
    use sysread to get the raw data??

    thanks
    Larry, Oct 6, 2008
    #4
  5. Larry

    Willem Guest

    Larry wrote:
    ) this is something I forgot to tell you about:
    )
    ) "sysread bypasses buffered IO, so mixing this with other kinds of reads,
    ) print, write, seek, tell, or eof can cause confusion because the perlio
    ) or stdio layers usually buffers data."
    )
    ) what is buffered IO ?? how size is this buffer ??
    )
    ) May I just use read() for getting important data (like the header) then
    ) use sysread to get the raw data??

    Why do you not want to use read() to get the raw data ?


    SaSW, Willem
    --
    Disclaimer: I am in no way responsible for any of the statements
    made in the above text. For all I know I might be
    drugged or something..
    No I'm not paranoid. You all think I'm paranoid, don't you !
    #EOT
    Willem, Oct 6, 2008
    #5
  6. Larry

    Larry Guest

    In article <>,
    Larry <> wrote:

    > "sysread bypasses buffered IO, so mixing this with other kinds of reads,
    > print, write, seek, tell, or eof can cause confusion because the perlio
    > or stdio layers usually buffers data."
    >
    > what is buffered IO ?? how size is this buffer ??


    Ok, I have found this:

    use IO::Handle '_IOLBF';

    $io->setvbuf($buffer_var, _IOLBF, 1024);

    (still I don't know I need to set _IOLBF or _IOFBF)

    Will it set input buffer also? I would like to keep it as small as
    possible.

    And yes, I'll be reading from STDIN by using read, not sysread:
    (hopefully the following will work, untested yet)

    use IO::Handle;

    $io = new IO::Handle;
    $io->setvbuf($buffer_var, _IOLBF, 1024);

    if ( $io->fdopen(fileno(STDIN),"r") )
    {
    $io->read($buf, 4);
    $io->read($header, unpack("N", $buf));

    while( $io->read($raw, 1024) )
    {
    ... raw data
    }

    }
    Larry, Oct 6, 2008
    #6
  7. Larry

    Larry Guest

    In article <>,
    Larry <> wrote:

    > And yes, I'll be reading from STDIN by using read, not sysread:
    > (hopefully the following will work, untested yet)
    >


    I tried the following and it worked great (although I don't know how
    much '_IONBF' affected the script)

    Do you think I should read the header in small chunks or I can read it
    all on the fly with just one read() call ??

    If I set the internal buffer to be as small as possible, will I be able
    to read the header all at once with just one reda() call ??

    Still, I don't know what size the internal buffer is...

    #!/usr/bin/perl

    use strict;
    use warnings;
    use IO::Handle '_IONBF';

    my $io = new IO::Handle;
    my $buf;
    my $len;
    my $header;

    if( $ENV{"REQUEST_METHOD"} eq 'GET' || $ENV{"REQUEST_METHOD"} eq 'HEAD' )
    {
    close(STDIN);
    print "Content-type: text/plain\n\n";
    print "Hello World!\n";
    exit;
    }

    if( $ENV{"REQUEST_METHOD"} ne 'POST' )
    {
    close(STDIN);
    print "Content-type: text/plain\n\n";
    print "What are you trying to do?\n";
    exit;
    }

    open my $fh1, ">", "rawdata.txt" or die "$!\n";
    binmode $fh1;

    if ( $io->fdopen(fileno(STDIN),"r") )
    {

    $io->read($len, 4);
    $io->read($header, unpack("N", $len));

    {open my $fh2, ">", "header.txt" or die "$!\n";
    binmode $fh2;
    print $fh2 $header;
    close $fh2;}

    while( $io->read($buf, 1024) )
    {
    print $fh1 $buf;
    }

    $io->close;
    }

    close $fh1;

    print "Content-type: text/plain\n\n";
    Larry, Oct 6, 2008
    #7
  8. Tad J McClellan, Oct 6, 2008
    #8
  9. Larry

    Guest

    Larry <> wrote:
    > In article <20081005192912.762$>,
    > wrote:
    >
    > > It might be possible, but extremely unlikely, that this could return
    > > early after reading less than 4 bytes, even in the absence of an error
    > > condition. If I had to use sysread, I'd probably take that chance,
    > > myself. But why not just use read? That will restart as needed until
    > > it gets 4 bytes, unless there are errors.

    >
    > this is something I forgot to tell you about:
    >
    > "sysread bypasses buffered IO, so mixing this with other kinds of reads,
    > print, write, seek, tell, or eof can cause confusion because the perlio
    > or stdio layers usually buffers data."
    >
    > what is buffered IO ??


    IO that is buffered by perl or by C on behalf of perl.

    > how size is this buffer ??


    That depends on your system. Why do you care? The point of using
    high-level languages is that you usually don't need to worry about such
    things.

    > May I just use read() for getting important data (like the header) then
    > use sysread to get the raw data??


    That would be mixing sysread with "other kinds of reads", which as you
    just quoted, is a bad idea.

    Why are you bound and determined to shoot yourself in the foot?

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
    , Oct 6, 2008
    #9
  10. Larry wrote:
    > I have just read up on sysread and I was struck by the following:

    ....
    > I'm writing up a script to get binary data from <STDIN> (the script is
    > run on a normal web server and the data is sent to it by using http's
    > POST method)

    ....
    > any help will be apreciated,


    If I was having trouble reinventing a wheel, I'd consider using the free
    wheel.

    According to the docs, CGI.pm handles binary uploads of this type
    without requiring you to use sysread in your code.

    Just my ยค0.02 worth.
    --
    RGB
    RedGrittyBrick, Oct 6, 2008
    #10
  11. Larry

    Larry Guest

    Larry, Oct 6, 2008
    #11
  12. Larry

    Ben Morrow Guest

    Quoth Larry <>:
    > In article <>,
    > Tad J McClellan <> wrote:
    >
    > > http://perl.plover.com/FAQs/Buffering.html

    >
    > Thanks, I read that. Too bad my system (Mac OS X) doesn't support:
    > setvbuf();


    Modern perls (as of 5.8) use PerlIO instead of stdio, so what your
    system supports is irrelevant.

    Ben

    --
    I must not fear. Fear is the mind-killer. I will face my fear and
    I will let it pass through me. When the fear is gone there will be
    nothing. Only I will remain.
    Frank Herbert, 'Dune'
    Ben Morrow, Oct 6, 2008
    #12
  13. Larry

    Guest

    On 06 Oct 2008 15:47:16 GMT, wrote:

    >Larry <> wrote:
    >> In article <20081005192912.762$>,
    >> wrote:
    >>
    >> > It might be possible, but extremely unlikely, that this could return
    >> > early after reading less than 4 bytes, even in the absence of an error
    >> > condition. If I had to use sysread, I'd probably take that chance,
    >> > myself. But why not just use read? That will restart as needed until
    >> > it gets 4 bytes, unless there are errors.

    >>
    >> this is something I forgot to tell you about:
    >>
    >> "sysread bypasses buffered IO, so mixing this with other kinds of reads,
    >> print, write, seek, tell, or eof can cause confusion because the perlio
    >> or stdio layers usually buffers data."
    >>
    >> what is buffered IO ??

    >
    >IO that is buffered by perl or by C on behalf of perl.
    >

    I would think that Perl calls level II i/o functions on its own, does it not?
    >> how size is this buffer ??

    >
    >That depends on your system.

    Does it? Do you have an example?

    > Why do you care? The point of using
    >high-level languages is that you usually don't need to worry about such
    >things.
    >

    Freedom of information. I have the freedom to ask if you know what your talking about.

    sln
    , Oct 7, 2008
    #13
  14. Larry

    Guest

    wrote:
    > On 06 Oct 2008 15:47:16 GMT, wrote:
    >
    > >Larry <> wrote:
    > >> In article <20081005192912.762$>,
    > >> wrote:
    > >>
    > >> > It might be possible, but extremely unlikely, that this could return
    > >> > early after reading less than 4 bytes, even in the absence of an
    > >> > error condition. If I had to use sysread, I'd probably take that
    > >> > chance, myself. But why not just use read? That will restart as
    > >> > needed until it gets 4 bytes, unless there are errors.
    > >>
    > >> this is something I forgot to tell you about:
    > >>
    > >> "sysread bypasses buffered IO, so mixing this with other kinds of
    > >> reads, print, write, seek, tell, or eof can cause confusion because
    > >> the perlio or stdio layers usually buffers data."
    > >>
    > >> what is buffered IO ??

    > >
    > >IO that is buffered by perl or by C on behalf of perl.
    > >

    > I would think that Perl calls level II i/o functions on its own, does it
    > not?


    I have no idea what a level II i/o function is. The man pages don't seem
    to break them down into that classification. Is this a standard usage
    or are you just making crap up?


    > >> how size is this buffer ??

    > >
    > >That depends on your system.

    > Does it? Do you have an example?


    perl -e 'print "asdfsfsfdsldfjlskdfjlsjflsadjflsdfsadfsf" foreach
    (1..3e5)'\
    | truss perl -le '<STDIN>'

    shows "read" system calls of size 5120 on SunOS 5.9. The same
    thing but with strace rather than truss gives ones of 4096 bytes on
    Linux 2.6.16.54-0.2.5-smp. An old Irix system I no longer have access
    to used 10240 bytes. At least Perl for Windows uses 512 bytes, but since
    it doesn't have a "strace" equivalent I don't remember how I deduced that.


    >
    > > Why do you care? The point of using
    > >high-level languages is that you usually don't need to worry about such
    > >things.
    > >

    > Freedom of information. I have the freedom to ask if you know what your
    > talking about.


    Coming from you, that is rather rich.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
    , Oct 7, 2008
    #14
  15. Larry

    Hans Mulder Guest

    Ben Morrow wrote:
    > Quoth Larry <>:
    >> In article <>,
    >> Tad J McClellan <> wrote:
    >>
    >>> http://perl.plover.com/FAQs/Buffering.html

    >> Thanks, I read that. Too bad my system (Mac OS X) doesn't support:
    >> setvbuf();

    >
    > Modern perls (as of 5.8) use PerlIO instead of stdio, so what your
    > system supports is irrelevant.


    The stdio on my Mac OS X system supports setvbuf(), but perlIO does not.
    For example, if I do:

    use IO::Handle '_IOFBF';
    open FH, "</etc/motd" or die $!;
    FH->setvbuf($buffer_var, _IOFBF, 8151);

    , I get:

    IO::Handle::setvbuf not implemented on this architecture at t.pl line 3.

    This is using the perl 5.8.6 that came bundled with Mac OS X 10.4.2

    -- HansM
    Hans Mulder, Oct 7, 2008
    #15
  16. Larry

    Ben Morrow Guest

    Quoth Hans Mulder <>:
    > Ben Morrow wrote:
    > > Quoth Larry <>:
    > >> In article <>,
    > >> Tad J McClellan <> wrote:
    > >>
    > >>> http://perl.plover.com/FAQs/Buffering.html
    > >> Thanks, I read that. Too bad my system (Mac OS X) doesn't support:
    > >> setvbuf();

    > >
    > > Modern perls (as of 5.8) use PerlIO instead of stdio, so what your
    > > system supports is irrelevant.

    >
    > The stdio on my Mac OS X system supports setvbuf(), but perlIO does not.
    > For example, if I do:
    >
    > use IO::Handle '_IOFBF';
    > open FH, "</etc/motd" or die $!;
    > FH->setvbuf($buffer_var, _IOFBF, 8151);
    >
    > , I get:
    >
    > IO::Handle::setvbuf not implemented on this architecture at t.pl line 3.


    Yes. From perldoc IO::Handle:

    WARNING: The IO::Handle::setvbuf() is not available by default on
    Perls 5.8.0 and later because setvbuf() is rather specific to using
    the stdio library, while Perl prefers the new perlio subsystem
    instead.

    If you need to use your own buffer, push :unix (or just use sysread) and
    do it by hand.

    Ben

    --
    "Faith has you at a disadvantage, Buffy."
    "'Cause I'm not crazy, or 'cause I don't kill people?"
    "Both, actually."
    []
    Ben Morrow, Oct 7, 2008
    #16
  17. Larry

    Larry Guest

    In article <>,
    Ben Morrow <> wrote:

    > If you need to use your own buffer, push :unix (or just use sysread) and
    > do it by hand.


    Can it actually be done? I've been wondering how to do it for a long time
    Larry, Oct 8, 2008
    #17
  18. Larry

    Larry Guest

    In article <>,
    Larry <> wrote:

    > > If you need to use your own buffer, push :unix (or just use sysread) and
    > > do it by hand.


    I have been using the following to read from STDIN:

    use IO::Handle;

    $io = new IO::Handle;
    if ($io->fdopen(fileno(STDIN),"r")) {
    print $io->getline;
    $io->close;
    }

    How can I use :unix with that?

    thanks
    Larry, Oct 8, 2008
    #18
  19. Larry

    Ben Morrow Guest

    Quoth Larry <>:
    > In article <>,
    > Larry <> wrote:
    >
    > > > If you need to use your own buffer, push :unix (or just use sysread) and
    > > > do it by hand.

    >
    > I have been using the following to read from STDIN:
    >
    > use IO::Handle;
    >
    > $io = new IO::Handle;
    > if ($io->fdopen(fileno(STDIN),"r")) {


    binmode $io, ":unix";

    > print $io->getline;
    > $io->close;
    > }
    >
    > How can I use :unix with that?


    ....but that almost certainly isn't what you want. Why do you think you
    need to do your own buffering in any case? readline (<>,
    IO::Handle->getline) depend on perl's buffering to work; while they will
    work with a :unix filehandle, it's incredibly inefficient.

    What are you actually trying to do?

    Ben

    --
    You poor take courage, you rich take care:
    The Earth was made a common treasury for everyone to share
    All things in common, all people one.
    'We come in peace'---the order came to cut them down. []
    Ben Morrow, Oct 8, 2008
    #19
  20. Larry

    Larry Guest

    In article <>,
    Ben Morrow <> wrote:

    > ...but that almost certainly isn't what you want. Why do you think you
    > need to do your own buffering in any case? readline (<>,
    > IO::Handle->getline) depend on perl's buffering to work; while they will
    > work with a :unix filehandle, it's incredibly inefficient.
    >
    > What are you actually trying to do?


    I'm sorry I copied and pasted the code above. Actually what I am trying
    to do is:

    if ( $io->fdopen(fileno(STDIN),"r") )
    {
    while($io->read(my $buf, 1024))
    {
    # ...deal with $buf...
    }
    $io->close;
    }

    thanks
    Larry, Oct 8, 2008
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hal Fulton

    sysread and buffered I/O

    Hal Fulton, Jul 21, 2004, in forum: Ruby
    Replies:
    39
    Views:
    633
    Tanaka Akira
    Jul 24, 2004
  2. William E. Rubin
    Replies:
    3
    Views:
    110
    William E. Rubin
    Dec 8, 2005
  3. Muazzam Mushtaq
    Replies:
    0
    Views:
    356
    Muazzam Mushtaq
    Mar 28, 2006
  4. pihentagy

    IO#sysread on windows

    pihentagy, Jun 14, 2006, in forum: Ruby
    Replies:
    3
    Views:
    169
    pihentagy
    Jun 14, 2006
  5. Mento Ruby
    Replies:
    0
    Views:
    127
    Mento Ruby
    Dec 11, 2006
Loading...

Share This Page