reading from stdin via pipe, buffering?

Discussion in 'Perl Misc' started by Rudy Gevaert, Mar 3, 2008.

  1. Rudy Gevaert

    Rudy Gevaert Guest

    Hi,

    I have written a perl program that reads from stdin:
    while(<STDIN>)
    {
    chomp
    do_it($_);
    }
    Data is fed to it via a pipe:

    cat myfile | ./myprogram

    When running the program, the program doesn't always read the whole
    line. I'm guessing this has something to do with the stdin buffer.

    I would like to know how I can make that perl gets the whole line. As
    it is clearly failing from time to time.

    Thanks in advance,

    Rudy
     
    Rudy Gevaert, Mar 3, 2008
    #1
    1. Advertising

  2. Rudy Gevaert wrote:
    >
    > I have written a perl program that reads from stdin:
    > while(<STDIN>)
    > {
    > chomp
    > do_it($_);
    > }
    > Data is fed to it via a pipe:
    >
    > cat myfile | ./myprogram
    >
    > When running the program, the program doesn't always read the whole
    > line. I'm guessing this has something to do with the stdin buffer.
    >
    > I would like to know how I can make that perl gets the whole line. As
    > it is clearly failing from time to time.


    It is not your program's STDIN that is the problem, it is the STDOUT
    through the pipe that is the problem. If possible, try opening 'myfile'
    in your program instead, possibly using sysopen instead of open.

    perldoc -f open
    perldoc perlopentut
    perldoc -f sysopen


    John
    --
    Perl isn't a toolbox, but a small machine shop where you
    can special-order certain sorts of tools at low cost and
    in short order. -- Larry Wall
     
    John W. Krahn, Mar 3, 2008
    #2
    1. Advertising

  3. Rudy Gevaert

    Rudy Gevaert Guest

    John W. Krahn wrote:
    > Rudy Gevaert wrote:
    >>
    >> I have written a perl program that reads from stdin:
    >> while(<STDIN>)
    >> {
    >> chomp
    >> do_it($_);
    >> }
    >> Data is fed to it via a pipe:
    >>
    >> cat myfile | ./myprogram
    >>
    >> When running the program, the program doesn't always read the whole
    >> line. I'm guessing this has something to do with the stdin buffer.
    >>
    >> I would like to know how I can make that perl gets the whole line. As
    >> it is clearly failing from time to time.

    >
    > It is not your program's STDIN that is the problem, it is the STDOUT
    > through the pipe that is the problem. If possible, try opening 'myfile'
    > in your program instead, possibly using sysopen instead of open.


    Unfortunately it needs to work with a pipe. What is the 'perl way' to
    handle input coming from a pipe?

    Thanks in advance,

    Rudy
     
    Rudy Gevaert, Mar 3, 2008
    #3
  4. Rudy Gevaert

    Guest

    Rudy Gevaert <> wrote:
    > Hi,
    >
    > I have written a perl program that reads from stdin:
    > while(<STDIN>)
    > {
    > chomp
    > do_it($_);
    > }
    > Data is fed to it via a pipe:
    >
    > cat myfile | ./myprogram
    >
    > When running the program, the program doesn't always read the whole
    > line. I'm guessing this has something to do with the stdin buffer.


    My guess is that you are misinterpreting something. Maybe you have
    cross OS line ending problems.

    >
    > I would like to know how I can make that perl gets the whole line. As
    > it is clearly failing from time to time.


    That is far from clear to me. Can you produce an example we can
    run to show this problem?

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
     
    , Mar 3, 2008
    #4
  5. Rudy Gevaert

    Rudy Gevaert Guest

    wrote:
    > Rudy Gevaert <> wrote:
    >> Hi,
    >>
    >> I have written a perl program that reads from stdin:
    >> while(<STDIN>)
    >> {
    >> chomp
    >> do_it($_);
    >> }
    >> Data is fed to it via a pipe:
    >>
    >> cat myfile | ./myprogram
    >>
    >> When running the program, the program doesn't always read the whole
    >> line. I'm guessing this has something to do with the stdin buffer.

    >
    > My guess is that you are misinterpreting something. Maybe you have
    > cross OS line ending problems.


    I have checked the file and it only has \n on the end of all the lines.

    >> I would like to know how I can make that perl gets the whole line. As
    >> it is clearly failing from time to time.

    >
    > That is far from clear to me. Can you produce an example we can
    > run to show this problem?


    I have looked at my problem a bit more and changed the program so I call
    open(FILE, "-") || die("Can't read from stdin: $!");
    while(<FILE>){ ...}

    The following happens:

    1) I run my command: cat bigfile | ./myprogram
    2) I do an strace on 'myprogram' and on 'cat'. I observe:
    a) cat is reading the whole file!
    b) cat finishes reading the whole and 'myprogram' happily carries on
    c) cat reads from/ writes to the file in blocks of 4096 bytes:
    read(3, "\\"..., 4096) = 4096
    write(1, "\\"..., 4096) = 4096
    d) 'myprogram' reads from stdin in blocks of 4096 bytes:
    read(0, "e\\nsofie."..., 4096) = 4096
    e) But then, 'myprogram' reads:
    read(0, "", 4096) = 0
    But that is not at the end of the file!

    I'll try to come up with a program. But reading a lot of lines and just
    printing them doesn't trigger the error.

    Thanks in advance,

    Rudy
     
    Rudy Gevaert, Mar 4, 2008
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Charlie Zender

    Reading stdin once confuses second stdin read

    Charlie Zender, Jun 19, 2004, in forum: C Programming
    Replies:
    6
    Views:
    794
    Dan Pop
    Jun 21, 2004
  2. Rudy Gevaert
    Replies:
    2
    Views:
    3,851
    Joe Smith
    Mar 15, 2008
  3. Ara.T.Howard

    pipe buffering

    Ara.T.Howard, Oct 15, 2004, in forum: Ruby
    Replies:
    0
    Views:
    111
    Ara.T.Howard
    Oct 15, 2004
  4. Stefano Sabatini
    Replies:
    6
    Views:
    296
    Stefano Sabatini
    Jul 29, 2007
  5. DaLoverhino

    Reading from pipe or from stdin, how?

    DaLoverhino, May 1, 2009, in forum: Perl Misc
    Replies:
    1
    Views:
    167
    Jim Gibson
    May 2, 2009
Loading...

Share This Page