$line = <FH> returns incomplete lines

Discussion in 'Perl Misc' started by martyjc2000@yahoo.co.uk, Mar 24, 2005.

  1. Guest

    Hi

    I have a logscanner script (don't we all!).
    However I have recently discovered that if the log that it is reading
    has, say half a line written, then a pause and then the rest of the
    line is written, $line = <FH> will read in the first half of the line
    first and then read in the second half of the line separately. Here is
    a simple e.g. script:



    #!/usr/bin/perl
    # logscanner.pl

    $|=1;

    open (MYLOG, "/tmp/mylog");

    while ( 1 )
    {
    while ( ($line = <MYLOG> ) )
    {
    print ("GOT THIS: ",$line);
    }
    seek (MYLOG,0, 1);
    sleep (1);
    }



    To recreate problem, touch /tmp/mylog then run the above script.

    Then in a separate terminal run this script which will let you write to
    the log:



    #!/opt/DKBperl/bin/perl
    # writer.pl

    $|=1;

    open (LOG, ">>/tmp/mylog");
    select LOG;
    $|=1;
    $stuff="";
    while ( $stuff ne "exit" )
    {
    $stuff = <STDIN>;
    if ( $stuff !~ /n$/ )
    {
    chomp ($stuff);
    }
    print LOG $stuff;
    }
    close (LOG);



    The writer.pl script will write whatever you type into the log, without
    buffering.

    To make a newline enter n at the end of your line.

    here is an e.g. run:

    ../writer.pl
    line1n
    line2n
    line3 beginning
    line3 endn


    The final /tmp/mylog looks like this:
    line1n
    line2n
    line3 beginningline3 endn


    output from logscanner.pl is:
    GOT THIS: line1n
    GOT THIS: line2n
    GOT THIS: line3 beginningGOT THIS: line3 endn

    e.g. even though line3beginning did not have a new line, $line =
    <MYLOG> returned that partial line. I thought it would wait until a
    whole line terminated by a new line character is present.

    Any ideas how I can change the logscanner script to only return the
    whole line? This has to work in real time (e.g. in a similar way to
    tail -f).

    big thanks

    Martin
     
    , Mar 24, 2005
    #1
    1. Advertising

  2. Guest

    Just to clarify, I want the output of logscanner.pl should look like
    this:

    GOT THIS: line1n
    GOT THIS: line2n
    GOT THIS: line3 beginningline3 endn

    NOT this:

    GOT THIS: line1n
    GOT THIS: line2n
    GOT THIS: line3 beginningGOT THIS: line3 endn


    wrote:
    > Hi
    >
    > I have a logscanner script (don't we all!).
    > However I have recently discovered that if the log that it is reading
    > has, say half a line written, then a pause and then the rest of the
    > line is written, $line = <FH> will read in the first half of the line
    > first and then read in the second half of the line separately. Here

    is
    > a simple e.g. script:
    >
    >
    >
    > #!/usr/bin/perl
    > # logscanner.pl
    >
    > $|=1;
    >
    > open (MYLOG, "/tmp/mylog");
    >
    > while ( 1 )
    > {
    > while ( ($line = <MYLOG> ) )
    > {
    > print ("GOT THIS: ",$line);
    > }
    > seek (MYLOG,0, 1);
    > sleep (1);
    > }
    >
    >
    >
    > To recreate problem, touch /tmp/mylog then run the above script.
    >
    > Then in a separate terminal run this script which will let you write

    to
    > the log:
    >
    >
    >
    > #!/opt/DKBperl/bin/perl
    > # writer.pl
    >
    > $|=1;
    >
    > open (LOG, ">>/tmp/mylog");
    > select LOG;
    > $|=1;
    > $stuff="";
    > while ( $stuff ne "exit" )
    > {
    > $stuff = <STDIN>;
    > if ( $stuff !~ /n$/ )
    > {
    > chomp ($stuff);
    > }
    > print LOG $stuff;
    > }
    > close (LOG);
    >
    >
    >
    > The writer.pl script will write whatever you type into the log,

    without
    > buffering.
    >
    > To make a newline enter n at the end of your line.
    >
    > here is an e.g. run:
    >
    > ./writer.pl
    > line1n
    > line2n
    > line3 beginning
    > line3 endn
    >
    >
    > The final /tmp/mylog looks like this:
    > line1n
    > line2n
    > line3 beginningline3 endn
    >
    >
    > output from logscanner.pl is:
    > GOT THIS: line1n
    > GOT THIS: line2n
    > GOT THIS: line3 beginningGOT THIS: line3 endn
    >
    > e.g. even though line3beginning did not have a new line, $line =
    > <MYLOG> returned that partial line. I thought it would wait until a
    > whole line terminated by a new line character is present.
    >
    > Any ideas how I can change the logscanner script to only return the
    > whole line? This has to work in real time (e.g. in a similar way to
    > tail -f).
    >
    > big thanks
    >
    > Martin
     
    , Mar 24, 2005
    #2
    1. Advertising

  3. Guest

    Just to clarify, I want the output of logscanner.pl should look like
    this:

    GOT THIS: line1n
    GOT THIS: line2n
    GOT THIS: line3 beginningline3 endn

    NOT this:

    GOT THIS: line1n
    GOT THIS: line2n
    GOT THIS: line3 beginningGOT THIS: line3 endn


    wrote:
    > Hi
    >
    > I have a logscanner script (don't we all!).
    > However I have recently discovered that if the log that it is reading
    > has, say half a line written, then a pause and then the rest of the
    > line is written, $line = <FH> will read in the first half of the line
    > first and then read in the second half of the line separately. Here

    is
    > a simple e.g. script:
    >
    >
    >
    > #!/usr/bin/perl
    > # logscanner.pl
    >
    > $|=1;
    >
    > open (MYLOG, "/tmp/mylog");
    >
    > while ( 1 )
    > {
    > while ( ($line = <MYLOG> ) )
    > {
    > print ("GOT THIS: ",$line);
    > }
    > seek (MYLOG,0, 1);
    > sleep (1);
    > }
    >
    >
    >
    > To recreate problem, touch /tmp/mylog then run the above script.
    >
    > Then in a separate terminal run this script which will let you write

    to
    > the log:
    >
    >
    >
    > #!/opt/DKBperl/bin/perl
    > # writer.pl
    >
    > $|=1;
    >
    > open (LOG, ">>/tmp/mylog");
    > select LOG;
    > $|=1;
    > $stuff="";
    > while ( $stuff ne "exit" )
    > {
    > $stuff = <STDIN>;
    > if ( $stuff !~ /n$/ )
    > {
    > chomp ($stuff);
    > }
    > print LOG $stuff;
    > }
    > close (LOG);
    >
    >
    >
    > The writer.pl script will write whatever you type into the log,

    without
    > buffering.
    >
    > To make a newline enter n at the end of your line.
    >
    > here is an e.g. run:
    >
    > ./writer.pl
    > line1n
    > line2n
    > line3 beginning
    > line3 endn
    >
    >
    > The final /tmp/mylog looks like this:
    > line1n
    > line2n
    > line3 beginningline3 endn
    >
    >
    > output from logscanner.pl is:
    > GOT THIS: line1n
    > GOT THIS: line2n
    > GOT THIS: line3 beginningGOT THIS: line3 endn
    >
    > e.g. even though line3beginning did not have a new line, $line =
    > <MYLOG> returned that partial line. I thought it would wait until a
    > whole line terminated by a new line character is present.
    >
    > Any ideas how I can change the logscanner script to only return the
    > whole line? This has to work in real time (e.g. in a similar way to
    > tail -f).
    >
    > big thanks
    >
    > Martin
     
    , Mar 24, 2005
    #3
  4. John Bokma Guest

    John Bokma, Mar 24, 2005
    #4
  5. wrote:

    > I have a logscanner script (don't we all!).
    > However I have recently discovered that if the log that it is reading
    > has, say half a line written, then a pause and then the rest of the
    > line is written, $line = <FH> will read in the first half of the line
    > first and then read in the second half of the line separately.


    Yes. <> will return the final unterminated line of a file.

    Try File::Tail.
     
    Brian McCauley, Mar 24, 2005
    #5
  6. Guest

    Thanks

    but, I really wanted to use core perl, rather than other modules.
    Reason being that I use this perl script (not this perl script, this is
    just an example script!) on hundreds of servers.

    Was therefore hoping to find a magic answer which didn't rely on other
    modules....

    One workaround is to check each line that is read for a newline
    character, if it doesn't exist then keep the line and append on the
    next line that is read.... but thats not very elegent.

    any other ideas???
     
    , Mar 24, 2005
    #6
  7. wrote in
    news::

    > Thanks


    Please quote some context when you are replying.

    > but, I really wanted to use core perl, rather than other modules.
    > Reason being that I use this perl script (not this perl script, this
    > is just an example script!) on hundreds of servers.


    It is then all the more important not to re-invent the wheel.

    > Was therefore hoping to find a magic answer which didn't rely on other
    > modules....


    OK.

    > One workaround is to check each line that is read for a newline
    > character, if it doesn't exist then keep the line and append on the
    > next line that is read.... but thats not very elegent.
    >
    > any other ideas???


    use the File::Tail module.

    You could also read the source code of the File::Tail module, but cut &
    paste reuse of code is much worse than reuse through modules. The latter
    allows you to benefit from everyone else's work in maintaining the
    modules.

    Sinan
     
    A. Sinan Unur, Mar 24, 2005
    #7
  8. wrote:

    > but, I really wanted to use core perl, rather than other modules.
    > Reason being that I use this perl script (not this perl script, this is
    > just an example script!) on hundreds of servers.


    Instead of limiting yourself this way - and not using CPAN modules is a
    *major* limitation - I think it would be more productive for you to
    research ways that you could manage CPAN modules without having to
    individually install them on all those hundreds of machines.

    For instance, Perl has the notion of a "vendor library" directory that's
    included in @INC, along with the more commonly used core and site dirs. You
    could configure your Perl(s) to use a network-mounted volume for this. A
    single install to that volume would then make a module available for all of
    the machines that have been configured to use it.

    sherm--

    --
    Cocoa programming in Perl: http://camelbones.sourceforge.net
    Hire me! My resume: http://www.dot-app.org
     
    Sherm Pendley, Mar 24, 2005
    #8
  9. Jeff Guest

    Sherm Pendley wrote:
    > Instead of limiting yourself this way - and not using CPAN modules is a
    > *major* limitation - I think it would be more productive for you to
    > research ways that you could manage CPAN modules without having to
    > individually install them on all those hundreds of machines.
    >
    > For instance, Perl has the notion of a "vendor library" directory that's
    > included in @INC, along with the more commonly used core and site dirs. You
    > could configure your Perl(s) to use a network-mounted volume for this. A
    > single install to that volume would then make a module available for all of
    > the machines that have been configured to use it.


    And if you don't want to reconfigure/recompile perl on all the machines,
    you can add a BEGIN block to your scripts that adds the network location
    to @INC via 'use lib'.

    Laziness will help you write better programs.

    ~Jeff
     
    Jeff, Mar 24, 2005
    #9
  10. Shawn Corey Guest

    A. Sinan Unur wrote:
    > use the File::Tail module.
    >
    > You could also read the source code of the File::Tail module, but cut &
    > paste reuse of code is much worse than reuse through modules. The latter
    > allows you to benefit from everyone else's work in maintaining the
    > modules.
    >
    > Sinan


    Don't forget that many of the CPAN modules are released under GPL (or
    similar licenses). If you cut & paste from them, your code is a derived
    work which means you *must* publish it under the terms of GPL.

    Check the terms of any open source license before you cut & paste.

    --- Shawn
     
    Shawn Corey, Mar 24, 2005
    #10
  11. Guest

    Sherm Pendley <> wrote:
    > wrote:


    >> but, I really wanted to use core perl, rather than other modules.
    >> Reason being that I use this perl script (not this perl script, this is
    >> just an example script!) on hundreds of servers.


    > Instead of limiting yourself this way - and not using CPAN modules is a
    > *major* limitation - I think it would be more productive for you to
    > research ways that you could manage CPAN modules without having to
    > individually install them on all those hundreds of machines.


    > For instance, Perl has the notion of a "vendor library" directory that's
    > included in @INC, along with the more commonly used core and site dirs. You
    > could configure your Perl(s) to use a network-mounted volume for this. A
    > single install to that volume would then make a module available for all of
    > the machines that have been configured to use it.


    Hmmm... using a network mounted volume is a option if it works in
    the existing network environment.

    But usually when installing a perl script on a production server I
    just place any extra required modules in the same directory as the
    script. It saves a lot of bother, especially if a script needs to
    be moved to a different server. It also obviates the need for root
    access in many cases.

    Axel
     
    , Mar 25, 2005
    #11
  12. Guest


    > Hmmm... using a network mounted volume is a option if it works in
    > the existing network environment.


    Yeah, i can't use a network mounted volume. Although we have a central
    nfs server, not all servers mount from it. Besides, I don't know about
    you guy but I am not keen on having all servers dependent on a single
    nfs share (even if it is clustered).


    >
    > But usually when installing a perl script on a production server I
    > just place any extra required modules in the same directory as the
    > script. It saves a lot of bother, especially if a script needs to
    > be moved to a different server. It also obviates the need for root
    > access in many cases.


    I may take a look at this option, as it saves having to install the
    module as root on all servers.

    thanks
    >
    > Axel
     
    , Mar 25, 2005
    #12
  13. Guest

    $line = '';
    while(1) {
    while ( ($line .= <MYLOG> ) )
    {
    if ($line =~ /[\n]+/) {
    print ("GOT THIS: ",$line);
    $line = '';
    }
    }
    seek (MYLOG,0, 1);
    sleep (1);
    }

    Quote from somewhere: "each call reads and returns the next line, until
    end-of-file is reached".
    'seek (MYLOG,0, 1)' is resetting the eof condition on the FH which you
    get everytime.
    It just so happends there was no "\n" in the last chars flushed from
    the buffer but it does go to an eof.
    <MYLOG> will return on $/ or eof otherwise it would not return at the
    end.
     
    , Mar 25, 2005
    #13
  14. <> wrote:

    > if ($line =~ /[\n]+/) {

    ^ ^
    ^ ^

    Those 2 characters can serve no useful purpose, so why are they there?

    Do you understand the code you write?


    --
    Tad McClellan SGML consulting
    Perl programming
    Fort Worth, Texas
     
    Tad McClellan, Mar 25, 2005
    #14
  15. Guest

    Tad McClellan wrote:
    > <> wrote:
    >
    > > if ($line =~ /[\n]+/) {

    > ^ ^
    > ^ ^
    >
    > Those 2 characters can serve no useful purpose, so why are they

    there?
    >
    > Do you understand the code you write?
    >
    >
    > --
    > Tad McClellan SGML consulting
    > Perl programming
    > Fort Worth, Texas


    Staying on topic, please give an example of what code is not
    understandable to you.
     
    , Mar 25, 2005
    #15
  16. "" <> wrote in
    news::

    > Tad McClellan wrote:
    >> <> wrote:
    >>
    >> > if ($line =~ /[\n]+/) {

    >> ^ ^
    >> ^ ^
    >>
    >> Those 2 characters can serve no useful purpose, so why are they
    >> there?
    >>
    >> Do you understand the code you write?
    >>
    >> --


    [ Don't quote signatures ]

    > Staying on topic, please give an example of what code is not
    > understandable to you.


    You misunderstood Tad's point.

    You will need to explain what you think the code above does.

    Sinan
     
    A. Sinan Unur, Mar 26, 2005
    #16
  17. <> wrote:
    > Tad McClellan wrote:
    >> <> wrote:
    >>
    >> > if ($line =~ /[\n]+/) {

    >> ^ ^
    >> ^ ^
    >>
    >> Those 2 characters can serve no useful purpose, so why are they

    > there?
    >>
    >> Do you understand the code you write?


    > Staying on topic,



    Huh?


    > please give an example of what code is not
    > understandable to you.



    I didn't say that *I* didn't understand the code.

    I said that _you_ didn't understand the code.

    Your code that I quoted is equivalent to:

    if ($line =~ /\n+/) {

    The 2 underlined characters served no purpose, so I'm wondering
    why you included them. Maybe you (mis)think that they _do_ serve
    some purpose, in which case you will learn something.


    --
    Tad McClellan SGML consulting
    Perl programming
    Fort Worth, Texas
     
    Tad McClellan, Mar 26, 2005
    #17
  18. Chris Barts Guest

    Shawn Corey wrote:

    >
    > Don't forget that many of the CPAN modules are released under GPL (or
    > similar licenses). If you cut & paste from them, your code is a derived
    > work which means you *must* publish it under the terms of GPL.


    Well, to the extent you publish it at all you must abide by the terms of
    the GPL. If the code never leaves your company, the issue never really
    comes up.

    >
    > Check the terms of any open source license before you cut & paste.


    This is good advice.
     
    Chris Barts, Mar 26, 2005
    #18
  19. Guest

    What does GPL, character class, Tail or personal insults have to do
    with anything
    this gentleman posted a question about? I answered his question, its
    the EOF condition
    he has problems with. Is his question OFF-TOPIC on this group??? If so,
    that damn near
    every posted subject is OFF-TOPIC.

    I personally find Tad Mclelan the most insulting individual I'd seen
    here. He is totally out of context,
    rarely can see the forest for the trees. I am wondering what Google is
    going to do when they get
    another complaint about this group and some of these individuals
    professing to be ... uh, I don't
    know... have the power of GOD to be so insulting and personally
    demeaning on here.
    Obvious lurkers who have power trips. Keep this rude behaviour up and
    this group will dissolve
    I assure you.

    Regards,
    RFC
     
    , Mar 26, 2005
    #19
  20. "" <> wrote in
    news::

    > I am wondering what Google is going to do when they get
    > another complaint about this group


    You seem to be unaware that it Google has no authority over the existence
    of this, or any other, UseNet newsgroup.

    > Keep this rude behaviour up and this group will dissolve
    > I assure you.


    Do I sense a threat?

    Sinan.
     
    A. Sinan Unur, Mar 26, 2005
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. vizlab
    Replies:
    3
    Views:
    4,234
    Michael Bar-Sinai
    Oct 17, 2007
  2. ThePants
    Replies:
    9
    Views:
    6,211
    Joerg Jooss
    Aug 31, 2006
  3. ++imanshu
    Replies:
    7
    Views:
    485
    ++imanshu
    Aug 23, 2008
  4. Replies:
    3
    Views:
    1,325
    Patrice
    Jul 31, 2009
  5. Replies:
    27
    Views:
    430
    Peter J. Holzer
    May 18, 2007
Loading...

Share This Page