$line = <FH> returns incomplete lines

[martyjc2000]

Hi

I have a logscanner script (don't we all!).
However I have recently discovered that if the log that it is reading
has, say half a line written, then a pause and then the rest of the
line is written, $line = <FH> will read in the first half of the line
first and then read in the second half of the line separately. Here is
a simple e.g. script:



#!/usr/bin/perl
# logscanner.pl

$|=1;

open (MYLOG, "/tmp/mylog");

while ( 1 )
{
while ( ($line = <MYLOG> ) )
{
print ("GOT THIS: ",$line);
}
seek (MYLOG,0, 1);
sleep (1);
}



To recreate problem, touch /tmp/mylog then run the above script.

Then in a separate terminal run this script which will let you write to
the log:



#!/opt/DKBperl/bin/perl
# writer.pl

$|=1;

open (LOG, ">>/tmp/mylog");
select LOG;
$|=1;
$stuff="";
while ( $stuff ne "exit" )
{
$stuff = <STDIN>;
if ( $stuff !~ /n$/ )
{
chomp ($stuff);
}
print LOG $stuff;
}
close (LOG);



The writer.pl script will write whatever you type into the log, without
buffering.

To make a newline enter n at the end of your line.

here is an e.g. run:

../writer.pl
line1n
line2n
line3 beginning
line3 endn


The final /tmp/mylog looks like this:
line1n
line2n
line3 beginningline3 endn


output from logscanner.pl is:
GOT THIS: line1n
GOT THIS: line2n
GOT THIS: line3 beginningGOT THIS: line3 endn

e.g. even though line3beginning did not have a new line, $line =
<MYLOG> returned that partial line. I thought it would wait until a
whole line terminated by a new line character is present.

Any ideas how I can change the logscanner script to only return the
whole line? This has to work in real time (e.g. in a similar way to
tail -f).

big thanks

Martin

 

[martyjc2000]

Just to clarify, I want the output of logscanner.pl should look like
this:

GOT THIS: line1n
GOT THIS: line2n
GOT THIS: line3 beginningline3 endn

NOT this:

GOT THIS: line1n
GOT THIS: line2n
GOT THIS: line3 beginningGOT THIS: line3 endn

 

[martyjc2000]

Just to clarify, I want the output of logscanner.pl should look like
this:

GOT THIS: line1n
GOT THIS: line2n
GOT THIS: line3 beginningline3 endn

NOT this:

GOT THIS: line1n
GOT THIS: line2n
GOT THIS: line3 beginningGOT THIS: line3 endn

 

[John Bokma]

wrote:

Hi

I have a logscanner script (don't we all!).

Nop, I use tail -f access_log

sometimes like:

tail -f access_log | grep -i googlebot

 

[Brian McCauley]

I have a logscanner script (don't we all!).
However I have recently discovered that if the log that it is reading
has, say half a line written, then a pause and then the rest of the
line is written, $line = <FH> will read in the first half of the line
first and then read in the second half of the line separately.

Yes. <> will return the final unterminated line of a file.

Try File::Tail.

 

[martyjc2000]

Thanks

but, I really wanted to use core perl, rather than other modules.
Reason being that I use this perl script (not this perl script, this is
just an example script!) on hundreds of servers.

Was therefore hoping to find a magic answer which didn't rely on other
modules....

One workaround is to check each line that is read for a newline
character, if it doesn't exist then keep the line and append on the
next line that is read.... but thats not very elegent.

any other ideas???

 

[A. Sinan Unur]

(e-mail address removed) wrote in

Thanks

Please quote some context when you are replying.

but, I really wanted to use core perl, rather than other modules.
Reason being that I use this perl script (not this perl script, this
is just an example script!) on hundreds of servers.

It is then all the more important not to re-invent the wheel.

Was therefore hoping to find a magic answer which didn't rely on other
modules....
OK.

One workaround is to check each line that is read for a newline
character, if it doesn't exist then keep the line and append on the
next line that is read.... but thats not very elegent.

any other ideas???

use the File::Tail module.

You could also read the source code of the File::Tail module, but cut &
paste reuse of code is much worse than reuse through modules. The latter
allows you to benefit from everyone else's work in maintaining the
modules.

Sinan

 

[Sherm Pendley]

but, I really wanted to use core perl, rather than other modules.
Reason being that I use this perl script (not this perl script, this is
just an example script!) on hundreds of servers.

Instead of limiting yourself this way - and not using CPAN modules is a
*major* limitation - I think it would be more productive for you to
research ways that you could manage CPAN modules without having to
individually install them on all those hundreds of machines.

For instance, Perl has the notion of a "vendor library" directory that's
included in @INC, along with the more commonly used core and site dirs. You
could configure your Perl(s) to use a network-mounted volume for this. A
single install to that volume would then make a module available for all of
the machines that have been configured to use it.

sherm--

 

[Jeff]

Instead of limiting yourself this way - and not using CPAN modules is a
*major* limitation - I think it would be more productive for you to
research ways that you could manage CPAN modules without having to
individually install them on all those hundreds of machines.

For instance, Perl has the notion of a "vendor library" directory that's
included in @INC, along with the more commonly used core and site dirs. You
could configure your Perl(s) to use a network-mounted volume for this. A
single install to that volume would then make a module available for all of
the machines that have been configured to use it.

And if you don't want to reconfigure/recompile perl on all the machines,
you can add a BEGIN block to your scripts that adds the network location
to @INC via 'use lib'.

Laziness will help you write better programs.

~Jeff

 

[Shawn Corey]

use the File::Tail module.

You could also read the source code of the File::Tail module, but cut &
paste reuse of code is much worse than reuse through modules. The latter
allows you to benefit from everyone else's work in maintaining the
modules.

Sinan

Don't forget that many of the CPAN modules are released under GPL (or
similar licenses). If you cut & paste from them, your code is a derived
work which means you *must* publish it under the terms of GPL.

Check the terms of any open source license before you cut & paste.

--- Shawn

 

[axel]

(e-mail address removed) wrote:

Instead of limiting yourself this way - and not using CPAN modules is a
*major* limitation - I think it would be more productive for you to
research ways that you could manage CPAN modules without having to
individually install them on all those hundreds of machines.

For instance, Perl has the notion of a "vendor library" directory that's
included in @INC, along with the more commonly used core and site dirs. You
could configure your Perl(s) to use a network-mounted volume for this. A
single install to that volume would then make a module available for all of
the machines that have been configured to use it.

Hmmm... using a network mounted volume is a option if it works in
the existing network environment.

But usually when installing a perl script on a production server I
just place any extra required modules in the same directory as the
script. It saves a lot of bother, especially if a script needs to
be moved to a different server. It also obviates the need for root
access in many cases.

Axel

 

[martyjc2000]

Hmmm... using a network mounted volume is a option if it works in
the existing network environment.

Yeah, i can't use a network mounted volume. Although we have a central
nfs server, not all servers mount from it. Besides, I don't know about
you guy but I am not keen on having all servers dependent on a single
nfs share (even if it is clustered).

But usually when installing a perl script on a production server I
just place any extra required modules in the same directory as the
script. It saves a lot of bother, especially if a script needs to
be moved to a different server. It also obviates the need for root
access in many cases.

I may take a look at this option, as it saves having to install the
module as root on all servers.

thanks

 

[robic0]

$line = '';
while(1) {
while ( ($line .= <MYLOG> ) )
{
if ($line =~ /[\n]+/) {
print ("GOT THIS: ",$line);
$line = '';
}
}
seek (MYLOG,0, 1);
sleep (1);
}

Quote from somewhere: "each call reads and returns the next line, until
end-of-file is reached".
'seek (MYLOG,0, 1)' is resetting the eof condition on the FH which you
get everytime.
It just so happends there was no "\n" in the last chars flushed from
the buffer but it does go to an eof.
<MYLOG> will return on $/ or eof otherwise it would not return at the
end.

 

[Tad McClellan]

if ($line =~ /[\n]+/) {

^ ^
^ ^

Those 2 characters can serve no useful purpose, so why are they there?

Do you understand the code you write?

 

[robic0]

if ($line =~ /[\n]+/) {

^ ^
^ ^

Those 2 characters can serve no useful purpose, so why are they there?

Do you understand the code you write?

Staying on topic, please give an example of what code is not
understandable to you.

 

[A. Sinan Unur]

if ($line =~ /[\n]+/) {

^ ^
^ ^

Those 2 characters can serve no useful purpose, so why are they
there?

Do you understand the code you write?

--

[ Don't quote signatures ]

Staying on topic, please give an example of what code is not
understandable to you.

You misunderstood Tad's point.

You will need to explain what you think the code above does.

Sinan

 

[Tad McClellan]

if ($line =~ /[\n]+/) {

^ ^
^ ^

Those 2 characters can serve no useful purpose, so why are they there?

Do you understand the code you write?

Staying on topic,

Huh?


please give an example of what code is not
understandable to you.

I didn't say that *I* didn't understand the code.

I said that _you_ didn't understand the code.

Your code that I quoted is equivalent to:

if ($line =~ /\n+/) {

The 2 underlined characters served no purpose, so I'm wondering
why you included them. Maybe you (mis)think that they _do_ serve
some purpose, in which case you will learn something.

 

[Chris Barts]

Don't forget that many of the CPAN modules are released under GPL (or
similar licenses). If you cut & paste from them, your code is a derived
work which means you *must* publish it under the terms of GPL.

Well, to the extent you publish it at all you must abide by the terms of
the GPL. If the code never leaves your company, the issue never really
comes up.

Check the terms of any open source license before you cut & paste.

This is good advice.

 

[robic0]

What does GPL, character class, Tail or personal insults have to do
with anything
this gentleman posted a question about? I answered his question, its
the EOF condition
he has problems with. Is his question OFF-TOPIC on this group??? If so,
that damn near
every posted subject is OFF-TOPIC.

I personally find Tad Mclelan the most insulting individual I'd seen
here. He is totally out of context,
rarely can see the forest for the trees. I am wondering what Google is
going to do when they get
another complaint about this group and some of these individuals
professing to be ... uh, I don't
know... have the power of GOD to be so insulting and personally
demeaning on here.
Obvious lurkers who have power trips. Keep this rude behaviour up and
this group will dissolve
I assure you.

Regards,
RFC

 

[A. Sinan Unur]

I am wondering what Google is going to do when they get
another complaint about this group

You seem to be unaware that it Google has no authority over the existence
of this, or any other, UseNet newsgroup.

Keep this rude behaviour up and this group will dissolve
I assure you.

Do I sense a threat?

Sinan.