How to open a file from the end and read the last 100 lines

R

Rui Vilao

Greetings,

I am writing some Perl script to check some patterns errors in an
alert log (Oracle alert log). The file can't get quite large.
Therefore I would like write a Perl script that runs every day via at
and opens the file from the end to read the last 100 lines. How can I
do this?
Any help/suggestion is highly appreciated.

Thanks in advance for your help,

Kind Regards,

Rui Vilao
 
U

Uri Guttman

RV> I am writing some Perl script to check some patterns errors in an
RV> alert log (Oracle alert log). The file can't get quite large.
RV> Therefore I would like write a Perl script that runs every day via at
RV> and opens the file from the end to read the last 100 lines. How can I
RV> do this?

use File::ReadBackawards ;

uri
 
U

Uri Guttman

TA> *bzzzt* Should be 'File::ReadBackwards'.

bah!

it is a module that tells you who won awards in reverse order! :)

uri
 
D

David

I am writing some Perl script to check some patterns errors in an
alert log (Oracle alert log). The file can't get quite large.
Therefore I would like write a Perl script that runs every day via at
and opens the file from the end to read the last 100 lines. How can I
do this?
Any help/suggestion is highly appreciated.

I believe File::Tail is well-suited to this task.

- David
 
I

Ingo Menger

Greetings,

I am writing some Perl script to check some patterns errors in an
alert log (Oracle alert log). The file can't get quite large.
Therefore I would like write a Perl script that runs every day via at
and opens the file from the end to read the last 100 lines. How can I
do this?
Any help/suggestion is highly appreciated.

Thanks in advance for your help,

Instead of

open(FILE, "alert.log")

write

open(FILE, "tail -100 alert.log|")

Dont forget the | sign.

If you don't have tail, you'll also want to replace your OS.
 
S

Sara

Greetings,

I am writing some Perl script to check some patterns errors in an
alert log (Oracle alert log). The file can't get quite large.
Therefore I would like write a Perl script that runs every day via at
and opens the file from the end to read the last 100 lines. How can I
do this?
Any help/suggestion is highly appreciated.

Thanks in advance for your help,

Kind Regards,

Rui Vilao


Good Day Rui:

There are some modules in CPAN that would be useful - but it's more
fun to roll your own! You DO say that the file CAN'T get quite large,
which is curious because my logs DO get quite large here on thie RH8
server. But if it's small why not just:

die "I hate MONDAYS!\n" unless open F, 'log';
my @l = <F>;
close F;
@l = splice @l, @l-100;

and botta bing you have your last 100 lines!

G
 
A

Anno Siegel

Sara said:
(e-mail address removed) (Rui Vilao) wrote in message



Good Day Rui:

There are some modules in CPAN that would be useful - but it's more
fun to roll your own! You DO say that the file CAN'T get quite large,
which is curious because my logs DO get quite large here on thie RH8
server. But if it's small why not just:

die "I hate MONDAYS!\n" unless open F, 'log';
my @l = <F>;
close F;
@l = splice @l, @l-100;

and botta bing you have your last 100 lines!

....except when the file has fewer than 100 lines, in which case a fatal
run-time error results. Uri's File::ReadBackwards deals correctly with
that case.

This is a good demonstration why "rolling your own" is a bad idea, even
if the problem looks trivial.

Anno
 
S

Sara

...except when the file has fewer than 100 lines, in which case a fatal
run-time error results. Uri's File::ReadBackwards deals correctly with
that case.

This is a good demonstration why "rolling your own" is a bad idea, even
if the problem looks trivial.

Anno

Oh yes a small mod for that trivial case:

splice @l, @l-100 if @l > 100;


Oh no NEVER roll your own- never think about solving a problem for
yourself; always look for someone else's solution. You'll really get
to be a great programmer that way..


G
 
J

James Willmore

On 6 Dec 2003 07:13:55 -0800
(e-mail address removed)-berlin.de (Anno Siegel) wrote in message
Oh no NEVER roll your own- never think about solving a problem for
yourself; always look for someone else's solution. You'll really get
to be a great programmer that way..

That statement is uncalled for. I don't even view it as funny - if
that was the goal of the statement.

One of the main concepts of the Perl culture is laziness. That
doesn't mean don't work :) It means don't do more work than is
absolutely necessary. So ... if you can do it with a module, then
that should be the route to take. Why re-invent the wheel?

For example - you *can* code SQL to operate with a vendor interface,
but that's not idiomatic Perl. It *is* idiomatic Perl to use the DBI
module. It also makes for less work *and* more reliable code. *Most*
modules worth thier salt are tested and time proven. Rolling your own
is not. Unless you're learning, want to develop something for CPAN
(in which case, it *may* become tested and time proven one day), or
just want to hack something together just for the heck of it.

HTH and have a nice day :)

--
Jim

Copyright notice: all code written by the author in this post is
released under the GPL. http://www.gnu.org/licenses/gpl.txt
for more information.

a fortune quote ...
Innovation is hard to schedule. -- Dan Fylstra
 
A

Anno Siegel

Sara said:
(e-mail address removed)-berlin.de (Anno Siegel) wrote in message


Oh yes a small mod for that trivial case:

splice @l, @l-100 if @l > 100;

Don't shrug off trivial bugs. In a production program, this kind of
bug can go unnoticed for a long time. Then the program fails for no
good reason, perhaps because it's called a few times a day instead of
just once. Not good. CPAN modules don't *have* trivial bugs like that.
Oh no NEVER roll your own- never think about solving a problem for
yourself; always look for someone else's solution. You'll really get
to be a great programmer that way..

Oh, come on. You offered your code as a solution, after a dismissive
glance at possible CPAN modules. Be prepared to be measured against
them.

Anno
 
U

Uri Guttman

AS> Don't shrug off trivial bugs. In a production program, this kind of
AS> bug can go unnoticed for a long time. Then the program fails for no
AS> good reason, perhaps because it's called a few times a day instead of
AS> just once. Not good. CPAN modules don't *have* trivial bugs like that.

and my module is much faster than her code as well. slurping in a whole
file to get last 100 lines is a waste of ram and cpu. and if the file is
a large log, forget it. sara will just have to learn that rolling your
own all the time is fruitless. the ultimate result is write your own in
c because perl is just a large c based application.

uri
 
M

Mihai N.

AS> Don't shrug off trivial bugs. In a production program, this kind
of AS> bug can go unnoticed for a long time. Then the program fails
for no AS> good reason, perhaps because it's called a few times a day
instead of AS> just once. Not good. CPAN modules don't *have*
trivial bugs like that.

and my module is much faster than her code as well. slurping in a whole
file to get last 100 lines is a waste of ram and cpu. and if the file is
a large log, forget it. sara will just have to learn that rolling your
own all the time is fruitless. the ultimate result is write your own in
c because perl is just a large c based application.

uri

what about this:
while( <> ) {
push @lines, $_;
shift @lines if $#lines > 100;
}

I agree CPAN is great.
But you have to put hings in balance.

When the time + effort to search for what you need,
evaluate the 20 possible modules, select one or two, compare,
understand how they work, I would rather write my own two lines.

I would not do this for complex stuff, like parsing XML/HTML,
sending emails, DB interogations, etc.
Where is the limit for "complex" for each one, it is for each
one to decide. If I am not able to write a line of perl to
do my stuff, chances are I will not be able to use properly a
CPAN module.
CPAN is not going to think for you.
 
U

Uri Guttman

MN> what about this:
MN> while( <> ) {
MN> push @lines, $_;
MN> shift @lines if $#lines > 100;
MN> }

MN> I agree CPAN is great.
MN> But you have to put hings in balance.

that still reads in the ENTIRE file. and it does push and shift for most
lines. just try it on a large log file. i will be waiting for you.

MN> I would not do this for complex stuff, like parsing XML/HTML,
MN> sending emails, DB interogations, etc. Where is the limit for
MN> "complex" for each one, it is for each one to decide. If I am not
MN> able to write a line of perl to do my stuff, chances are I will
MN> not be able to use properly a CPAN module. CPAN is not going to
MN> think for you.

complex, slow, simpler, faster are all in the coder's control. too many
people asked how to read from the end of a file so i wrote
File::Readbackwards to solve that problem. so now it is the standard
answer for that. it is simple (simpler than rolling your own anyway you
code it), faster than all other methods (if you can show me a faster
method, i will use it) and on cpan for all to use. so there is no reason
to do it your own way. it is not such an interesting problem that you
can learn much by doing it yourself. and most common and obvious methods
are much slower and more cumbersome. that is the essence of resuable
code - solve a common problem well and put it in to some reuseable
thing. then focus your energies where they are needed. rolling your own
each time is a waste of your own time. searching cpan or asking
somewhere about a module is almost always going to be more productive.

uri
 
J

James Willmore

I agree CPAN is great.
But you have to put hings in balance.

When the time + effort to search for what you need,
evaluate the 20 possible modules, select one or two, compare,
understand how they work, I would rather write my own two lines.

I would not do this for complex stuff, like parsing XML/HTML,
sending emails, DB interogations, etc.
Where is the limit for "complex" for each one, it is for each
one to decide. If I am not able to write a line of perl to
do my stuff, chances are I will not be able to use properly a
CPAN module.
CPAN is not going to think for you.

Problem is ... you have been *told* which module will work :) So,
evaluation of a module that *works* doesn't appear to be something you
*need* to do.

It would be a different story if you *didn't* know which module would
work - but the author (no less) of a working module has told you it
works and why you should use it.

So, what's the problem with using it? Have you even *looked* at it?
If you "evaluated" the module and found it doesn't work for you, have
you let the author know this?

--
Jim

Copyright notice: all code written by the author in this post is
released under the GPL. http://www.gnu.org/licenses/gpl.txt
for more information.

a fortune quote ...
Prof: So the American government went to IBM to come up with a
data encryption standard and they came up with ... Student:
EBCDIC!"
 
J

James Willmore

what about this:
while( <> ) {
push @lines, $_;
shift @lines if $#lines > 100;
}

I agree CPAN is great.
But you have to put hings in balance.

When the time + effort to search for what you need,
evaluate the 20 possible modules, select one or two, compare,
understand how they work, I would rather write my own two lines.

I would not do this for complex stuff, like parsing XML/HTML,
sending emails, DB interogations, etc.
Where is the limit for "complex" for each one, it is for each
one to decide. If I am not able to write a line of perl to
do my stuff, chances are I will not be able to use properly a
CPAN module.
CPAN is not going to think for you.

If you *must* "roll your own", you might try this. However, read the
notes at the end of the code before using.

--untested--
#!/usr/bin/perl -w

#use strict pragma
use strict;

#get the filename of the file to process -
#die if no filename is provided
my $in_file = shift
or die "No filename provided\n";
#open the file to process -
#die if it can't be opened
open(IN, $in_file)
or die "Can't open $in_file: $!\n";
#read the file - in reverse - into an array
my @reverse = reverse <IN>;
#close the file
close IN;

#declare a counter
my $count = 0;
#declare an array to hold the lines we want
my @lines;
#while there is still an array
#containing the original file (in reverse) ...
while(@reverse){
#increment the counter
$count++;
#push onto the array for the lines we want the
#value of the current line - shift it from the
#original file (in reverse) array
push @lines, shift @reverse;
#break the loop if we reach the amount of lines
#from the end of the file (ie last 100 lines
#of the original file)
last if $count == 100;
}

#print the filnal results (ie last 100 lines of the file)
print reverse @lines;
--untested--

I tested (yes, I *always* put untested for the code, because IMHO it's
never tested enough) this with two files. One with 5 lines and my
messages file (which has well over 100 lines). It worked for both.
However, this method has some issues. First, if the file being read
exceeds the system memory, the script will most likely crap. It also
uses open versus sysopen.. And, there's no file locking. And, you
will most likely miss log messages if you use this againist an active
log file. There are propbably other issues with this method, but if
you want to use it instead of using a tested method, be my guest :)

This was "quick and dirty" - so the quality is most likely below
standards.

HTH

--
Jim

Copyright notice: all code written by the author in this post is
released under the GPL. http://www.gnu.org/licenses/gpl.txt
for more information.

a fortune quote ...
"We have reason to believe that man first walked upright to free
his hands for masturbation." -- Lily Tomlin
 
M

Mihai N.

Problem is ... you have been *told* which module will work :) So,
evaluation of a module that *works* doesn't appear to be something you
*need* to do.

It would be a different story if you *didn't* know which module would
work - but the author (no less) of a working module has told you it
works and why you should use it.
"I" was not told. I am not the original poster.
So, what's the problem with using it? Have you even *looked* at it?
If you "evaluated" the module and found it doesn't work for you, have
you let the author know this?

There is no problem using it. I am pretty sure it does it's work pretty well.
And I was not talking about this special case.

If you need the script to work without installing other 20 modules
from CPAN, this will do.

I was talking about the principle of checking CPAN for trivial problems.
Don't get me wrong, this may not be a trivial problem if you have to run
this script on GB or TB of data, several times a minute/second.
But if you run it once a day/week on some MB, should be ok.

The main think is productivity and what I am advocating is "balance."
Sometimes is faster/better your own, sometimes some CPAN module.
Decide when to go for one or the other is the difficult part.
 
I

Iain Chalmers

TA> *bzzzt* Should be 'File::ReadBackwards'.

bah!

it is a module that tells you who won awards in reverse order! :)[/QUOTE]

Heh, I still reckon it _should_ have been called File::sdrawkcaBdaeR

big
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,769
Messages
2,569,580
Members
45,054
Latest member
TrimKetoBoost

Latest Threads

Top