remove previous lines of a log file

[rajesh]

Hi,
I need a parser logic in perl. I need to read a log file line by
line and search for a keyword. Once the keyword is found in any line,
I need to remove the previous two lines and current line. My new log
file, should not have these three lines.

Thanks in Advance
Rajesh

 

[Jens Thoms Toerring]

I need a parser logic in perl. I need to read a log file line by
line and search for a keyword. Once the keyword is found in any line,
I need to remove the previous two lines and current line. My new log
file, should not have these three lines.

On most systems files are simply a sequence of bytes and you
can't cut out something from the middle. It's not like a deck
of cards, where each card is a line in the file, and where
you can pull out a few cards from the middle. You have to
copy everything from the old file to a new one (of course ex-
cept the bits you want to remove) and then replace the old
file by the new one. So in your case you have to keep three
lines in memory, writing the oldest one out if the newest
one does not contain the keyword you were looking for. Only
if it's in the newest line forget about the three lines and
copy all the rest of the contents of the old file to the new
one. Something like this should do:

#!/usr/bin/perl

use strict;
use warnings;

open my $in, '<', 'log' or die "Can't open log file.\n";
open my $out, '>', 'log.new' or die "Can't open replacement file.\n";

my @lines;
$lines[ 0 ] = <$in>;
$lines[ 1 ] = <$in>;

while ( <$in> ) {
if ( /KEYWORD/ ) {
while ( <$in> ) {
print $out $_;
}
last;
}
print $out shift @lines;
push @lines, $_;
}

close $out;
close $in;
rename 'log.new', 'log';

Regards, Jens

 

[Dave B]

Hi,
I need a parser logic in perl. I need to read a log file line by
line and search for a keyword. Once the keyword is found in any line,
I need to remove the previous two lines and current line. My new log
file, should not have these three lines.

Thanks in Advance
Rajesh

You don't say whether the keyword can appear multiple times, and if so, if
lines can be overlapping. I'll assume the keyword can appear multiple times,
but lines do not overlap, and the file has at least three lines.

A solution that reads the file line by line has already been given.
If you can afford to slurp the whole file, then

$f=~s/(?:.*\n){2}.*keyword.*\n//g;

should do what you want. (Note that the last line in the file should be
correctly terminated by a newline).

 

[Jens Thoms Toerring]

On most systems files are simply a sequence of bytes and you
can't cut out something from the middle. It's not like a deck
of cards, where each card is a line in the file, and where
you can pull out a few cards from the middle. You have to
copy everything from the old file to a new one (of course ex-
cept the bits you want to remove) and then replace the old
file by the new one. So in your case you have to keep three
lines in memory, writing the oldest one out if the newest
one does not contain the keyword you were looking for. Only
if it's in the newest line forget about the three lines and
copy all the rest of the contents of the old file to the new
one. Something like this should do:

use strict;
use warnings;

open my $in, '<', 'log' or die "Can't open log file.\n";
open my $out, '>', 'log.new' or die "Can't open replacement file.\n";

my @lines;
$lines[ 0 ] = <$in>;
$lines[ 1 ] = <$in>;

while ( <$in> ) {
if ( /KEYWORD/ ) {
while ( <$in> ) {
print $out $_;
}
last;
}
print $out shift @lines;
push @lines, $_;
}

Sorry, just noticed that this will lose the last two lines
if 'KEYWORD' is never found, so change that to

while ( <$in> ) {
if ( /KEYWORD/ ) {
@lines = ( );
last;
}
print $out shift @lines;
push @lines, $_;
}

if ( @lines ) {
print $out, $_ for @lines;
} else {
while ( <$in> ) {
print $out, $_;
}
}

or something along that line.

close $out;
close $in;
rename 'log.new', 'log';

Regards, Jens

 

[Leon Timmermans]

if ( @lines ) {
print $out, $_ for @lines;
} else {
while ( <$in> ) {
print $out, $_;
}
}

I'd skip those comma's after $out though ;-)

Leon Timmermans

 

[Leon Timmermans]

$f=~s/(?:.*\n){2}.*keyword.*\n//g;

should do what you want. (Note that the last line in the file should be
correctly terminated by a newline).

I think that last issue can be solved by the /m modifier

$f =~ s/ (?:.*\n){2} .* keyword .* $ //xmg;

Leon Timmermans

 

[Jens Thoms Toerring]

I'd skip those comma's after $out though ;-)

I guess I also would do so after a bit more of thinking;-)

Regards, Jens

 

[Dave B]

I think that last issue can be solved by the /m modifier

$f =~ s/ (?:.*\n){2} .* keyword .* $ //xmg;

That will take care of the corner case, but will leave an unnecessary \n in
the file in the regular cases.

This should do the job:

$f=~s/(?:.*\n){2}.*keyword.*(?:\n|$)//g;

 

[rajesh]

Hi,
I am unable to use this script. It gives the output file with
the lines until the keyword is found. after the first occurance of the
keyword, starting from there all lines are getting deleted.
I am adding code for your reference. I want to find out ENUMERR which
is there in the line no.11. my resultant output file should not have
line no: 9,10,11. Remaining all should be intact.

Thanks in Advance
rajesh


endtask : collect_e_daita
3 |
4 ncvlog: *W,LBLMAT (../_src/_hard/_vc/lcdport_vc/sv/
nec_lcdport_vc_etype_monitor.sv,1 68|24): Label/name mismatch:
collect_e_data - collect_e_daita.
5 (`include file: ../_src/_hard/_vc/lcdport_vc/sv/
nec_lcdport_vc_etype_monitor.sv line 168, `include file: ../_src/
_hard/_vc/lcdport_vc/sv/nec_lcdport_vc_top.sv line 47, `include
file: ../_src/_hard/_tb/_lcdbif_top.sv line 26, file: ../_src/_hard/
_tb/nec _lcdbif_tb_top.sv line 39)
6 apb_trans_cov.sample();
7 |
8 ncvlog: *W,ECSSDM (./INCA_libs/irun.lnx86.06.20.nc/svpplib/h/
sivajar/lcdbif/_veri/_rando m/_src/_hard/_uvc/apb_uvc/sv/
apb_uvc_master_monitor.sv,149|24): The 'sample' met hod is called
upon a covergroup instance whose declaration has a default sampling
condit ion associated with it.
9 `ovm_field_int (rd_wr, OVM_ALL_ON)
10 |
11 ncvlog: *W,ENUMERR (../_src/_hard/_uvc/apb_uvc/examples/../
examples/dummy_memconfig_ types.sv,8|46): This assignment is a
violation of SystemVerilog strong typing rules for enumeration
datatypes.
12 `ovm_field_int (rdy_int, OVM_ALL_ON)

      I need a parser logic in perl. I need to read a log file line by
line and search for a keyword. Once the keyword is found in any line,
I need to remove the previous two lines and current line. My new log
file, should not have these three lines.

On most systems files are simply a sequence of bytes and you
can't cut out something from the middle. It's not like a deck
of cards, where each card is a line in the file, and where
you can pull out a few cards from the middle. You have to
copy everything from the old file to a new one (of course ex-
cept the bits you want to remove) and then replace the old
file by the new one. So in your case you have to keep three
lines in memory, writing the oldest one out if the newest
one does not contain the keyword you were looking for. Only
if it's in the newest line forget about the three lines and
copy all the rest of the contents of the old file to the new
one. Something like this should do:

#!/usr/bin/perl

use strict;
use warnings;

open my $in,  '<', 'log'     or die "Can't open log file.\n";
open my $out, '>', 'log.new' or die "Can't open replacement file.\n";

my @lines;
$lines[ 0 ] = <$in>;
$lines[ 1 ] = <$in>;

while ( <$in> ) {
    if ( /KEYWORD/ ) {
        while ( <$in> ) {
            print $out $_;
        }
        last;
    }
    print $out shift @lines;
    push @lines, $_;

}

close $out;
close $in;
rename 'log.new', 'log';

                                Regards, Jens

 

[Jens Thoms Toerring]

I am unable to use this script. It gives the output file with
the lines until the keyword is found. after the first occurance of the
keyword, starting from there all lines are getting deleted.
I am adding code for your reference. I want to find out ENUMERR which
is there in the line no.11. my resultant output file should not have
line no: 9,10,11. Remaining all should be intact.

endtask : collect_e_daita
3 |
4 ncvlog: *W,LBLMAT (../_src/_hard/_vc/lcdport_vc/sv/
nec_lcdport_vc_etype_monitor.sv,1 68|24): Label/name mismatch:
collect_e_data - collect_e_daita.
5 (`include file: ../_src/_hard/_vc/lcdport_vc/sv/
nec_lcdport_vc_etype_monitor.sv line 168, `include file: ../_src/
_hard/_vc/lcdport_vc/sv/nec_lcdport_vc_top.sv line 47, `include
file: ../_src/_hard/_tb/_lcdbif_top.sv line 26, file: ../_src/_hard/
_tb/nec _lcdbif_tb_top.sv line 39)
6 apb_trans_cov.sample();
7 |
8 ncvlog: *W,ECSSDM (./INCA_libs/irun.lnx86.06.20.nc/svpplib/h/
sivajar/lcdbif/_veri/_rando m/_src/_hard/_uvc/apb_uvc/sv/
apb_uvc_master_monitor.sv,149|24): The 'sample' met hod is called
upon a covergroup instance whose declaration has a default sampling
condit ion associated with it.
9 `ovm_field_int (rd_wr, OVM_ALL_ON)
10 |
11 ncvlog: *W,ENUMERR (../_src/_hard/_uvc/apb_uvc/examples/../
examples/dummy_memconfig_ types.sv,8|46): This assignment is a
violation of SystemVerilog strong typing rules for enumeration
datatypes.
12 `ovm_field_int (rdy_int, OVM_ALL_ON)

Did you try the corrected version (modulo the bug Leon Timmermans
pointed out)? Here it is with hopefully everything cleaned up:

-----8<------------------------------

#!/usr/bin/perl

use strict;
use warnings;

open my $in, '<', 'log' or die "Can't open log file.\n";
open my $out, '>', 'log.new' or die "Can't open replacement file.\n";

my @lines;
$lines[ 0 ] = <$in>;
$lines[ 1 ] = <$in>;

while ( <$in> ) {
if ( /ENUMERR/ ) {
@lines = ( );
last;
}
print $out shift @lines;
push @lines, $_;
}

if ( @lines ) {
print $out $_ for @lines;
} else {
while ( <$in> ) {
print $out $_;
}
}

rename 'log.new', 'log';

-----8<------------------------------

At least for your input file it deletes just the lines 9, 10 and
11, as you wanted to. But please note:

a) if 'ENUMERR' does appear in the first two lines nothing happens
to that lines, your specifications did not cover the case, so I
have no idea what you want to happen then
b) it does only remove the first instance 'ENUMERR' (and the two
lines before, you didn't tell what to do if it shows up a
second (or third etc.) time.
Regards, Jens

 

[rajesh]

       I am unable to use this script. It gives the output filewith
the lines until the keyword is found. after the first occurance of the
keyword, starting from there all lines are getting deleted.
I am adding code for your reference. I want to find out ENUMERR which
is there in the line no.11. my resultant output file should not have
line no: 9,10,11. Remaining all should be intact.
 endtask : collect_e_daita
    3                         |
    4 ncvlog: *W,LBLMAT (../_src/_hard/_vc/lcdport_vc/sv/
nec_lcdport_vc_etype_monitor.sv,1      68|24): Label/name mismatch:
collect_e_data - collect_e_daita.
    5 (`include file: ../_src/_hard/_vc/lcdport_vc/sv/
nec_lcdport_vc_etype_monitor.sv line       168, `include file: ../_src/
_hard/_vc/lcdport_vc/sv/nec_lcdport_vc_top.sv line 47,       `include
file: ../_src/_hard/_tb/_lcdbif_top.sv line 26, file: ../_src/_hard/
_tb/nec      _lcdbif_tb_top.sv line 39)
    6      apb_trans_cov.sample();
    7                         |
    8 ncvlog: *W,ECSSDM (./INCA_libs/irun.lnx86.06.20.nc/svpplib/h/
sivajar/lcdbif/_veri/_rando      m/_src/_hard/_uvc/apb_uvc/sv/
apb_uvc_master_monitor.sv,149|24): The 'sample' met      hod is called
upon a covergroup instance whose declaration has a default sampling
condit      ion associated with it.
    9     `ovm_field_int   (rd_wr,        OVM_ALL_ON)
   10                                               |
   11 ncvlog: *W,ENUMERR (../_src/_hard/_uvc/apb_uvc/examples/../
examples/dummy_memconfig_      types.sv,8|46): This assignment isa
violation of SystemVerilog strong typing rules for       enumeration
datatypes.
   12     `ovm_field_int   (rdy_int,        OVM_ALL_ON)

Did you try the corrected version (modulo the bug Leon Timmermans
pointed out)? Here it is with hopefully everything cleaned up:

-----8<------------------------------

#!/usr/bin/perl

use strict;
use warnings;

open my $in,  '<', 'log'     or die "Can't open log file.\n";
open my $out, '>', 'log.new' or die "Can't open replacement file.\n";

my @lines;
$lines[ 0 ] = <$in>;
$lines[ 1 ] = <$in>;

while ( <$in> ) {
    if ( /ENUMERR/ ) {
        @lines = ( );
        last;
    }
    print $out shift @lines;
    push @lines, $_;

}

if ( @lines ) {
    print $out $_ for @lines;} else {

    while ( <$in> ) {
        print $out $_;
    }

}

rename 'log.new', 'log';

-----8<------------------------------

At least for your input file it deletes just the lines 9, 10 and
11, as you wanted to. But please note:

a) if 'ENUMERR' does appear in the first two lines nothing happens
   to that lines, your specifications did not cover the case, so I
   have no idea what you want to happen then
b) it does only remove the first instance 'ENUMERR' (and the two
   lines before, you didn't tell what to do if it shows up a
   second (or third etc.) time.
                                    Regards, Jens
--
  \   Jens Thoms Toerring  ___      (e-mail address removed)
   \__________________________      http://toerring.de- Hide quoted text -

- Show quoted text -

Hi Jens,
Thanks for the Solution. It looks work fine. But What I
need is that it should happen through out the file where the keyword
occurs.

Thanks in Advance,
Rajesh

 

[Henry Law]

Hi,
I am unable to use this script. It gives the output file with
the lines until the keyword is found. after the first occurance of the
keyword, starting from there all lines are getting deleted.

Well did you try to fix it? Or did you just pop back here?

It's interesting how most people who post in this group asking for
someone to write their program for them get short shrift, yet this guy
gets a draft and some immediate corrections, and then comes back for
debugging help. Twice.

 

[Jens Thoms Toerring]

Thanks for the Solution. It looks work fine. But What I
need is that it should happen through out the file where the keyword
occurs.

Sorry, but that was never meant as a complete solution for
your (somewhat underspecified) problem but just as a sketch
of a way you could try to do it. If you have problems get-
ting it done yourself going from there don't hesitate to post
here for discussions and hints (but include your own attempt).
But if you just want a full-blown solution with all bells and
whistles without writing it yourself this is not the right
place to ask. Look instead for groups with the word 'jobs' in
their names (or websites like www.rentacoder.com).

Regards, Jens