remove previous lines of a log file

Discussion in 'Perl Misc' started by rajesh, Jul 12, 2008.

  1. rajesh

    rajesh Guest

    Hi,
    I need a parser logic in perl. I need to read a log file line by
    line and search for a keyword. Once the keyword is found in any line,
    I need to remove the previous two lines and current line. My new log
    file, should not have these three lines.

    Thanks in Advance
    Rajesh
    rajesh, Jul 12, 2008
    #1
    1. Advertising

  2. rajesh <> wrote:
    > I need a parser logic in perl. I need to read a log file line by
    > line and search for a keyword. Once the keyword is found in any line,
    > I need to remove the previous two lines and current line. My new log
    > file, should not have these three lines.


    On most systems files are simply a sequence of bytes and you
    can't cut out something from the middle. It's not like a deck
    of cards, where each card is a line in the file, and where
    you can pull out a few cards from the middle. You have to
    copy everything from the old file to a new one (of course ex-
    cept the bits you want to remove) and then replace the old
    file by the new one. So in your case you have to keep three
    lines in memory, writing the oldest one out if the newest
    one does not contain the keyword you were looking for. Only
    if it's in the newest line forget about the three lines and
    copy all the rest of the contents of the old file to the new
    one. Something like this should do:

    #!/usr/bin/perl

    use strict;
    use warnings;

    open my $in, '<', 'log' or die "Can't open log file.\n";
    open my $out, '>', 'log.new' or die "Can't open replacement file.\n";

    my @lines;
    $lines[ 0 ] = <$in>;
    $lines[ 1 ] = <$in>;

    while ( <$in> ) {
    if ( /KEYWORD/ ) {
    while ( <$in> ) {
    print $out $_;
    }
    last;
    }
    print $out shift @lines;
    push @lines, $_;
    }

    close $out;
    close $in;
    rename 'log.new', 'log';

    Regards, Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
    Jens Thoms Toerring, Jul 12, 2008
    #2
    1. Advertising

  3. rajesh

    Dave B Guest

    rajesh wrote:

    > Hi,
    > I need a parser logic in perl. I need to read a log file line by
    > line and search for a keyword. Once the keyword is found in any line,
    > I need to remove the previous two lines and current line. My new log
    > file, should not have these three lines.
    >
    > Thanks in Advance
    > Rajesh


    You don't say whether the keyword can appear multiple times, and if so, if
    lines can be overlapping. I'll assume the keyword can appear multiple times,
    but lines do not overlap, and the file has at least three lines.

    A solution that reads the file line by line has already been given.
    If you can afford to slurp the whole file, then

    $f=~s/(?:.*\n){2}.*keyword.*\n//g;

    should do what you want. (Note that the last line in the file should be
    correctly terminated by a newline).

    --
    D.
    Dave B, Jul 12, 2008
    #3
  4. Jens Thoms Toerring <> wrote:
    > rajesh <> wrote:
    > > I need a parser logic in perl. I need to read a log file line by
    > > line and search for a keyword. Once the keyword is found in any line,
    > > I need to remove the previous two lines and current line. My new log
    > > file, should not have these three lines.


    > On most systems files are simply a sequence of bytes and you
    > can't cut out something from the middle. It's not like a deck
    > of cards, where each card is a line in the file, and where
    > you can pull out a few cards from the middle. You have to
    > copy everything from the old file to a new one (of course ex-
    > cept the bits you want to remove) and then replace the old
    > file by the new one. So in your case you have to keep three
    > lines in memory, writing the oldest one out if the newest
    > one does not contain the keyword you were looking for. Only
    > if it's in the newest line forget about the three lines and
    > copy all the rest of the contents of the old file to the new
    > one. Something like this should do:


    > #!/usr/bin/perl


    > use strict;
    > use warnings;


    > open my $in, '<', 'log' or die "Can't open log file.\n";
    > open my $out, '>', 'log.new' or die "Can't open replacement file.\n";


    > my @lines;
    > $lines[ 0 ] = <$in>;
    > $lines[ 1 ] = <$in>;


    > while ( <$in> ) {
    > if ( /KEYWORD/ ) {
    > while ( <$in> ) {
    > print $out $_;
    > }
    > last;
    > }
    > print $out shift @lines;
    > push @lines, $_;
    > }


    Sorry, just noticed that this will lose the last two lines
    if 'KEYWORD' is never found, so change that to

    while ( <$in> ) {
    if ( /KEYWORD/ ) {
    @lines = ( );
    last;
    }
    print $out shift @lines;
    push @lines, $_;
    }

    if ( @lines ) {
    print $out, $_ for @lines;
    } else {
    while ( <$in> ) {
    print $out, $_;
    }
    }

    or something along that line.

    > close $out;
    > close $in;
    > rename 'log.new', 'log';


    Regards, Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
    Jens Thoms Toerring, Jul 12, 2008
    #4
  5. On Sat, 12 Jul 2008 20:03:11 +0000, Jens Thoms Toerring wrote:
    >
    > if ( @lines ) {
    > print $out, $_ for @lines;
    > } else {
    > while ( <$in> ) {
    > print $out, $_;
    > }
    > }
    >


    I'd skip those comma's after $out though ;-)

    Leon Timmermans
    Leon Timmermans, Jul 13, 2008
    #5
  6. On Sat, 12 Jul 2008 18:34:53 +0200, Dave B wrote:

    > $f=~s/(?:.*\n){2}.*keyword.*\n//g;
    >
    > should do what you want. (Note that the last line in the file should be
    > correctly terminated by a newline).


    I think that last issue can be solved by the /m modifier

    $f =~ s/ (?:.*\n){2} .* keyword .* $ //xmg;

    Leon Timmermans
    Leon Timmermans, Jul 13, 2008
    #6
  7. Leon Timmermans <> wrote:
    > On Sat, 12 Jul 2008 20:03:11 +0000, Jens Thoms Toerring wrote:
    > >
    > > if ( @lines ) {
    > > print $out, $_ for @lines;
    > > } else {
    > > while ( <$in> ) {
    > > print $out, $_;
    > > }
    > > }
    > >


    > I'd skip those comma's after $out though ;-)


    I guess I also would do so after a bit more of thinking;-)

    Regards, Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
    Jens Thoms Toerring, Jul 13, 2008
    #7
  8. rajesh

    Dave B Guest

    Leon Timmermans wrote:
    > On Sat, 12 Jul 2008 18:34:53 +0200, Dave B wrote:
    >
    >> $f=~s/(?:.*\n){2}.*keyword.*\n//g;
    >>
    >> should do what you want. (Note that the last line in the file should be
    >> correctly terminated by a newline).

    >
    > I think that last issue can be solved by the /m modifier
    >
    > $f =~ s/ (?:.*\n){2} .* keyword .* $ //xmg;


    That will take care of the corner case, but will leave an unnecessary \n in
    the file in the regular cases.

    This should do the job:

    $f=~s/(?:.*\n){2}.*keyword.*(?:\n|$)//g;

    --
    D.
    Dave B, Jul 13, 2008
    #8
  9. rajesh

    rajesh Guest

    Hi,
    I am unable to use this script. It gives the output file with
    the lines until the keyword is found. after the first occurance of the
    keyword, starting from there all lines are getting deleted.
    I am adding code for your reference. I want to find out ENUMERR which
    is there in the line no.11. my resultant output file should not have
    line no: 9,10,11. Remaining all should be intact.

    Thanks in Advance
    rajesh


    endtask : collect_e_daita
    3 |
    4 ncvlog: *W,LBLMAT (../_src/_hard/_vc/lcdport_vc/sv/
    nec_lcdport_vc_etype_monitor.sv,1 68|24): Label/name mismatch:
    collect_e_data - collect_e_daita.
    5 (`include file: ../_src/_hard/_vc/lcdport_vc/sv/
    nec_lcdport_vc_etype_monitor.sv line 168, `include file: ../_src/
    _hard/_vc/lcdport_vc/sv/nec_lcdport_vc_top.sv line 47, `include
    file: ../_src/_hard/_tb/_lcdbif_top.sv line 26, file: ../_src/_hard/
    _tb/nec _lcdbif_tb_top.sv line 39)
    6 apb_trans_cov.sample();
    7 |
    8 ncvlog: *W,ECSSDM (./INCA_libs/irun.lnx86.06.20.nc/svpplib/h/
    sivajar/lcdbif/_veri/_rando m/_src/_hard/_uvc/apb_uvc/sv/
    apb_uvc_master_monitor.sv,149|24): The 'sample' met hod is called
    upon a covergroup instance whose declaration has a default sampling
    condit ion associated with it.
    9 `ovm_field_int (rd_wr, OVM_ALL_ON)
    10 |
    11 ncvlog: *W,ENUMERR (../_src/_hard/_uvc/apb_uvc/examples/../
    examples/dummy_memconfig_ types.sv,8|46): This assignment is a
    violation of SystemVerilog strong typing rules for enumeration
    datatypes.
    12 `ovm_field_int (rdy_int, OVM_ALL_ON)


    On Jul 12, 9:24 pm, (Jens Thoms Toerring) wrote:
    > rajesh <> wrote:
    > >       I need a parser logic in perl. I need to read a log file line by
    > > line and search for a keyword. Once the keyword is found in any line,
    > > I need to remove the previous two lines and current line. My new log
    > > file, should not have these three lines.

    >
    > On most systems files are simply a sequence of bytes and you
    > can't cut out something from the middle. It's not like a deck
    > of cards, where each card is a line in the file, and where
    > you can pull out a few cards from the middle. You have to
    > copy everything from the old file to a new one (of course ex-
    > cept the bits you want to remove) and then replace the old
    > file by the new one. So in your case you have to keep three
    > lines in memory, writing the oldest one out if the newest
    > one does not contain the keyword you were looking for. Only
    > if it's in the newest line forget about the three lines and
    > copy all the rest of the contents of the old file to the new
    > one. Something like this should do:
    >
    > #!/usr/bin/perl
    >
    > use strict;
    > use warnings;
    >
    > open my $in,  '<', 'log'     or die "Can't open log file.\n";
    > open my $out, '>', 'log.new' or die "Can't open replacement file.\n";
    >
    > my @lines;
    > $lines[ 0 ] = <$in>;
    > $lines[ 1 ] = <$in>;
    >
    > while ( <$in> ) {
    >     if ( /KEYWORD/ ) {
    >         while ( <$in> ) {
    >             print $out $_;
    >         }
    >         last;
    >     }
    >     print $out shift @lines;
    >     push @lines, $_;
    >
    > }
    >
    > close $out;
    > close $in;
    > rename 'log.new', 'log';
    >
    >                                 Regards, Jens
    > --
    >   \   Jens Thoms Toerring  ___      
    >    \__________________________      http://toerring.de
    rajesh, Jul 15, 2008
    #9
  10. rajesh <> wrote:
    > I am unable to use this script. It gives the output file with
    > the lines until the keyword is found. after the first occurance of the
    > keyword, starting from there all lines are getting deleted.
    > I am adding code for your reference. I want to find out ENUMERR which
    > is there in the line no.11. my resultant output file should not have
    > line no: 9,10,11. Remaining all should be intact.


    > endtask : collect_e_daita
    > 3 |
    > 4 ncvlog: *W,LBLMAT (../_src/_hard/_vc/lcdport_vc/sv/
    > nec_lcdport_vc_etype_monitor.sv,1 68|24): Label/name mismatch:
    > collect_e_data - collect_e_daita.
    > 5 (`include file: ../_src/_hard/_vc/lcdport_vc/sv/
    > nec_lcdport_vc_etype_monitor.sv line 168, `include file: ../_src/
    > _hard/_vc/lcdport_vc/sv/nec_lcdport_vc_top.sv line 47, `include
    > file: ../_src/_hard/_tb/_lcdbif_top.sv line 26, file: ../_src/_hard/
    > _tb/nec _lcdbif_tb_top.sv line 39)
    > 6 apb_trans_cov.sample();
    > 7 |
    > 8 ncvlog: *W,ECSSDM (./INCA_libs/irun.lnx86.06.20.nc/svpplib/h/
    > sivajar/lcdbif/_veri/_rando m/_src/_hard/_uvc/apb_uvc/sv/
    > apb_uvc_master_monitor.sv,149|24): The 'sample' met hod is called
    > upon a covergroup instance whose declaration has a default sampling
    > condit ion associated with it.
    > 9 `ovm_field_int (rd_wr, OVM_ALL_ON)
    > 10 |
    > 11 ncvlog: *W,ENUMERR (../_src/_hard/_uvc/apb_uvc/examples/../
    > examples/dummy_memconfig_ types.sv,8|46): This assignment is a
    > violation of SystemVerilog strong typing rules for enumeration
    > datatypes.
    > 12 `ovm_field_int (rdy_int, OVM_ALL_ON)


    Did you try the corrected version (modulo the bug Leon Timmermans
    pointed out)? Here it is with hopefully everything cleaned up:

    -----8<------------------------------

    #!/usr/bin/perl

    use strict;
    use warnings;

    open my $in, '<', 'log' or die "Can't open log file.\n";
    open my $out, '>', 'log.new' or die "Can't open replacement file.\n";

    my @lines;
    $lines[ 0 ] = <$in>;
    $lines[ 1 ] = <$in>;

    while ( <$in> ) {
    if ( /ENUMERR/ ) {
    @lines = ( );
    last;
    }
    print $out shift @lines;
    push @lines, $_;
    }

    if ( @lines ) {
    print $out $_ for @lines;
    } else {
    while ( <$in> ) {
    print $out $_;
    }
    }

    rename 'log.new', 'log';

    -----8<------------------------------

    At least for your input file it deletes just the lines 9, 10 and
    11, as you wanted to. But please note:

    a) if 'ENUMERR' does appear in the first two lines nothing happens
    to that lines, your specifications did not cover the case, so I
    have no idea what you want to happen then
    b) it does only remove the first instance 'ENUMERR' (and the two
    lines before, you didn't tell what to do if it shows up a
    second (or third etc.) time.
    Regards, Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
    Jens Thoms Toerring, Jul 16, 2008
    #10
  11. rajesh

    rajesh Guest

    On Jul 16, 5:44 am, (Jens Thoms Toerring) wrote:
    > rajesh <> wrote:
    > >        I am unable to use this script. It gives the output filewith
    > > the lines until the keyword is found. after the first occurance of the
    > > keyword, starting from there all lines are getting deleted.
    > > I am adding code for your reference. I want to find out ENUMERR which
    > > is there in the line no.11. my resultant output file should not have
    > > line no: 9,10,11. Remaining all should be intact.
    > >  endtask : collect_e_daita
    > >     3                         |
    > >     4 ncvlog: *W,LBLMAT (../_src/_hard/_vc/lcdport_vc/sv/
    > > nec_lcdport_vc_etype_monitor.sv,1      68|24): Label/name mismatch:
    > > collect_e_data - collect_e_daita.
    > >     5 (`include file: ../_src/_hard/_vc/lcdport_vc/sv/
    > > nec_lcdport_vc_etype_monitor.sv line       168, `include file: ../_src/
    > > _hard/_vc/lcdport_vc/sv/nec_lcdport_vc_top.sv line 47,       `include
    > > file: ../_src/_hard/_tb/_lcdbif_top.sv line 26, file: ../_src/_hard/
    > > _tb/nec      _lcdbif_tb_top.sv line 39)
    > >     6      apb_trans_cov.sample();
    > >     7                         |
    > >     8 ncvlog: *W,ECSSDM (./INCA_libs/irun.lnx86.06.20.nc/svpplib/h/
    > > sivajar/lcdbif/_veri/_rando      m/_src/_hard/_uvc/apb_uvc/sv/
    > > apb_uvc_master_monitor.sv,149|24): The 'sample' met      hod is called
    > > upon a covergroup instance whose declaration has a default sampling
    > > condit      ion associated with it.
    > >     9     `ovm_field_int   (rd_wr,        OVM_ALL_ON)
    > >    10                                               |
    > >    11 ncvlog: *W,ENUMERR (../_src/_hard/_uvc/apb_uvc/examples/../
    > > examples/dummy_memconfig_      types.sv,8|46): This assignment isa
    > > violation of SystemVerilog strong typing rules for       enumeration
    > > datatypes.
    > >    12     `ovm_field_int   (rdy_int,        OVM_ALL_ON)

    >
    > Did you try the corrected version (modulo the bug Leon Timmermans
    > pointed out)? Here it is with hopefully everything cleaned up:
    >
    > -----8<------------------------------
    >
    > #!/usr/bin/perl
    >
    > use strict;
    > use warnings;
    >
    > open my $in,  '<', 'log'     or die "Can't open log file.\n";
    > open my $out, '>', 'log.new' or die "Can't open replacement file.\n";
    >
    > my @lines;
    > $lines[ 0 ] = <$in>;
    > $lines[ 1 ] = <$in>;
    >
    > while ( <$in> ) {
    >     if ( /ENUMERR/ ) {
    >         @lines = ( );
    >         last;
    >     }
    >     print $out shift @lines;
    >     push @lines, $_;
    >
    > }
    >
    > if ( @lines ) {
    >     print $out $_ for @lines;} else {
    >
    >     while ( <$in> ) {
    >         print $out $_;
    >     }
    >
    > }
    >
    > rename 'log.new', 'log';
    >
    > -----8<------------------------------
    >
    > At least for your input file it deletes just the lines 9, 10 and
    > 11, as you wanted to. But please note:
    >
    > a) if 'ENUMERR' does appear in the first two lines nothing happens
    >    to that lines, your specifications did not cover the case, so I
    >    have no idea what you want to happen then
    > b) it does only remove the first instance 'ENUMERR' (and the two
    >    lines before, you didn't tell what to do if it shows up a
    >    second (or third etc.) time.
    >                                     Regards, Jens
    > --
    >   \   Jens Thoms Toerring  ___      
    >    \__________________________      http://toerring.de- Hide quoted text -
    >
    > - Show quoted text -


    Hi Jens,
    Thanks for the Solution. It looks work fine. But What I
    need is that it should happen through out the file where the keyword
    occurs.

    Thanks in Advance,
    Rajesh
    rajesh, Jul 17, 2008
    #11
  12. rajesh

    Henry Law Guest

    rajesh wrote:
    > Hi,
    > I am unable to use this script. It gives the output file with
    > the lines until the keyword is found. after the first occurance of the
    > keyword, starting from there all lines are getting deleted.


    Well did you try to fix it? Or did you just pop back here?

    It's interesting how most people who post in this group asking for
    someone to write their program for them get short shrift, yet this guy
    gets a draft and some immediate corrections, and then comes back for
    debugging help. Twice.

    --

    Henry Law Manchester, England
    Henry Law, Jul 17, 2008
    #12
  13. rajesh <> wrote:
    > On Jul 16, 5:44 am, (Jens Thoms Toerring) wrote:
    > > At least for your input file it deletes just the lines 9, 10 and
    > > 11, as you wanted to. But please note:
    > >
    > > a) if 'ENUMERR' does appear in the first two lines nothing happens
    > >    to that lines, your specifications did not cover the case, so I
    > >    have no idea what you want to happen then
    > > b) it does only remove the first instance 'ENUMERR' (and the two
    > >    lines before, you didn't tell what to do if it shows up a
    > >    second (or third etc.) time.

    >
    > Thanks for the Solution. It looks work fine. But What I
    > need is that it should happen through out the file where the keyword
    > occurs.


    Sorry, but that was never meant as a complete solution for
    your (somewhat underspecified) problem but just as a sketch
    of a way you could try to do it. If you have problems get-
    ting it done yourself going from there don't hesitate to post
    here for discussions and hints (but include your own attempt).
    But if you just want a full-blown solution with all bells and
    whistles without writing it yourself this is not the right
    place to ask. Look instead for groups with the word 'jobs' in
    their names (or websites like www.rentacoder.com).

    Regards, Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
    Jens Thoms Toerring, Jul 18, 2008
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Amratash
    Replies:
    0
    Views:
    501
    Amratash
    Apr 13, 2004
  2. Joe Wright
    Replies:
    0
    Views:
    499
    Joe Wright
    Jul 27, 2003
  3. unomystEz
    Replies:
    0
    Views:
    532
    unomystEz
    Nov 19, 2006
  4. Murali
    Replies:
    2
    Views:
    544
    Jerry Coffin
    Mar 9, 2006
  5. Tom Robinson
    Replies:
    8
    Views:
    211
    Ryan Davis
    Jul 29, 2006
Loading...

Share This Page