Q: Find, print, and remove lines from logfile

Discussion in 'Perl Misc' started by Magnus, Jan 20, 2004.

  1. Magnus

    Magnus Guest

    Hi!

    Lets say I have a logfile which looks like:

    Text bla bla...
    Text....
    foo1
    More text bla bla...
    Even more...
    bar1
    Text again...
    foo2
    and more..
    bar2
    bla bla..

    I know (I think) how to print foo1 and bar1 and whats between:

    while(<>){
    if (/foo1/ .. /bar1/) {
    print $_;
    }
    }

    But what if I want to search the file all over again and print foo2
    and bar2 and whats between and how do I remove the printed lines from
    the logfile?

    Any help is appreciated, even if it is just a hint.

    Thanks in advance!
    Magnus
     
    Magnus, Jan 20, 2004
    #1
    1. Advertising

  2. Magnus

    Tore Aursand Guest

    On Tue, 20 Jan 2004 08:01:25 -0800, Magnus wrote:
    > Lets say I have a logfile which looks like:
    >
    > Text bla bla...
    > Text....
    > foo1
    > More text bla bla...
    > Even more...
    > bar1
    > Text again...
    > foo2
    > and more..
    > bar2
    > bla bla..
    >
    > I know (I think) how to print foo1 and bar1 and whats between:
    >
    > while(<>){
    > if (/foo1/ .. /bar1/) {
    > print $_;
    > }
    > }
    >
    > But what if I want to search the file all over again and print foo2
    > and bar2 and whats between [...]


    Your requirements are quite vague; What's the _actual_ format of the
    logfile(s) you're dealing with? Why do you need to read the file over
    again (once should be enough, you know). Can't you just match on 'foo\d+'
    and the corresponding 'bar'?

    while ( <LOG> ) {
    print if ( /foo(\d+)/ .. /bar$1/ );
    }

    > [...] and how do I remove the printed lines from the logfile?


    Write the data you _want_ to a temporary file, delete the old file, and
    then rename the temporary file to the name of the old file.


    --
    Tore Aursand <>
    "War is too serious a matter to entrust to military men." -- Georges
    Clemenceau
     
    Tore Aursand, Jan 20, 2004
    #2
    1. Advertising

  3. In message <>, Magnus
    <> writes
    >Hi!
    >
    >Lets say I have a logfile which looks like:
    >
    >Text bla bla...
    >Text....
    >foo1
    >More text bla bla...
    >Even more...
    >bar1
    >Text again...
    >foo2
    >and more..
    >bar2
    >bla bla..
    >
    >I know (I think) how to print foo1 and bar1 and whats between:
    >
    >while(<>){
    > if (/foo1/ .. /bar1/) {
    > print $_;
    > }
    >}


    This won't work because having found foo1 you do not keep track of the
    fact so it will only print each time it finds a line with foo1 in it,
    not the lines following.

    At the risk of bringing down the wrath of the group on me, this is
    neater to do with the awk one-liner:

    #! awk
    /foo1/,/bar1/

    >But what if I want to search the file all over again and print foo2
    >and bar2 and whats between and how do I remove the printed lines from
    >the logfile?


    I'm not quite sure what you mean. Why do you want to remove the lines
    when you can test for foo2 and bar2 at the same time?

    #! awk
    /foo1/,/bar1/
    /foo2/,/bar2/

    will print out lines between foo1/bar1 and foo2/bar2. If you want
    foo2/bar2 without foo1/bar1 then simply use the second line and delete
    the first.

    Regards,
    --
    Bruce Horrocks
    Surrey
    England
    <firstname>@<surname>.plus.com -- fix the obvious for email
     
    Bruce Horrocks, Jan 20, 2004
    #3
  4. Magnus

    Ben Morrow Guest

    Bruce Horrocks <> wrote:
    > In message <>, Magnus
    > <> writes
    > >I know (I think) how to print foo1 and bar1 and whats between:
    > >
    > >while(<>){
    > > if (/foo1/ .. /bar1/) {
    > > print $_;
    > > }
    > >}

    >
    > This won't work because having found foo1 you do not keep track of the
    > fact so it will only print each time it finds a line with foo1 in it,
    > not the lines following.
    >
    > At the risk of bringing down the wrath of the group on me, this is
    > neater to do with the awk one-liner:
    >
    > #! awk
    > /foo1/,/bar1/


    You need to read up on what the .. operator does in scalar context :).

    Ben

    --
    $.=1;*g=sub{print@_};sub r($$\$){my($w,$x,$y)=@_;for(keys%$x){/main/&&next;*p=$
    $x{$_};/(\w)::$/&&(r($w.$1,$x.$_,$y),next);$y eq\$p&&&g("$w$_")}};sub t{for(@_)
    {$f&&($_||&g(" "));$f=1;r"","::",$_;$_&&&g(chr(0012))}};t #
    $J::u::s::t, $a::n::eek:::t::h::e::r, $P::e::r::l, $h::a::c::k::e::r, $.
     
    Ben Morrow, Jan 20, 2004
    #4
  5. Magnus

    Tore Aursand Guest

    On Tue, 20 Jan 2004 21:35:28 +0000, Bruce Horrocks wrote:
    >> while(<>){
    >> if (/foo1/ .. /bar1/) {
    >> print $_;
    >> }
    >> }


    > This won't work because having found foo1 you do not keep track of the
    > fact so it will only print each time it finds a line with foo1 in it,
    > not the lines following.


    Actually, it does. 'perldoc perlop' tells ut the following:

    In scalar context, ".." returns a boolean value. The operator is
    bistable, like a flip-flop, and emulates the line-range (comma) opera-
    tor of sed, awk, and various editors. Each ".." operator maintains its
    own boolean state. It is false as long as its left operand is false.
    Once the left operand is true, the range operator stays true until the
    right operand is true, AFTER which the range operator becomes false
    again. It doesn't become false till the next time the range operator is
    evaluated. It can test the right operand and become false on the same
    evaluation it became true (as in awk), but it still returns true once.


    --
    Tore Aursand <>
    "A teacher is never a giver of truth - he is a guide, a pointer to the
    truth that each student must find for himself. A good teacher is
    merely a catalyst." -- Bruce Lee
     
    Tore Aursand, Jan 20, 2004
    #5
  6. Magnus

    Magnus Guest

    Tore Aursand <> wrote in message news:<>...



    > Your requirements are quite vague; What's the _actual_ format of the
    > logfile(s) you're dealing with? Why do you need to read the file over
    > again (once should be enough, you know). Can't you just match on 'foo\d+'
    > and the corresponding 'bar'?


    The goal is to write a perlscript for analyzing exceptions and
    stacktraces in a huge java logfile. I want to print out known
    exceptions and there stacktraces and remove them from the logfile.

    > while ( <LOG> ) {
    > print if ( /foo(\d+)/ .. /bar$1/ );
    > }
    >
    > > [...] and how do I remove the printed lines from the logfile?

    >
    > Write the data you _want_ to a temporary file, delete the old file, and
    > then rename the temporary file to the name of the old file.


    Okey! I understand. Here is what I have done so far...I seems to work
    even if it slow because the logfile is so big.

    for-loop which set startstring and stopstring) {
    open (LOGFILE, "./logfile.txt");
    open (KNOWNEXCEPTIONS, ">./known.txt");
    open (UNKNOWNEXCEPTIONS, ">./unknown.txt");
    while (<LOGFILE>) {
    if (/startstring/ .. /stopstring/) {
    print KNOWNEXCEPTIONS "$_";
    }
    if (! (/startstring/ .. /stopstring/) {
    print UNKNOWNEXCEPTIONS "$_";
    }
    close the files...
    rename (unknown.txt, logfile.txt);
    }
    }

    Thank you for your help!

    Magnus
     
    Magnus, Jan 21, 2004
    #6
  7. Magnus

    Tore Aursand Guest

    On Wed, 21 Jan 2004 00:30:05 -0800, Magnus wrote:
    > for-loop which set startstring and stopstring) {
    > open (LOGFILE, "./logfile.txt");
    > open (KNOWNEXCEPTIONS, ">./known.txt");
    > open (UNKNOWNEXCEPTIONS, ">./unknown.txt");


    You _really_ want to know what goes wrong _when_ something goes wrong.
    You also want to drop those unnecessary double quotes;

    open(LOGFILE, './logfile.txt') or die "$!\n";

    > while (<LOGFILE>) {
    > if (/startstring/ .. /stopstring/) {
    > print KNOWNEXCEPTIONS "$_";
    > }
    > if (! (/startstring/ .. /stopstring/) {
    > print UNKNOWNEXCEPTIONS "$_";
    > }


    Still no need to use the double quotes, and please consider not using the
    '!' operator (for readability). That's my personal opionion. As far as I
    can see, the above can be written as:

    if ( /start/ .. /stop/ ) {
    print KNOWNEXCEPTIONS $_;
    }
    else {
    print UNKNOWNEXCEPTIONS $_;
    }

    > close the files...


    Do you _really_ want to close the files while processing them? I think
    you should move the close()'ing to the same scope as where you open()'ed
    the files.

    > rename (unknown.txt, logfile.txt);


    Same goes with this one.


    --
    Tore Aursand <>
    "Time only seems to matter when it's running out." -- Peter Strup
     
    Tore Aursand, Jan 21, 2004
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. keto
    Replies:
    0
    Views:
    964
  2. David Cournapeau

    print a vs print '%s' % a vs print '%f' a

    David Cournapeau, Dec 30, 2008, in forum: Python
    Replies:
    0
    Views:
    363
    David Cournapeau
    Dec 30, 2008
  3. Wolfgang
    Replies:
    1
    Views:
    157
    Paul Lalli
    Feb 13, 2004
  4. Replies:
    5
    Views:
    268
    Paul Lalli
    Mar 3, 2007
  5. Thomas Kratz

    threads and logfile rotation

    Thomas Kratz, Jul 30, 2007, in forum: Perl Misc
    Replies:
    5
    Views:
    82
    Thomas Kratz
    Aug 1, 2007
Loading...

Share This Page