log parsing: gaps between transactions

Discussion in 'Perl' started by Arthur Dent, Dec 2, 2003.

  1. Arthur Dent

    Arthur Dent Guest

    I have a log file in this format:

    YYYYMMDD HHMMSS DATADATADATADATADATA RESULTCODE

    and I want to be able to parse the log and gleen information about gaps in
    time between transactions. For example, it would be nice to know that 8
    times in the past 24 hours we had periods over 5 minutes where no
    transactions were successful. In order to do that, I need to compare each
    line against the next based on result code and time elapsed. To make this
    a bit more tricky, not every line in the log is in this format (approx 45%
    are) but I only want to parse the lines that are. To add another problem,
    I am an inexperienced perl programmer. If someone can point me in the
    right direction it would be most appreciated. I have all of the O'Reilly
    books and have referenced them but nothing seems to be close to what I
    need. I also did an extensive search on deja before posting.

    Thanks,

    Arthur
     
    Arthur Dent, Dec 2, 2003
    #1
    1. Advertising

  2. Arthur Dent

    Jim Gibson Guest

    In article <qn8ihvdw8iyp.s3q484ucmkd5$>, Arthur Dent
    <> wrote:

    > I have a log file in this format:
    >
    > YYYYMMDD HHMMSS DATADATADATADATADATA RESULTCODE
    >
    > and I want to be able to parse the log and gleen information about gaps in
    > time between transactions. For example, it would be nice to know that 8
    > times in the past 24 hours we had periods over 5 minutes where no
    > transactions were successful. In order to do that, I need to compare each
    > line against the next based on result code and time elapsed. To make this
    > a bit more tricky, not every line in the log is in this format (approx 45%
    > are) but I only want to parse the lines that are. To add another problem,
    > I am an inexperienced perl programmer. If someone can point me in the
    > right direction it would be most appreciated. I have all of the O'Reilly
    > books and have referenced them but nothing seems to be close to what I
    > need. I also did an extensive search on deja before posting.
    >
    > Thanks,
    >
    > Arthur


    Have you tried any code? Do you have access to the perl online
    documentation (try "perldoc perl" at a command line)? Check out
    'perldoc perlquickre' for regular expressions and 'perldoc perlfaq5'
    for file I/O.

    Do you know how to read text files? You can identify the transaction
    lines and extract the time information with a regular expression such
    as

    /(\d{8})\s+(\d\d)(\d\d)(\d\d)/

    This looks for and saves an 8-digit date, followed by some white space,
    followed by 3 sets of 2-digit values. If your times are in 24-hour
    format, you can compute a time-since-midnight easily. Here is a sample
    program:

    #!/opt/perl/bin/perl

    use strict;
    use warnings;

    my $prev = 86400;
    while(<DATA>) {
    if( /(\d{8})\s+(\d\d)(\d\d)(\d\d)/ ) {
    my $date = $1;
    my $time = $4 + 60*( $3 + 60*$2);
    my $gap = $time - $prev;
    if( $gap > 300 ) {
    print "date = $date, time = $time, prev = $prev, gap = $gap
    seconds\n";
    }
    $prev = $time;
    }
    }

    __DATA__
    line 1
    line 2
    20031127 123456 data1 code1
    line 3
    line 4
    20031127 123457 data2 code2
    line 5
    line 6
    20031127 123958 data3 code3
    line 7
    line 8
    20031127 124000 data4 code4
    line 9
    line 10
    line 11

    __END__


    Output:
    date = 20031127, time = 45598, prev = 45297, gap = 301 seconds

    You will have to modify this program to handle the case around
    midnight, but that shouldn't be too hard. If you need more accurate,
    absolute time, check out the Time::parseDate module from
    http://www.cpan.org/

    FYI: this newsgroup is defunct. Try comp.lang.perl.misc in the future,
    paying attention to the guidelines for that group, available at
    http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
     
    Jim Gibson, Dec 3, 2003
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. greg
    Replies:
    4
    Views:
    389
  2. Mark247
    Replies:
    1
    Views:
    633
    Marrow
    Sep 3, 2004
  3. Andrey
    Replies:
    6
    Views:
    377
    Andrew Koenig
    Jun 24, 2004
  4. Replies:
    1
    Views:
    568
    Jonathan N. Little
    Nov 7, 2006
  5. cool2005

    Eliminate gaps between table cells...

    cool2005, May 20, 2005, in forum: Javascript
    Replies:
    5
    Views:
    141
    enrique
    May 20, 2005
Loading...

Share This Page