real time log parser?

Discussion in 'Perl Misc' started by Tomasz Chmielewski, Oct 12, 2010.

  1. I would like to write a log parser which would work "in real time".

    Meaning, it will read i.e. /var/log/mail.info and append interesting
    entries it finds to a database, according to some criteria.


    What should I look at / read to achieve it? I'm OK to create a "static"
    parser like this, where the file is parsed once, but I don't have much
    experience with continuous processing of files which grow (and at times,
    are rotated/removed/truncated).


    --
    Tomasz Chmielewski
    http://wpkg.org
    Tomasz Chmielewski, Oct 12, 2010
    #1
    1. Advertising

  2. On 12/10/2010 13:33, Tomasz Chmielewski wrote:
    > I would like to write a log parser which would work "in real time".
    >
    > Meaning, it will read i.e. /var/log/mail.info and append interesting
    > entries it finds to a database, according to some criteria.
    >
    >
    > What should I look at / read to achieve it? I'm OK to create a "static"
    > parser like this, where the file is parsed once, but I don't have much
    > experience with continuous processing of files which grow (and at times,
    > are rotated/removed/truncated).
    >
    >



    (tail -f /var/log/mail.info | ./parser.pl) &


    --
    RGB
    RedGrittyBrick, Oct 12, 2010
    #2
    1. Advertising

  3. On 12.10.2010 15:06, RedGrittyBrick wrote:

    >> What should I look at / read to achieve it? I'm OK to create a "static"
    >> parser like this, where the file is parsed once, but I don't have much
    >> experience with continuous processing of files which grow (and at times,
    >> are rotated/removed/truncated).
    >>
    >>

    >
    >
    > (tail -f /var/log/mail.info | ./parser.pl) &


    And then, logrotate / restart syslog

    echo blah >> /var/log/mail.info

    Oops, nothing new gets to the parser!


    Besides, it'd be interesting to get rid of the tail binary, too.


    --
    Tomasz Chmielewski
    http://wpkg.org
    Tomasz Chmielewski, Oct 12, 2010
    #3
  4. On 12/10/2010 14:23, Tomasz Chmielewski wrote:
    > On 12.10.2010 15:06, RedGrittyBrick wrote:
    >
    >>> What should I look at / read to achieve it? I'm OK to create a "static"
    >>> parser like this, where the file is parsed once, but I don't have much
    >>> experience with continuous processing of files which grow (and at times,
    >>> are rotated/removed/truncated).
    >>>
    >>>

    >>
    >>
    >> (tail -f /var/log/mail.info | ./parser.pl) &

    >
    > And then,


    yes sorry.


    > logrotate


    kill and restart the parser using logrotate's postrotate feature?


    > / restart syslog


    ???


    >
    > Besides, it'd be interesting to get rid of the tail binary, too.


    "This is the Unix philosophy: Write programs that do one thing and do it
    well. Write programs to work together. Write programs to handle text
    streams, because that is a universal interface." -- Doug McIlroy

    --
    RGB
    RedGrittyBrick, Oct 12, 2010
    #4
  5. On 12/10/2010 14:33, RedGrittyBrick wrote:
    > On 12/10/2010 14:23, Tomasz Chmielewski wrote:
    >> On 12.10.2010 15:06, RedGrittyBrick wrote:
    >>
    >>>> What should I look at / read to achieve it? I'm OK to create a "static"
    >>>> parser like this, where the file is parsed once, but I don't have much
    >>>> experience with continuous processing of files which grow (and at
    >>>> times,
    >>>> are rotated/removed/truncated).
    >>>>
    >>>>
    >>>
    >>>
    >>> (tail -f /var/log/mail.info | ./parser.pl) &

    >>
    >> And then,

    >
    > yes sorry.
    >
    >
    >> logrotate

    >
    > kill and restart the parser using logrotate's postrotate feature?
    >
    >
    >> / restart syslog

    >
    > ???
    >
    >
    >>
    >> Besides, it'd be interesting to get rid of the tail binary, too.

    >
    > "This is the Unix philosophy: Write programs that do one thing and do it
    > well. Write programs to work together. Write programs to handle text
    > streams, because that is a universal interface." -- Doug McIlroy
    >



    I should have mentioned File::Tail. TIMTOWTDI after all.

    --
    RGB
    RedGrittyBrick, Oct 12, 2010
    #5
  6. Tomasz Chmielewski

    Ted Zlatanov Guest

    On Tue, 12 Oct 2010 15:23:38 +0200 Tomasz Chmielewski <> wrote:

    TC> On 12.10.2010 15:06, RedGrittyBrick wrote:
    >> (tail -f /var/log/mail.info | ./parser.pl) &


    TC> And then, logrotate / restart syslog

    TC> echo blah >> /var/log/mail.info

    TC> Oops, nothing new gets to the parser!

    That's why we use "tail -F" ("same as --follow=name --retry") if it's
    available :)

    TC> Besides, it'd be interesting to get rid of the tail binary, too.

    Well, not necessarily.

    Ted
    Ted Zlatanov, Oct 12, 2010
    #6
  7. On 12.10.2010 15:56, Ted Zlatanov wrote:

    > That's why we use "tail -F" ("same as --follow=name --retry") if it's
    > available :)


    Nice tip, thanks.


    > TC> Besides, it'd be interesting to get rid of the tail binary, too.
    >
    > Well, not necessarily.


    ....because I'd like the parser only to parse the lines it didn't parse
    before. And other featuritis like this.

    Say - the parser crashed for some reason or didn't run for half a day?
    Start it again, it will figure out where it last ended.

    Certainly, there are more ways to do it, but I don't think tail helps
    here a lot, quite the contrary. File::Tail is a good one, too, thanks
    for the tips.


    --
    Tomasz Chmielewski
    http://wpkg.org
    Tomasz Chmielewski, Oct 12, 2010
    #7
  8. > I would like to write a log parser which would work "in real time".

    i wrote a similar one at previous work to monitor oracle logs and send
    alerts to nagios server.

    > Meaning, it will read i.e. /var/log/mail.info and append interesting
    > entries it finds to a database, according to some criteria.
    >
    > What should I look at / read to achieve it? I'm OK to create a "static"


    i utilised file::tail module and ran the program as a daemon.
    file::tail is pretty good when the files are rotated so you dont have
    to keep track of offsets and inodes etc.

    also i created a small shell script and ran it as cron job to monitor
    the status of my perl daemon.
    alfonsobaldaserra, Oct 13, 2010
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. senthil
    Replies:
    5
    Views:
    1,365
    senthil
    Jan 24, 2004
  2. Curious Trigger
    Replies:
    2
    Views:
    1,799
    Curious Trigger
    Sep 9, 2006
  3. flamesrock
    Replies:
    8
    Views:
    438
    Hendrik van Rooyen
    Nov 24, 2006
  4. m
    Replies:
    2
    Views:
    400
    Shawn Milochik
    Aug 6, 2008
  5. Brian

    Real time converted to Unix time

    Brian, Aug 29, 2003, in forum: Perl Misc
    Replies:
    1
    Views:
    376
    James Willmore
    Aug 29, 2003
Loading...

Share This Page