real time log parser?

[Tomasz Chmielewski]

I would like to write a log parser which would work "in real time".

Meaning, it will read i.e. /var/log/mail.info and append interesting
entries it finds to a database, according to some criteria.


What should I look at / read to achieve it? I'm OK to create a "static"
parser like this, where the file is parsed once, but I don't have much
experience with continuous processing of files which grow (and at times,
are rotated/removed/truncated).

 

[RedGrittyBrick]

I would like to write a log parser which would work "in real time".

Meaning, it will read i.e. /var/log/mail.info and append interesting
entries it finds to a database, according to some criteria.


What should I look at / read to achieve it? I'm OK to create a "static"
parser like this, where the file is parsed once, but I don't have much
experience with continuous processing of files which grow (and at times,
are rotated/removed/truncated).

(tail -f /var/log/mail.info | ./parser.pl) &

 

[Tomasz Chmielewski]

(tail -f /var/log/mail.info | ./parser.pl) &

And then, logrotate / restart syslog

echo blah >> /var/log/mail.info

Oops, nothing new gets to the parser!


Besides, it'd be interesting to get rid of the tail binary, too.

 

[RedGrittyBrick]

And then,

yes sorry.

logrotate

kill and restart the parser using logrotate's postrotate feature?

/ restart syslog
???



Besides, it'd be interesting to get rid of the tail binary, too.

"This is the Unix philosophy: Write programs that do one thing and do it
well. Write programs to work together. Write programs to handle text
streams, because that is a universal interface." -- Doug McIlroy

 

[RedGrittyBrick]

yes sorry.



kill and restart the parser using logrotate's postrotate feature?



"This is the Unix philosophy: Write programs that do one thing and do it
well. Write programs to work together. Write programs to handle text
streams, because that is a universal interface." -- Doug McIlroy

I should have mentioned File::Tail. TIMTOWTDI after all.

 

[Ted Zlatanov]

TC> And then, logrotate / restart syslog

TC> echo blah >> /var/log/mail.info

TC> Oops, nothing new gets to the parser!

That's why we use "tail -F" ("same as --follow=name --retry") if it's
available

TC> Besides, it'd be interesting to get rid of the tail binary, too.

Well, not necessarily.

Ted

 

[Tomasz Chmielewski]

That's why we use "tail -F" ("same as --follow=name --retry") if it's
available

Nice tip, thanks.

TC> Besides, it'd be interesting to get rid of the tail binary, too.

Well, not necessarily.

....because I'd like the parser only to parse the lines it didn't parse
before. And other featuritis like this.

Say - the parser crashed for some reason or didn't run for half a day?
Start it again, it will figure out where it last ended.

Certainly, there are more ways to do it, but I don't think tail helps
here a lot, quite the contrary. File::Tail is a good one, too, thanks
for the tips.

 

[alfonsobaldaserra]

I would like to write a log parser which would work "in real time".

i wrote a similar one at previous work to monitor oracle logs and send
alerts to nagios server.

Meaning, it will read i.e. /var/log/mail.info and append interesting
entries it finds to a database, according to some criteria.

What should I look at / read to achieve it? I'm OK to create a "static"

i utilised file::tail module and ran the program as a daemon.
file::tail is pretty good when the files are rotated so you dont have
to keep track of offsets and inodes etc.

also i created a small shell script and ran it as cron job to monitor
the status of my perl daemon.