Application log file parser

[Ben_]

Hello,

Are there tools to parse application log files (typically stdout & stderr,
but also Log4J output) in a smart way ?

Basically, I'd like to parse them and compute stats like when, how many
times and how often exceptions of a class have been raised.

Format is predictable, because we'll look for well known patterns, but there
may be so many different log statements to search for that I'm looking for a
flexible and productive way of doing.

Thanks.

 

[scott]

The the new version of log4j's Chainsaw, configured to use a
LogFilePatternReceiver, will do exactly this.

LogFilePatternReceiver can parse events from a log file based on a
pattern you provide, and can 'tail' the file as well.

Chainsaw also provides a simple expression syntax which can be used to
dynamically filter, find and colorize events.

Here's a link to Chainsaw V2. It's available via Web Start:
http://logging.apache.org/log4j/docs/chainsaw.html

Here's a link to the LogFilePatternReceiver (read the javadoc for
information on how to specify the pattern in your file and the
'tailing' parameter):
http://cvs.apache.org/viewcvs.cgi/l...ogFilePatternReceiver.java?rev=1.12&view=auto

If you have questions, feel free to send them to the log4j-user
mailing list.

Scott
(e-mail address removed)



Configure Chainsaw to use a LogFilePatternReceiver

 

[Ben_]

I'll check the LogFilePatternReceiver.

I think LogParser from Microsoft
(http://www.microsoft.com/downloads/details.aspx?FamilyID=8cde4028-e247-45be
-bab9-ac851fc166a4&DisplayLang=en) implements pretty much what I have in
mind: it searches files for named-fields with SQL-like syntax. What's
missing is the ability to search any file formats (it's targeted at http
logs) by allowing to define new named-fields.

 

[Thomas Weidenfeller]

Are there tools to parse application log files (typically stdout & stderr,
but also Log4J output) in a smart way ?

Basically, I'd like to parse them and compute stats like when, how many
times and how often exceptions of a class have been raised.

Perl had been invented to do things like this. But also the classic Unix
tools like awk are still not bad at it. Construct regular expressions to
match the format of the lines you are interested in, and use a hash
(Perl) or associative array to count per event, class, or whatever you want.

If you need some graphics, consider to generate simple tables from the
accumulated data, and run gnuplot on them.

/Thomas

 

[Craig]

Ok Scott/All, you got me interested. I downloaded Chainsaw, but the
instructions on creating receivers is almost non-existant. Here's
what I want to do:

1) Run a Java app which creates a log file on a Unix server somewhere.
The Java app does not use log4j to create entries -- its older, and
uses its own internally defined class/methods to create log entries.
2) Run Chainsaw on my local PC in order to look at the log. I'd like
to use the "tail" feature to monitor events as they are being
generated.

What kind of connectivity is used by Chainsaw to montitor a remote
file? Just HTTP? Do I need to NFS?

I tried creating a receiver. It asks for a file name -- but not a UNC
or server name/directory location. How do I specify this information?

TIA!,
-Craig

 

[scott]

Hi Craig,

Receivers are new to log4j1.3, which isn't released yet (will be in
alpha soon), but we're using them with Chainsaw). We'll work on
improving the docs.

LogFilePatternReceiver expects a path to look like this:
c:/somepath/somefile.

We're working on adding the ability to process files accessible via
VFS, but that'll be a while before it's available. Right now
LogFilePatternReceiver expects a file path, so you'll probably need to
use NFS for now.

By the way, Chainsaw has built in support for ORO regular expressions
(define a regexp using the LIKE operator). You can also combine
expressions using parentheses and logical operators (&&, ||, !). Just
make sure to put spaces around all operators, operands and
parentheses.

If you have fields in your log file that don't match to log4j's
built-in fields, specify the field(s) in the logFormat as PROP(MYKEY)
for example, and the field will end up in as the 'MYKEY' column in
Chainsaw, which can then be used to filter expressions.

Note that you can specify a filter expression, which can be used to
pre-filter events from being processed by Chainsaw. Leave this param
out if you want to load the entire file.

If possible, specify a logger in the logFormat - it can be used in the
logger tree panel to quickly access events without having to build an
expression rule.

The receiver may not process events if a logger field isn't specified
(I can't recall).

Here's a sample log4j.xml file that can be used as a Chainsaw
configuration file that will process your file's events on Chainsaw's
startup (specify this xml file here: view menu, show application-wide
preferences, automatic configuration.

log4j.xml:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/"
debug="true">
<plugin name="LogFileReceiver"
class="org.apache.log4j.varia.LogFilePatternReceiver">
<param name="timestampFormat" value="yyyy-MM-d HH:mm:ss,SSS"/>
<param name="logFormat" value="RELATIVETIME [THREAD] LEVEL
LOGGER * - MESSAGE"/>
<param name="fileName" value="c:/test/A4.log"/>
<param name="filterExpression" value="LEVEL > WARN"/>
<param name="tailing" value="true"/>
</plugin>
<root>
<level value="debug"/>
</root>
</log4j:configuration>

I hope this info helps. If you have further questions, let me know.

Scott

(e-mail address removed)