Application log file parser

Discussion in 'Java' started by Ben_, Jun 25, 2004.

  1. Ben_

    Ben_ Guest

    Hello,

    Are there tools to parse application log files (typically stdout & stderr,
    but also Log4J output) in a smart way ?

    Basically, I'd like to parse them and compute stats like when, how many
    times and how often exceptions of a class have been raised.

    Format is predictable, because we'll look for well known patterns, but there
    may be so many different log statements to search for that I'm looking for a
    flexible and productive way of doing.

    Thanks.
     
    Ben_, Jun 25, 2004
    #1
    1. Advertising

  2. Ben_

    scott Guest

    The the new version of log4j's Chainsaw, configured to use a
    LogFilePatternReceiver, will do exactly this.

    LogFilePatternReceiver can parse events from a log file based on a
    pattern you provide, and can 'tail' the file as well.

    Chainsaw also provides a simple expression syntax which can be used to
    dynamically filter, find and colorize events.

    Here's a link to Chainsaw V2. It's available via Web Start:
    http://logging.apache.org/log4j/docs/chainsaw.html

    Here's a link to the LogFilePatternReceiver (read the javadoc for
    information on how to specify the pattern in your file and the
    'tailing' parameter):
    http://cvs.apache.org/viewcvs.cgi/l...ogFilePatternReceiver.java?rev=1.12&view=auto

    If you have questions, feel free to send them to the log4j-user
    mailing list.

    Scott




    Configure Chainsaw to use a LogFilePatternReceiver
    "Ben_" <> wrote in message news:<40dc8cb1$0$8981$>...
    > Hello,
    >
    > Are there tools to parse application log files (typically stdout & stderr,
    > but also Log4J output) in a smart way ?
    >
    > Basically, I'd like to parse them and compute stats like when, how many
    > times and how often exceptions of a class have been raised.
    >
    > Format is predictable, because we'll look for well known patterns, but there
    > may be so many different log statements to search for that I'm looking for a
    > flexible and productive way of doing.
    >
    > Thanks.
     
    scott, Jun 26, 2004
    #2
    1. Advertising

  3. Ben_

    Ben_ Guest

    I'll check the LogFilePatternReceiver.

    I think LogParser from Microsoft
    (http://www.microsoft.com/downloads/details.aspx?FamilyID=8cde4028-e247-45be
    -bab9-ac851fc166a4&DisplayLang=en) implements pretty much what I have in
    mind: it searches files for named-fields with SQL-like syntax. What's
    missing is the ability to search any file formats (it's targeted at http
    logs) by allowing to define new named-fields.
     
    Ben_, Jun 28, 2004
    #3
  4. Ben_ wrote:
    > Are there tools to parse application log files (typically stdout & stderr,
    > but also Log4J output) in a smart way ?
    >
    > Basically, I'd like to parse them and compute stats like when, how many
    > times and how often exceptions of a class have been raised.


    Perl had been invented to do things like this. But also the classic Unix
    tools like awk are still not bad at it. Construct regular expressions to
    match the format of the lines you are interested in, and use a hash
    (Perl) or associative array to count per event, class, or whatever you want.

    If you need some graphics, consider to generate simple tables from the
    accumulated data, and run gnuplot on them.

    /Thomas
     
    Thomas Weidenfeller, Jun 28, 2004
    #4
  5. Ben_

    Craig Guest

    Ok Scott/All, you got me interested. I downloaded Chainsaw, but the
    instructions on creating receivers is almost non-existant. Here's
    what I want to do:

    1) Run a Java app which creates a log file on a Unix server somewhere.
    The Java app does not use log4j to create entries -- its older, and
    uses its own internally defined class/methods to create log entries.
    2) Run Chainsaw on my local PC in order to look at the log. I'd like
    to use the "tail" feature to monitor events as they are being
    generated.

    What kind of connectivity is used by Chainsaw to montitor a remote
    file? Just HTTP? Do I need to NFS?

    I tried creating a receiver. It asks for a file name -- but not a UNC
    or server name/directory location. How do I specify this information?

    TIA!,
    -Craig


    (scott) wrote in message news:<>...
    > The the new version of log4j's Chainsaw, configured to use a
    > LogFilePatternReceiver, will do exactly this.
    >
    > LogFilePatternReceiver can parse events from a log file based on a
    > pattern you provide, and can 'tail' the file as well.
    >
    > Chainsaw also provides a simple expression syntax which can be used to
    > dynamically filter, find and colorize events.
    >
    > Here's a link to Chainsaw V2. It's available via Web Start:
    > http://logging.apache.org/log4j/docs/chainsaw.html
    >
    > Here's a link to the LogFilePatternReceiver (read the javadoc for
    > information on how to specify the pattern in your file and the
    > 'tailing' parameter):
    > http://cvs.apache.org/viewcvs.cgi/l...ogFilePatternReceiver.java?rev=1.12&view=auto
    >
    > If you have questions, feel free to send them to the log4j-user
    > mailing list.
    >
    > Scott
    >
    >
    >
    >
    > Configure Chainsaw to use a LogFilePatternReceiver
    > "Ben_" <> wrote in message news:<40dc8cb1$0$8981$>...
    > > Hello,
    > >
    > > Are there tools to parse application log files (typically stdout & stderr,
    > > but also Log4J output) in a smart way ?
    > >
    > > Basically, I'd like to parse them and compute stats like when, how many
    > > times and how often exceptions of a class have been raised.
    > >
    > > Format is predictable, because we'll look for well known patterns, but there
    > > may be so many different log statements to search for that I'm looking for a
    > > flexible and productive way of doing.
    > >
    > > Thanks.
     
    Craig, Jun 28, 2004
    #5
  6. Ben_

    scott Guest

    Hi Craig,

    Receivers are new to log4j1.3, which isn't released yet (will be in
    alpha soon), but we're using them with Chainsaw). We'll work on
    improving the docs.

    LogFilePatternReceiver expects a path to look like this:
    c:/somepath/somefile.

    We're working on adding the ability to process files accessible via
    VFS, but that'll be a while before it's available. Right now
    LogFilePatternReceiver expects a file path, so you'll probably need to
    use NFS for now.

    By the way, Chainsaw has built in support for ORO regular expressions
    (define a regexp using the LIKE operator). You can also combine
    expressions using parentheses and logical operators (&&, ||, !). Just
    make sure to put spaces around all operators, operands and
    parentheses.

    If you have fields in your log file that don't match to log4j's
    built-in fields, specify the field(s) in the logFormat as PROP(MYKEY)
    for example, and the field will end up in as the 'MYKEY' column in
    Chainsaw, which can then be used to filter expressions.

    Note that you can specify a filter expression, which can be used to
    pre-filter events from being processed by Chainsaw. Leave this param
    out if you want to load the entire file.

    If possible, specify a logger in the logFormat - it can be used in the
    logger tree panel to quickly access events without having to build an
    expression rule.

    The receiver may not process events if a logger field isn't specified
    (I can't recall).

    Here's a sample log4j.xml file that can be used as a Chainsaw
    configuration file that will process your file's events on Chainsaw's
    startup (specify this xml file here: view menu, show application-wide
    preferences, automatic configuration.

    log4j.xml:

    <?xml version="1.0" encoding="UTF-8" ?>
    <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
    <log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/"
    debug="true">
    <plugin name="LogFileReceiver"
    class="org.apache.log4j.varia.LogFilePatternReceiver">
    <param name="timestampFormat" value="yyyy-MM-d HH:mm:ss,SSS"/>
    <param name="logFormat" value="RELATIVETIME [THREAD] LEVEL
    LOGGER * - MESSAGE"/>
    <param name="fileName" value="c:/test/A4.log"/>
    <param name="filterExpression" value="LEVEL > WARN"/>
    <param name="tailing" value="true"/>
    </plugin>
    <root>
    <level value="debug"/>
    </root>
    </log4j:configuration>

    I hope this info helps. If you have further questions, let me know.

    Scott




    (Craig) wrote in message news:<>...
    > Ok Scott/All, you got me interested. I downloaded Chainsaw, but the
    > instructions on creating receivers is almost non-existant. Here's
    > what I want to do:
    >
    > 1) Run a Java app which creates a log file on a Unix server somewhere.
    > The Java app does not use log4j to create entries -- its older, and
    > uses its own internally defined class/methods to create log entries.
    > 2) Run Chainsaw on my local PC in order to look at the log. I'd like
    > to use the "tail" feature to monitor events as they are being
    > generated.
    >
    > What kind of connectivity is used by Chainsaw to montitor a remote
    > file? Just HTTP? Do I need to NFS?
    >
    > I tried creating a receiver. It asks for a file name -- but not a UNC
    > or server name/directory location. How do I specify this information?
    >
    > TIA!,
    > -Craig
    >
    >
    > (scott) wrote in message news:<>...
    > > The the new version of log4j's Chainsaw, configured to use a
    > > LogFilePatternReceiver, will do exactly this.
    > >
    > > LogFilePatternReceiver can parse events from a log file based on a
    > > pattern you provide, and can 'tail' the file as well.
    > >
    > > Chainsaw also provides a simple expression syntax which can be used to
    > > dynamically filter, find and colorize events.
    > >
    > > Here's a link to Chainsaw V2. It's available via Web Start:
    > > http://logging.apache.org/log4j/docs/chainsaw.html
    > >
    > > Here's a link to the LogFilePatternReceiver (read the javadoc for
    > > information on how to specify the pattern in your file and the
    > > 'tailing' parameter):
    > > http://cvs.apache.org/viewcvs.cgi/l...ogFilePatternReceiver.java?rev=1.12&view=auto
    > >
    > > If you have questions, feel free to send them to the log4j-user
    > > mailing list.
    > >
    > > Scott
    > >
    > >
    > >
    > >
    > > Configure Chainsaw to use a LogFilePatternReceiver
    > > "Ben_" <> wrote in message news:<40dc8cb1$0$8981$>...
    > > > Hello,
    > > >
    > > > Are there tools to parse application log files (typically stdout & stderr,
    > > > but also Log4J output) in a smart way ?
    > > >
    > > > Basically, I'd like to parse them and compute stats like when, how many
    > > > times and how often exceptions of a class have been raised.
    > > >
    > > > Format is predictable, because we'll look for well known patterns, but there
    > > > may be so many different log statements to search for that I'm looking for a
    > > > flexible and productive way of doing.
    > > >
    > > > Thanks.
     
    scott, Jun 29, 2004
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Amratash
    Replies:
    0
    Views:
    548
    Amratash
    Apr 13, 2004
  2. =?Utf-8?B?VG9tIFdpbmdlcnQ=?=

    My.Log.Writeexception not writing to Application Event Log.

    =?Utf-8?B?VG9tIFdpbmdlcnQ=?=, Jan 20, 2006, in forum: ASP .Net
    Replies:
    0
    Views:
    2,395
    =?Utf-8?B?VG9tIFdpbmdlcnQ=?=
    Jan 20, 2006
  3. unomystEz
    Replies:
    0
    Views:
    570
    unomystEz
    Nov 19, 2006
  4. Replies:
    18
    Views:
    1,531
    Jeff Higgins
    Jul 25, 2007
  5. Replies:
    0
    Views:
    332
Loading...

Share This Page