Best way to read, and analyze a log file?

Discussion in 'Python' started by superflit@gmail.com, Aug 1, 2006.

  1. Guest

    Hi All,

    I am reading a log file, and wondering what is the best way to read and
    analize this.
    I am think in two options:

    1- Read the data and put all variables in a list
    2- Read the data and put all the variables in dictionary?

    the logs is in this format

    xxxxxxxxxxxxxxxxxxxxxxxxxx
    The separation is by byte size like

    xxx three bytes for code x , xxxx bytes for hour, etc..
    I have two main objectives.
    Show all data to user.
    Analyze the data: like total sum, average,etc..

    All advice will be great appreciated.
    And sorry about my bad english.
     
    , Aug 1, 2006
    #1
    1. Advertising

  2. Gregor Horvath, Aug 1, 2006
    #2
    1. Advertising

  3. flit Guest

    Not exactly Wha I am looking, this seems to be a good project, but the
    way to parse the file I already have done that.
    I am looking in "desing pattern" advices, how to deal with the data.
    Must I use dbm? Must I make all the statistics and save in a dmf file?
    Any insights?

    Gregor Horvath wrote:
    > Hi,
    >
    > schrieb:
    >
    > > I am reading a log file, and wondering what is the best way to read and
    > > analize this.

    >
    > Look at:
    >
    > http://pyparsing.wikispaces.com/
    >
    > There is also an example for parsing an apache log.
    >
    > --
    > Servus, Gregor
    > http://www.gregor-horvath.com
     
    flit, Aug 1, 2006
    #3
  4. Guest

    :

    > 1- Read the data and put all variables in a list
    > 2- Read the data and put all the variables in dictionary?
    > the logs is in this format
    > xxxxxxxxxxxxxxxxxxxxxxxxxx
    > The separation is by byte size like
    > xxx three bytes for code x , xxxx bytes for hour, etc..
    > I have two main objectives.
    > Show all data to user.
    > Analyze the data: like total sum, average,etc..


    If you want to do all by yourself you can create a list of starting
    positions, and you can add None to it, so you can use it to slice a
    line with a loop.
    You can also create a list of types, so you can cast the string parts
    to their correct types, catching the exceptions. You can do it in a
    second loop for clarity, or the first one to speed up a bit.
    So you can just create a list of data for each input line, a dict may
    be unnecessary because the number of fields seems fixed.

    Bye,
    bearophile
     
    , Aug 1, 2006
    #4
  5. flit Guest

    Thanks for all input..
    I think I should put all data on a mysql base..
    The company will need this for reports and statistics for a year.

    wrote:
    > :
    >
    > > 1- Read the data and put all variables in a list
    > > 2- Read the data and put all the variables in dictionary?
    > > the logs is in this format
    > > xxxxxxxxxxxxxxxxxxxxxxxxxx
    > > The separation is by byte size like
    > > xxx three bytes for code x , xxxx bytes for hour, etc..
    > > I have two main objectives.
    > > Show all data to user.
    > > Analyze the data: like total sum, average,etc..

    >
    > If you want to do all by yourself you can create a list of starting
    > positions, and you can add None to it, so you can use it to slice a
    > line with a loop.
    > You can also create a list of types, so you can cast the string parts
    > to their correct types, catching the exceptions. You can do it in a
    > second loop for clarity, or the first one to speed up a bit.
    > So you can just create a list of data for each input line, a dict may
    > be unnecessary because the number of fields seems fixed.
    >
    > Bye,
    > bearophile
     
    flit, Aug 1, 2006
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Amratash
    Replies:
    0
    Views:
    559
    Amratash
    Apr 13, 2004
  2. Pablo Tola

    Best way to write log file

    Pablo Tola, May 27, 2004, in forum: ASP .Net
    Replies:
    2
    Views:
    1,738
    =?Utf-8?B?RGF2aWQgQ29lLCBNQ1A=?=
    May 27, 2004
  3. =?Utf-8?B?TmVtbw==?=
    Replies:
    0
    Views:
    368
    =?Utf-8?B?TmVtbw==?=
    Nov 30, 2005
  4. Jhair Tocancipa Triana

    EJB - analyze xml file

    Jhair Tocancipa Triana, Dec 23, 2006, in forum: Java
    Replies:
    4
    Views:
    433
    Alfred
    Dec 24, 2006
  5. David.Cristian1987

    analyze file system in c++

    David.Cristian1987, Nov 22, 2008, in forum: C++
    Replies:
    6
    Views:
    418
    David.Cristian1987
    Nov 23, 2008
Loading...

Share This Page