Best way to read, and analyze a log file?

[superflit]

Hi All,

I am reading a log file, and wondering what is the best way to read and
analize this.
I am think in two options:

1- Read the data and put all variables in a list
2- Read the data and put all the variables in dictionary?

the logs is in this format

xxxxxxxxxxxxxxxxxxxxxxxxxx
The separation is by byte size like

xxx three bytes for code x , xxxx bytes for hour, etc..
I have two main objectives.
Show all data to user.
Analyze the data: like total sum, average,etc..

All advice will be great appreciated.
And sorry about my bad english.

 

[Gregor Horvath]

Hi,

I am reading a log file, and wondering what is the best way to read and
analize this.

Look at:

http://pyparsing.wikispaces.com/

There is also an example for parsing an apache log.

 

[flit]

Not exactly Wha I am looking, this seems to be a good project, but the
way to parse the file I already have done that.
I am looking in "desing pattern" advices, how to deal with the data.
Must I use dbm? Must I make all the statistics and save in a dmf file?
Any insights?

 

[bearophileHUGS]

(e-mail address removed):

1- Read the data and put all variables in a list
2- Read the data and put all the variables in dictionary?
the logs is in this format
xxxxxxxxxxxxxxxxxxxxxxxxxx
The separation is by byte size like
xxx three bytes for code x , xxxx bytes for hour, etc..
I have two main objectives.
Show all data to user.
Analyze the data: like total sum, average,etc..

If you want to do all by yourself you can create a list of starting
positions, and you can add None to it, so you can use it to slice a
line with a loop.
You can also create a list of types, so you can cast the string parts
to their correct types, catching the exceptions. You can do it in a
second loop for clarity, or the first one to speed up a bit.
So you can just create a list of data for each input line, a dict may
be unnecessary because the number of fields seems fixed.

Bye,
bearophile

 

[flit]

Thanks for all input..
I think I should put all data on a mysql base..
The company will need this for reports and statistics for a year.