design experiment?

[Peter]

Hi
I am going to design a flat data format, used to store the log
informations, the informations are : 1)Date Time 2) Critical Level 3)
log message.
There will be at least 100000+ log message, and i want my java
application to search it as fast as possible, so i think XML is not a
good idea(slow). But what design issus i should know to accomplish
this mission? My basic idea is keep each string in the file as short
as possible.


thanks
from Peter ([email protected])

 

[bugbear]

Hi
I am going to design a flat data format, used to store the log
informations, the informations are : 1)Date Time 2) Critical Level 3)
log message.
There will be at least 100000+ log message, and i want my java
application to search it as fast as possible, so i think XML is not a
good idea(slow). But what design issus i should know to accomplish
this mission? My basic idea is keep each string in the file as short
as possible.

Either use a database (which can use indexes on your behalf)
or use "indexes".

Simply searching *will* be slow.

BugBear

 

[Dotty]

Either use a database (which can use indexes on your behalf)
or use "indexes".

Simply searching *will* be slow.

If the date/times are monotonic, you can convert them to an integer or long,
put the data in an array, and use binary search.

 

[bugbear]

If the date/times are monotonic, you can convert them to an integer or long,
put the data in an array, and use binary search.

Only if you're searching using data as your access key.
In any case, Peter's log sounds too large to be RAM
resident.

BugBear

 

[Chris Uppal]

Either use a database (which can use indexes on your behalf)
or use "indexes".

Simply searching *will* be slow.

Depends on what you are searching for. If the OP wants to do free-text
searches in the logfile, for instance, then a plain text representation will
probably be best. (The scanning time is going to be dominated by the
disk-access time; that can be avoided by building a free-text index with
something like Lucene, but that would take a lot of extra space, and doesn't
comply with the requirement that the logs be reasonably compact -- not to
mention that building/updating the indexes will take time).

-- chris