Processing Multiple Large Files

[News123]

Hi cartercc,

If you are using an Intel-like processor, it multi processes, anyway.
There are only two ways to increase speed: increase the clocks of the
processor or increase the number of processors. . .

I don't fuly agree with your statement.

You should have added:
- increase the clock speed of the data bus
- increase the speed of a peripheral ( faster disk )
- distribute data over several peripherals (disks)
- optimize the algorithm
.. . .

Each problem has it's own bottle neck.
Just throwing multiple CPUs at a problem
or just increasing the processor speed without increasing the IO
might not be worth the effort.

As long as Kevind doesn't get a little more specific about
'analyzing some network' log' not much can be said.



In order to find out what is limiting some profiling is useful:

Interesting profiling data would be:
- time needed if the log files were only read but not analyzed
- this could be measured on Kevin's system with one thread
readign all 300 files or with multiple threads trying to
read the files in parallel
- time needed to do the 'analyzing'.



My bet (but only profiling can really tell) is on
optimizing the analysis algorithm:
- be sure to use the correct data types and data structures
- avoid re-calculating data for every line, which could be
calculated before
- try to cache operations, which might have been performed on prevous
lines (DNS lookup would be a good example)
.. . .


bye


N