P
Philip Rhoades
People,
I have script that does:
- statistical processing from data in 50x32x20 (32,000) large input files
- writes a small text file (22 lines with one or more columns of numbers)
for each input file
- read all small files back in again for final processing.
Profiling shows that IO is taking up more than 60% of the time - short of
making fewer, larger files for the data (which is inconvenient for random
viewing/ processing of individual results) are there other alternatives to
using the "File" and "IO" classes that would be faster?
Thanks,
Phil.
I have script that does:
- statistical processing from data in 50x32x20 (32,000) large input files
- writes a small text file (22 lines with one or more columns of numbers)
for each input file
- read all small files back in again for final processing.
Profiling shows that IO is taking up more than 60% of the time - short of
making fewer, larger files for the data (which is inconvenient for random
viewing/ processing of individual results) are there other alternatives to
using the "File" and "IO" classes that would be faster?
Thanks,
Phil.