Caching a large file - 2nd post

[Chris Uppal]

And one other thing: having 100 separate 1 Gb files would *definitely* be
more manageable. I just don't think that 100 operations could possibly be
as fast as one large operation. But we will do the benchmarks.

Do that. The time taken will, amongst other things, be affected by the number
of levels of indirection in the on-disk datastructure necessary for the file
system to manage the blocks of data in the file. To me it seems quite feasible
that the extra level(s) of disk access needed to find block X in a 100Gb file
could make the operation (whatever it is ;-) slower.

Don't forget that this will be different according the file system in use.

If (as you say) disk seek time is going to be an issue, and if your application
is (or can be) multi-threaded, then you should get higher throughput by putting
the data on many physical disks.

If (as you hint) IO bandwidth is going to be an issue then it may be worth
considering compressing the data. In some cases this can be a biggish win.

-- chris