Efficient format for huge amount of data

[Chorg]

if it is a file format you are looking for , i would go with
Berkeley DB

www.sleepycat.com

quick, in process, allows btree, hash and queue access, has java api
and lives in process, so it is as quick as it gets ...

 

[Alan Chen]

Maybe a binary format is more efficient, but I don't know which could
be the best way, nor how to implement it.
I've considered using Serialized, but since the source program is not
written in Java it may be hard to replicate exactly the Serialized
format - btw, where is it documented? if documented at all...

Any ideas are welcome.

You might look at some scientific data formats. NetCDF and HDF both
have java interfaces and are written with large datasets in mind.

netCDF - http://www.unidata.ucar.edu/packages/netcdf/
HDF - http://hdf.ncsa.uiuc.edu/

Good Luck,
- alan