Fastest way to read array of ints off disk?

[Chris]

I need to read several large arrays of integers off a disk into int []
variables.

I could read them into byte[] arrays and then convert, doing lots of
masks and bitshifting. I can't help but believe there's a faster way.

I've fooled with NIO, thinking that I could create an IntBuffer and read
stuff into it using FileChannel.read(), but no go. I don't think it will
work to create a ByteBuffer, read data into it, and then call
asIntBuffer() on it, because the .array() method I need to return the
int [] is described as an "optional operation" in the Javadoc.

Any other ideas how to do it?

 

[Knute Johnson]

I need to read several large arrays of integers off a disk into int []
variables.

I could read them into byte[] arrays and then convert, doing lots of
masks and bitshifting. I can't help but believe there's a faster way.

I've fooled with NIO, thinking that I could create an IntBuffer and read
stuff into it using FileChannel.read(), but no go. I don't think it will
work to create a ByteBuffer, read data into it, and then call
asIntBuffer() on it, because the .array() method I need to return the
int [] is described as an "optional operation" in the Javadoc.

Any other ideas how to do it?

You could use DataInputStream.readInt(). It does the masking and
shifting for you . Use a big buffer with it.

 

[Tom Hawtin]

I need to read several large arrays of integers off a disk into int []
variables.

I could read them into byte[] arrays and then convert, doing lots of
masks and bitshifting. I can't help but believe there's a faster way.

I've fooled with NIO, thinking that I could create an IntBuffer and read
stuff into it using FileChannel.read(), but no go. I don't think it will
work to create a ByteBuffer, read data into it, and then call
asIntBuffer() on it, because the .array() method I need to return the
int [] is described as an "optional operation" in the Javadoc.

The array() method returns the array that backs the buffer. If you start
with a 'default' ByteBuffer, the backing array will be of type byte[].

The easiest thing to do is to go through the IntBuffer and copy the data
into a new int[].

I'm not entirely sure what performance results you will get. Using
allocateDirect instead of allocate should improve the time to read the
buffer, but not sure about reading the results out of it.

In order for NIO to be efficient it requires multiple levels of calls to
be inlined. Therefore, make sure the method with your inner loop is as
small as absolutely possible and use the Server VM.

You might find it faster to manually copy data from a byte[] to an int[].

Tom Hawtin

 

[Chris Uppal]

I could read them into byte[] arrays and then convert, doing lots of
masks and bitshifting. I can't help but believe there's a faster way.

I doubt if there's a way that's /much/ faster. On my machine a simple
hand-coded byte[]-int[] conversion loops is more than an order of magnitude
faster than reading the same amount of the data off-disk, so I wouldn't expect
that you can improve the total time by more than a very small
margin. Of course your hardware/software combo may be different, but I'd be
surprised if it were /that/ different...

-- chris

 

[EJP]

Any other ideas how to do it?

Use a java.nio.MappedByteBuffer and get an IntBuffer view of it, then
set the appropriate ByteOrder on it. This is about as fast as you can get.

 

[Randolf Richardson]

I need to read several large arrays of integers off a disk into int []
variables.

I could read them into byte[] arrays and then convert, doing lots of
masks and bitshifting. I can't help but believe there's a faster way.

I've fooled with NIO, thinking that I could create an IntBuffer and read
stuff into it using FileChannel.read(), but no go. I don't think it will
work to create a ByteBuffer, read data into it, and then call
asIntBuffer() on it, because the .array() method I need to return the
int [] is described as an "optional operation" in the Javadoc.

Any other ideas how to do it?

Have you looked at java.io.RandomAccessFile.readFully(byte[])?