Fastest way to read array of ints off disk?

Discussion in 'Java' started by Chris, Jan 17, 2007.

  1. Chris

    Chris Guest

    I need to read several large arrays of integers off a disk into int []
    variables.

    I could read them into byte[] arrays and then convert, doing lots of
    masks and bitshifting. I can't help but believe there's a faster way.

    I've fooled with NIO, thinking that I could create an IntBuffer and read
    stuff into it using FileChannel.read(), but no go. I don't think it will
    work to create a ByteBuffer, read data into it, and then call
    asIntBuffer() on it, because the .array() method I need to return the
    int [] is described as an "optional operation" in the Javadoc.

    Any other ideas how to do it?
    Chris, Jan 17, 2007
    #1
    1. Advertising

  2. Chris wrote:
    > I need to read several large arrays of integers off a disk into int []
    > variables.
    >
    > I could read them into byte[] arrays and then convert, doing lots of
    > masks and bitshifting. I can't help but believe there's a faster way.
    >
    > I've fooled with NIO, thinking that I could create an IntBuffer and read
    > stuff into it using FileChannel.read(), but no go. I don't think it will
    > work to create a ByteBuffer, read data into it, and then call
    > asIntBuffer() on it, because the .array() method I need to return the
    > int [] is described as an "optional operation" in the Javadoc.
    >
    > Any other ideas how to do it?


    You could use DataInputStream.readInt(). It does the masking and
    shifting for you :). Use a big buffer with it.

    --

    Knute Johnson
    email s/nospam/knute/
    Knute Johnson, Jan 17, 2007
    #2
    1. Advertising

  3. Chris

    Tom Hawtin Guest

    Chris wrote:
    > I need to read several large arrays of integers off a disk into int []
    > variables.
    >
    > I could read them into byte[] arrays and then convert, doing lots of
    > masks and bitshifting. I can't help but believe there's a faster way.
    >
    > I've fooled with NIO, thinking that I could create an IntBuffer and read
    > stuff into it using FileChannel.read(), but no go. I don't think it will
    > work to create a ByteBuffer, read data into it, and then call
    > asIntBuffer() on it, because the .array() method I need to return the
    > int [] is described as an "optional operation" in the Javadoc.


    The array() method returns the array that backs the buffer. If you start
    with a 'default' ByteBuffer, the backing array will be of type byte[].

    The easiest thing to do is to go through the IntBuffer and copy the data
    into a new int[].

    I'm not entirely sure what performance results you will get. Using
    allocateDirect instead of allocate should improve the time to read the
    buffer, but not sure about reading the results out of it.

    In order for NIO to be efficient it requires multiple levels of calls to
    be inlined. Therefore, make sure the method with your inner loop is as
    small as absolutely possible and use the Server VM.

    You might find it faster to manually copy data from a byte[] to an int[].

    Tom Hawtin
    Tom Hawtin, Jan 17, 2007
    #3
  4. Chris

    Chris Uppal Guest

    Chris wrote:

    > I could read them into byte[] arrays and then convert, doing lots of
    > masks and bitshifting. I can't help but believe there's a faster way.


    I doubt if there's a way that's /much/ faster. On my machine a simple
    hand-coded byte[]-int[] conversion loops is more than an order of magnitude
    faster than reading the same amount of the data off-disk, so I wouldn't expect
    that you can improve the total time by more than a very small
    margin. Of course your hardware/software combo may be different, but I'd be
    surprised if it were /that/ different...

    -- chris
    Chris Uppal, Jan 17, 2007
    #4
  5. Chris

    EJP Guest

    Chris wrote:
    > Any other ideas how to do it?


    Use a java.nio.MappedByteBuffer and get an IntBuffer view of it, then
    set the appropriate ByteOrder on it. This is about as fast as you can get.
    EJP, Jan 21, 2007
    #5
  6. On Tue, 16 Jan 2007 19:41:41 -0800, Chris <> wrote:

    > I need to read several large arrays of integers off a disk into int []
    > variables.
    >
    > I could read them into byte[] arrays and then convert, doing lots of
    > masks and bitshifting. I can't help but believe there's a faster way.
    >
    > I've fooled with NIO, thinking that I could create an IntBuffer and read
    > stuff into it using FileChannel.read(), but no go. I don't think it will
    > work to create a ByteBuffer, read data into it, and then call
    > asIntBuffer() on it, because the .array() method I need to return the
    > int [] is described as an "optional operation" in the Javadoc.
    >
    > Any other ideas how to do it?


    Have you looked at java.io.RandomAccessFile.readFully(byte[])?

    --
    Randolf Richardson -
    The Lumber Cartel, local 42 (Canadian branch)
    http://www.lumbercartel.ca/
    Randolf Richardson, Jan 28, 2007
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Skybuck Flying

    ints ints ints and ints

    Skybuck Flying, Jul 8, 2004, in forum: C Programming
    Replies:
    24
    Views:
    831
    Jack Klein
    Jul 10, 2004
  2. Laszlo Nagy
    Replies:
    2
    Views:
    268
    Matthew Woodcraft
    Aug 9, 2008
  3. M.-A. Lemburg
    Replies:
    3
    Views:
    279
    castironpi
    Aug 10, 2008
  4. Laszlo Nagy
    Replies:
    0
    Views:
    287
    Laszlo Nagy
    Aug 8, 2008
  5. castironpi
    Replies:
    5
    Views:
    343
    castironpi
    Aug 24, 2008
Loading...

Share This Page