problems reading binary file

Discussion in 'Java' started by Mikee, Jun 9, 2010.

  1. Mikee

    Mikee Guest

    Hi

    I having trouble reading a binary file. The problem seems to occur
    with reading some float values so I've made a small test file (from
    SQL server using native file output) that contains just 5 floats.

    -13.73482 , 20.689 , -99.99999 , 20.157 ,1 9.454

    I read the file in and write out the values using

    DataInputStream in = new DataInputStream(new
    BufferedInputStream(new FileInputStream(filename)));
    System.out.print(ByteSwapper.swap(in.readFloat())+" ");
    System.out.print(ByteSwapper.swap(in.readFloat())+" ");
    System.out.print(ByteSwapper.swap(in.readFloat())+" ");
    System.out.print(ByteSwapper.swap(in.readFloat())+" ");
    System.out.println(ByteSwapper.swap(in.readFloat())+" ");

    The output is -13.734822 20.689 6.9055E-41 20.157 19.454

    ie -99.99999 is now 6.9055E-41.

    I'm trying to read in a much larger SQL outgest of mixed data types
    integers, longs, doubles, floats and bytes and the only problem I've
    spotted is floats of -99.9999 (which is a deafult value used for some
    columns)

    It was necessary to byte swap and ByteSwapper.swap(float) is something
    I find on the web an dis below.
    I tries other byte swap code with the same result.

    Any ideas what I'm doing wrong with the 3rd value and presumably in
    general.

    Python code of

    binaryFile = file('test.bin', 'rb')
    while True:
    rowBinary = binaryFile.read(20)
    if not rowBinary:
    break
    print struct.unpack('<5f', rowBinary)
    binaryFile.close()

    correctly gives -13.734822273254395, 20.688999176025391,
    -99.999992370605469, 20.156999588012695, 19.454000473022461

    java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_12-b04

    Thanks
    Mike

    public static float swap (float value)
    {
    int intValue = Float.floatToIntBits (value);
    intValue = swap (intValue);
    return Float.intBitsToFloat (intValue);
    }

    and

    public static int swap (int value)
    {
    int b1 = (value >> 0) & 0xff;
    int b2 = (value >> 8) & 0xff;
    int b3 = (value >> 16) & 0xff;
    int b4 = (value >> 24) & 0xff;

    return b1 << 24 | b2 << 16 | b3 << 8 | b4 << 0;
    }
     
    Mikee, Jun 9, 2010
    #1
    1. Advertising

  2. Mikee

    Paul Cager Guest

    On Jun 9, 11:14 am, Thomas Pornin <> wrote:
    > According to Mikee  <>:
    >
    > >            DataInputStream in = new DataInputStream(new
    > > BufferedInputStream(new FileInputStream(filename)));
    > >            System.out.print(ByteSwapper.swap(in.readFloat())+" ");

    [...]
    > You _might_ correct your code by using Float.float.toRawIntBits()
    > instead of Float.floatToIntBits(), but even if it works, it is
    > non-robust (it assumes that DataInputStream also does things that way).
    > A more correct way of doing things is to read the bytes as an "int"
    > directly, which basically removes steps 2 and 3 altogether. This would
    > look like this:
    >
    >     System.out.print(Float.intBitsToFloat(swap(in.readInt())) + " ");


    I haven't checked the API (http://java.sun.com/javase/6/docs/api/
    index.html?java/nio/package-summary.html) but I seem to remember that
    NIO allows you to read floats etc in the byte order of your choice.
    That might avoid the need to do all the bit twiddling yourself.
     
    Paul Cager, Jun 9, 2010
    #2
    1. Advertising

  3. Mikee

    Mikee Guest

    Hi Thomas

    Thanks for poitning out where I was going wrong.

    >     System.out.print(Float.intBitsToFloat(swap(in.readInt())) + " ");


    Yes that worked,

    Thanks again
    Mike
     
    Mikee, Jun 9, 2010
    #3
  4. Mikee

    Mikee Guest

    Thanks Paul

    Currently part of the environment I'm working in is constrained to 1.5
    but
    I'lll check 1.6 out for future ref.

    > I haven't checked the API (http://java.sun.com/javase/6/docs/api/
    > index.html?java/nio/package-summary.html) but I seem to remember that
    > NIO allows you to read floats etc in the byte order of your choice.
    > That might avoid the need to do all the bit twiddling yourself.


    Cheers
    Mike
     
    Mikee, Jun 9, 2010
    #4
  5. Mikee

    Lew Guest

    Please do not top-post.
    Please do cite the author of posts you quote.


    Paul Cager wrote:
    >> I haven't checked the API (http://java.sun.com/javase/6/docs/api/
    >> index.html?java/nio/package-summary.html) but I seem to remember that
    >> NIO allows you to read floats etc in the byte order of your choice.
    >> That might avoid the need to do all the bit twiddling yourself.

    >


    Mikee
    > Currently part of the environment I'm working in is constrained to 1.5
    > but I'lll check 1.6 out for future ref.
    >


    Why is that relevant?

    Paul suggested NIO, which was introduced in Java 1.4.

    You really should read the Javadocs.
    <http://java.sun.com/j2se/1.5.0/docs/api/java/nio/package-
    summary.html>

    --
    Lew
     
    Lew, Jun 9, 2010
    #5
  6. Mikee

    Nigel Wade Guest

    On Wed, 09 Jun 2010 10:14:29 +0000, Thomas Pornin wrote:

    >
    > You _might_ correct your code by using Float.float.toRawIntBits()
    > instead of Float.floatToIntBits(), but even if it works, it is
    > non-robust (it assumes that DataInputStream also does things that way).
    > A more correct way of doing things is to read the bytes as an "int"
    > directly, which basically removes steps 2 and 3 altogether. This would
    > look like this:
    >
    > System.out.print(Float.intBitsToFloat(swap(in.readInt())) + " ");
    >
    >


    I prefer to load the data into a byte array, then use a ByteBuffer to
    access the data directly from memory in the "correct" format. Everything
    necessary is already available in the standard API classes, no manual bit
    shifting, byte-swapping etc. is required.

    The basic scenario is:

    create a ByteBuffer
    set it to ByteBuffer.LITTLE_ENDIAN

    loop:

    read block of data into byte[]
    wrap ByteBuffer around byte[]
    read data from ByteBuffer

    E.g. this is an extract of some code which is run routinely here, and
    reads little-endian data files:

    DataInputStream dataIn;
    short fitRecordLength;
    short inxRecordLength;
    ....

    // read the fit and inx lengths (this is little endian)
    byte[] lengthBytes = new byte[4];
    dataIn.readFully( lengthBytes );

    // wrap the byte array in a ByteBuffer, this allows
    // reading of little endian data
    ByteBuffer bb = ByteBuffer.wrap( lengthBytes );
    bb.order( ByteOrder.LITTLE_ENDIAN );

    fitRecordLength = bb.getShort();
    inxRecordLength = bb.getShort();
    ....

    ByteBuffer dataBuffer;
    byte[] bufferArray;
    bufferArray = new byte[fitRecordLength];
    dataBuffer = ByteBuffer.wrap( bufferArray );
    dataBuffer.order( ByteOrder.LITTLE_ENDIAN );

    dataIn.readFully( bufferArray );
    dataBuffer.position( 0 );

    int recordNumber = dataBuffer.getInt();


    --
    Nigel Wade
     
    Nigel Wade, Jun 9, 2010
    #6
  7. Mikee

    Roedy Green Guest

    On Wed, 9 Jun 2010 00:17:53 -0700 (PDT), Mikee
    <> wrote, quoted or indirectly quoted someone
    who said :

    >
    >-13.73482 , 20.689 , -99.99999 , 20.157 ,1 9.454


    those are not floats. Those are characters.

    Floats are binary. You can't create them with a text editor.

    You can WRITE the file with floats. See
    http://mindprod.com/applet/fileio.html

    Then you can read them with DataInputStream.

    Your data look like a CSV file. See
    http://mindprod.com/jgloss/csv.html
    for how to read them.

    --
    Roedy Green Canadian Mind Products
    http://mindprod.com

    Have you ever noticed that any computer search in the movies, is always linear, with, for example, candidate fingerprints flashing up on the screen one after another? The public is still under the delusion that electronic files are microscopic filing cabinets made out of tiny wires or magnetic patches inside the computer. Most lay people are surprised that it is easy for a computer to file things simultaneously by a dozen different schemes, and that they can have any report printed in any number of different sorted orders. With physical files, they are limited to one ordering/access.
     
    Roedy Green, Jun 9, 2010
    #7
  8. Mikee

    Mikee Guest

    On Jun 9, 2:54 pm, Nigel Wade <> wrote:

    > I prefer to load the data into a byte array, then use a ByteBuffer to
    > access the data directly from memory in the "correct" format. Everything
    > necessary is already available in the standard API classes, no manual bit
    > shifting, byte-swapping etc. is required.
    >


    Hi Nige

    Thanks that method works well.

    Mike
     
    Mikee, Jun 10, 2010
    #8
  9. Mikee

    Mikee Guest

    On Jun 9, 2:34 pm, Lew <> wrote:

    > > Currently part of the environment I'm working in is constrained to 1.5
    > > but I'lll check 1.6 out for future ref.

    >
    > Why is that relevant?


    Saw the 1.6 in the URL and rushed to a wrong conclusion.

    Mike
     
    Mikee, Jun 10, 2010
    #9
  10. Mikee

    Mikee Guest

    On Jun 9, 9:24 pm, Roedy Green <>
    wrote:

    Hi Roedy

    Thanks for looking.

    >
    > >-13.73482 , 20.689 , -99.99999 , 20.157  ,1 9.454

    >
    > those are not floats. Those are characters.
    >


    They were floats, this was just cut n pasted from the SQL server GUI
    output (albeit with an erroneouse space in the 19.454 by my fat finger
    editing) as I was just trying to show what the values should be.

    Anyhow problem solved by the two methods suggested above.

    Cheers
    Mike
     
    Mikee, Jun 10, 2010
    #10
  11. Mikee

    Lew Guest

    Mikee wrote:
    >>> Currently part of the environment I'm working in is constrained to 1.5
    >>> but I'lll check 1.6 out for future ref.


    Lew<> wrote:
    >> Why is that relevant?


    Mikee wrote:
    > Saw the 1.6 in the URL and rushed to a wrong conclusion.


    When did you start programming in Java? NIO has been around since 2002. You
    shouldn't need to conclude anything about it from the URL.

    --
    Lew
     
    Lew, Jun 10, 2010
    #11
  12. On 10-06-2010 07:32, Lew wrote:
    > Mikee wrote:
    >>>> Currently part of the environment I'm working in is constrained to 1.5
    >>>> but I'lll check 1.6 out for future ref.

    >
    > Lew<> wrote:
    >>> Why is that relevant?

    >
    > Mikee wrote:
    >> Saw the 1.6 in the URL and rushed to a wrong conclusion.

    >
    > When did you start programming in Java? NIO has been around since 2002.
    > You shouldn't need to conclude anything about it from the URL.


    Not everyone programming in Java reads about all the new stuff
    in new Java versions.

    Arne
     
    Arne Vajhøj, Jun 10, 2010
    #12
  13. Mikee

    Lew Guest

    Lew wrote:
    >> When did you start programming in Java? NIO has been around since 2002.
    >> You shouldn't need to conclude anything about it from the URL.

    >


    Arne Vajhøj wrote:
    > Not everyone programming in Java reads about all the new stuff
    > in new Java versions.
    >


    True, but NIO is hardly new, and 1.4 is very, very far from being a
    new Java version.

    --
    Lew
     
    Lew, Jun 11, 2010
    #13
  14. Mikee

    Arne Vajhøj Guest

    On 11-06-2010 13:20, Lew wrote:
    > Lew wrote:
    >>> When did you start programming in Java? NIO has been around since 2002.
    >>> You shouldn't need to conclude anything about it from the URL.

    >
    > Arne Vajhøj wrote:
    >> Not everyone programming in Java reads about all the new stuff
    >> in new Java versions.

    >
    > True, but NIO is hardly new, and 1.4 is very, very far from being a
    > new Java version.


    Yes. But if someone does not read it when the new Java version
    come out, then the chance of reading it later is rather small
    unless somehow being pointed to it.

    Arne
     
    Arne Vajhøj, Jun 12, 2010
    #14
  15. Mikee

    Lew Guest

    Lew wrote:
    >>>> When did you start programming in Java? NIO has been around since 2002.
    >>>> You shouldn't need to conclude anything about it from the URL.


    Arne Vajhøj wrote:
    >>> Not everyone programming in Java reads about all the new stuff
    >>> in new Java versions.


    Lew wrote:
    >> True, but NIO is hardly new, and 1.4 is very, very far from being a
    >> new Java version.


    Arne Vajhøj wrote:
    > Yes. But if someone does not read it when the new Java version
    > come out, then the chance of reading it later is rather small
    > unless somehow being pointed to it.


    Shame on them, then.

    One should be generally aware of the major packages available for the version
    of Java one is using. The older and more major the package, the less excuse
    for not being aware at least of its existence and when it came in. Had the OP
    even looked minimally at the first link to NIO he got, after it was "somehow
    being pointed to", he'd've seen that it came in in Java 1.4. Your argument is
    moot because they *were* directed to look at it!

    NIO is old enough and major enough that everyone who does any I/O at all in
    Java should be aware that it exists, even if they don't use it or don't have
    much expertise with it. I have negligible expertise with NIO, but I read the
    Sun docs on the different versions and at least knew it existed since it came out.

    For those who want to know what versions of Java support what features,
    there's a good summary at
    <http://en.wikipedia.org/wiki/Java_version_history>
    so they don't need to make excuses like "no one somehow pointed me to it" as a
    response to when someone pointed them to it, or depend on apologists for
    ignorance as you're playing here.

    --
    Lew
     
    Lew, Jun 12, 2010
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Neo
    Replies:
    0
    Views:
    480
  2. Dimitri Papoutsis

    Problems with reading binary data files

    Dimitri Papoutsis, Mar 10, 2005, in forum: C++
    Replies:
    4
    Views:
    398
    Old Wolf
    Mar 11, 2005
  3. Ron Eggler

    writing binary file (ios::binary)

    Ron Eggler, Apr 25, 2008, in forum: C++
    Replies:
    9
    Views:
    962
    James Kanze
    Apr 28, 2008
  4. Guest
    Replies:
    6
    Views:
    1,754
    Guest
    Apr 25, 2010
  5. Jim
    Replies:
    6
    Views:
    755
Loading...

Share This Page