problems reading binary file

[Mikee]

Hi

I having trouble reading a binary file. The problem seems to occur
with reading some float values so I've made a small test file (from
SQL server using native file output) that contains just 5 floats.

-13.73482 , 20.689 , -99.99999 , 20.157 ,1 9.454

I read the file in and write out the values using

DataInputStream in = new DataInputStream(new
BufferedInputStream(new FileInputStream(filename)));
System.out.print(ByteSwapper.swap(in.readFloat())+" ");
System.out.print(ByteSwapper.swap(in.readFloat())+" ");
System.out.print(ByteSwapper.swap(in.readFloat())+" ");
System.out.print(ByteSwapper.swap(in.readFloat())+" ");
System.out.println(ByteSwapper.swap(in.readFloat())+" ");

The output is -13.734822 20.689 6.9055E-41 20.157 19.454

ie -99.99999 is now 6.9055E-41.

I'm trying to read in a much larger SQL outgest of mixed data types
integers, longs, doubles, floats and bytes and the only problem I've
spotted is floats of -99.9999 (which is a deafult value used for some
columns)

It was necessary to byte swap and ByteSwapper.swap(float) is something
I find on the web an dis below.
I tries other byte swap code with the same result.

Any ideas what I'm doing wrong with the 3rd value and presumably in
general.

Python code of

binaryFile = file('test.bin', 'rb')
while True:
rowBinary = binaryFile.read(20)
if not rowBinary:
break
print struct.unpack('<5f', rowBinary)
binaryFile.close()

correctly gives -13.734822273254395, 20.688999176025391,
-99.999992370605469, 20.156999588012695, 19.454000473022461

java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_12-b04

Thanks
Mike

public static float swap (float value)
{
int intValue = Float.floatToIntBits (value);
intValue = swap (intValue);
return Float.intBitsToFloat (intValue);
}

and

public static int swap (int value)
{
int b1 = (value >> 0) & 0xff;
int b2 = (value >> 8) & 0xff;
int b3 = (value >> 16) & 0xff;
int b4 = (value >> 24) & 0xff;

return b1 << 24 | b2 << 16 | b3 << 8 | b4 << 0;
}

 

[Paul Cager]

           DataInputStream in = new DataInputStream(new
BufferedInputStream(new FileInputStream(filename)));
           System.out.print(ByteSwapper.swap(in.readFloat())+" ");

[...]
You _might_ correct your code by using Float.float.toRawIntBits()
instead of Float.floatToIntBits(), but even if it works, it is
non-robust (it assumes that DataInputStream also does things that way).
A more correct way of doing things is to read the bytes as an "int"
directly, which basically removes steps 2 and 3 altogether. This would
look like this:

    System.out.print(Float.intBitsToFloat(swap(in.readInt())) + " ");

I haven't checked the API (http://java.sun.com/javase/6/docs/api/
index.html?java/nio/package-summary.html) but I seem to remember that
NIO allows you to read floats etc in the byte order of your choice.
That might avoid the need to do all the bit twiddling yourself.

 

[Mikee]

Hi Thomas

Thanks for poitning out where I was going wrong.

    System.out.print(Float.intBitsToFloat(swap(in.readInt())) + " ");

Yes that worked,

Thanks again
Mike

 

[Mikee]

Thanks Paul

Currently part of the environment I'm working in is constrained to 1.5
but
I'lll check 1.6 out for future ref.

I haven't checked the API (http://java.sun.com/javase/6/docs/api/
index.html?java/nio/package-summary.html) but I seem to remember that
NIO allows you to read floats etc in the byte order of your choice.
That might avoid the need to do all the bit twiddling yourself.

Cheers
Mike

 

[Lew]

Please do not top-post.
Please do cite the author of posts you quote.

Mikee
Currently part of the environment I'm working in is constrained to 1.5
but I'lll check 1.6 out for future ref.

Why is that relevant?

Paul suggested NIO, which was introduced in Java 1.4.

You really should read the Javadocs.
<http://java.sun.com/j2se/1.5.0/docs/api/java/nio/package-
summary.html>

 

[Nigel Wade]

You _might_ correct your code by using Float.float.toRawIntBits()
instead of Float.floatToIntBits(), but even if it works, it is
non-robust (it assumes that DataInputStream also does things that way).
A more correct way of doing things is to read the bytes as an "int"
directly, which basically removes steps 2 and 3 altogether. This would
look like this:

System.out.print(Float.intBitsToFloat(swap(in.readInt())) + " ");

I prefer to load the data into a byte array, then use a ByteBuffer to
access the data directly from memory in the "correct" format. Everything
necessary is already available in the standard API classes, no manual bit
shifting, byte-swapping etc. is required.

The basic scenario is:

create a ByteBuffer
set it to ByteBuffer.LITTLE_ENDIAN

loop:

read block of data into byte[]
wrap ByteBuffer around byte[]
read data from ByteBuffer

E.g. this is an extract of some code which is run routinely here, and
reads little-endian data files:

DataInputStream dataIn;
short fitRecordLength;
short inxRecordLength;
....

// read the fit and inx lengths (this is little endian)
byte[] lengthBytes = new byte[4];
dataIn.readFully( lengthBytes );

// wrap the byte array in a ByteBuffer, this allows
// reading of little endian data
ByteBuffer bb = ByteBuffer.wrap( lengthBytes );
bb.order( ByteOrder.LITTLE_ENDIAN );

fitRecordLength = bb.getShort();
inxRecordLength = bb.getShort();
....

ByteBuffer dataBuffer;
byte[] bufferArray;
bufferArray = new byte[fitRecordLength];
dataBuffer = ByteBuffer.wrap( bufferArray );
dataBuffer.order( ByteOrder.LITTLE_ENDIAN );

dataIn.readFully( bufferArray );
dataBuffer.position( 0 );

int recordNumber = dataBuffer.getInt();

 

[Roedy Green]

-13.73482 , 20.689 , -99.99999 , 20.157 ,1 9.454

those are not floats. Those are characters.

Floats are binary. You can't create them with a text editor.

You can WRITE the file with floats. See
http://mindprod.com/applet/fileio.html

Then you can read them with DataInputStream.

Your data look like a CSV file. See
http://mindprod.com/jgloss/csv.html
for how to read them.

 

[Mikee]

I prefer to load the data into a byte array, then use a ByteBuffer to
access the data directly from memory in the "correct" format. Everything
necessary is already available in the standard API classes, no manual bit
shifting, byte-swapping etc. is required.

Hi Nige

Thanks that method works well.

Mike

 

[Mikee]

Why is that relevant?

Saw the 1.6 in the URL and rushed to a wrong conclusion.

Mike

 

[Mikee]

On Jun 9, 9:24 pm, Roedy Green <[email protected]>
wrote:

Hi Roedy

Thanks for looking.

those are not floats. Those are characters.

They were floats, this was just cut n pasted from the SQL server GUI
output (albeit with an erroneouse space in the 19.454 by my fat finger
editing) as I was just trying to show what the values should be.

Anyhow problem solved by the two methods suggested above.

Cheers
Mike

 

[Lew]

Saw the 1.6 in the URL and rushed to a wrong conclusion.

When did you start programming in Java? NIO has been around since 2002. You
shouldn't need to conclude anything about it from the URL.

 

[Arne Vajhøj]

When did you start programming in Java? NIO has been around since 2002.
You shouldn't need to conclude anything about it from the URL.

Not everyone programming in Java reads about all the new stuff
in new Java versions.

Arne

 

[Lew]

Not everyone programming in Java reads about all the new stuff
in new Java versions.

True, but NIO is hardly new, and 1.4 is very, very far from being a
new Java version.

 

[Arne Vajhøj]

True, but NIO is hardly new, and 1.4 is very, very far from being a
new Java version.

Yes. But if someone does not read it when the new Java version
come out, then the chance of reading it later is rather small
unless somehow being pointed to it.

Arne

 

[Lew]

Yes. But if someone does not read it when the new Java version
come out, then the chance of reading it later is rather small
unless somehow being pointed to it.

Shame on them, then.

One should be generally aware of the major packages available for the version
of Java one is using. The older and more major the package, the less excuse
for not being aware at least of its existence and when it came in. Had the OP
even looked minimally at the first link to NIO he got, after it was "somehow
being pointed to", he'd've seen that it came in in Java 1.4. Your argument is
moot because they *were* directed to look at it!

NIO is old enough and major enough that everyone who does any I/O at all in
Java should be aware that it exists, even if they don't use it or don't have
much expertise with it. I have negligible expertise with NIO, but I read the
Sun docs on the different versions and at least knew it existed since it came out.

For those who want to know what versions of Java support what features,
there's a good summary at
<http://en.wikipedia.org/wiki/Java_version_history>
so they don't need to make excuses like "no one somehow pointed me to it" as a
response to when someone pointed them to it, or depend on apologists for
ignorance as you're playing here.