Writing Binaries to and Reading Binaries from Disk

[KevinSimonson]

I'm planning on writing a program that stores a lot of binary informa-
tion, and I'd like to be able to write it to disk when I'm done with
it, so that later I can read it in from disk again. How do you do
that in Java, write binary <short>s to disk and then later read in
those <short>s from disk again?

Kevin Simonson

"You'll never get to heaven, or even to LA,
if you don't believe there's a way."
from _Why Not_

 

[Knute Johnson]

I'm planning on writing a program that stores a lot of binary informa-
tion, and I'd like to be able to write it to disk when I'm done with
it, so that later I can read it in from disk again. How do you do
that in Java, write binary <short>s to disk and then later read in
those <short>s from disk again?

Kevin Simonson

"You'll never get to heaven, or even to LA,
if you don't believe there's a way."
from _Why Not_

I'm not sure what a binary <short> is but to write a Java short to a
stream you can use the DataOutputStream class. Also you can convert it
to bytes and use a regular OutputStream. Also see BufferedOutputStream.

import java.io.*;

public class test4 {
public static void main(String[] args) throws IOException {
FileOutputStream fos = new FileOutputStream("data.dat");
DataOutputStream dos = new DataOutputStream(fos);
short s1 = -32768;
dos.writeShort(s1);
dos.close();

FileInputStream fis = new FileInputStream("data.dat");
DataInputStream dis = new DataInputStream(fis);
short s2 = dis.readShort();
dis.close();

System.out.print(s2);
}
}

 

[Lothar Kimmeringer]

I'm planning on writing a program that stores a lot of binary informa-
tion, and I'd like to be able to write it to disk when I'm done with
it, so that later I can read it in from disk again. How do you do
that in Java, write binary <short>s to disk and then later read in
those <short>s from disk again?

IO-operations are byte-oriented (this is not java specific)
so when writing shorts (2 bytes) there are two ways to do
this and it depends if the data you write to disk should
be read by other programs as well. So before you use
DataOutputStream you should check if the format to be used
is LSB least significant byte first or HSB (high significant
byte first). HSB is used with DataOutputStream, if you
want to write LSB you need to do the writing for yourself:

outstream.write(myshort & 0xff);
outstream.write((myshort >> 8) & 0xff);

other direction:

short myshort = instream.read() & 0xff;
myshort |= (instream.read() << 8) & 0xff;


Regards, Lothar
--
Lothar Kimmeringer E-Mail: (e-mail address removed)
PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
questions!

 

[Eric Sosman]

IO-operations are byte-oriented (this is not java specific)
so when writing shorts (2 bytes) there are two ways to do
this and it depends if the data you write to disk should
be read by other programs as well. So before you use
DataOutputStream you should check if the format to be used
is LSB least significant byte first or HSB (high significant
byte first). HSB is used with DataOutputStream, if you
want to write LSB you need to do the writing for yourself:

outstream.write(myshort & 0xff);
outstream.write((myshort >> 8) & 0xff);

other direction:

short myshort = instream.read() & 0xff;
myshort |= (instream.read() << 8) & 0xff;

Three of the 0xff masks are superfluous and the fourth
is incorrect (assuming "outstream" is a java.iutputStream
and "instream" a java.io.InputStream). Instead, try

outstream.write(myshort);
outstream.write(myshort >> 8);

and

short myshort = instream.read();
myshort |= instream.read() << 8;

or even

short myshort = instream.read() | (instream.read() << 8);

.... although as a dyed-in-the-wool C programmer this last version
makes me cringe.

 

[Lothar Kimmeringer]

What advantages does this have compared to wrapping the output stream in
a DataOutputStream and using its writeShort method?

As I said (and you quoted it above):

Regards, Lothar
--
Lothar Kimmeringer E-Mail: (e-mail address removed)
PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
questions!

 

[Lothar Kimmeringer]

Three of the 0xff masks are superfluous

I know but it's a way of telling other that I really
want to write only the bits 0 to 7.

and the fourth
is incorrect

True. Slipped my mind.

(assuming "outstream" is a java.iutputStream
and "instream" a java.io.InputStream). Instead, try

outstream.write(myshort);
outstream.write(myshort >> 8);

As I said, the & 0xff is my way of documenting and to make
sure that I don't forget the parts where it's used.

and

short myshort = instream.read();
myshort |= instream.read() << 8;

or even

short myshort = instream.read() | (instream.read() << 8);

... although as a dyed-in-the-wool C programmer this last version
makes me cringe.

Of course it should always be checked if the end of stream
hasn't been reached ;-)


Regards, Lothar
--
Lothar Kimmeringer E-Mail: (e-mail address removed)
PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
questions!

 

[KevinSimonson]

On Sep 26, 1:45 pm, Knute Johnson <[email protected]>
wrote:

=I'm not sure what a binary <short> is but to write a Java short to a
=stream you can use the DataOutputStream class. Also you can convert
=it to bytes and use a regular OutputStream. Also see
=BufferedOutputStream.

Thanks for the example snippet of code; I'm going to start using it
right away. But when I looked in the API for the description of
<BufferedOutputStream> and <BufferedInputStream>, I couldn't figure
out exactly how I'd use those classes to write Java <short>s to disk
and read them from disk.

I guess I could convert each <short> to a <byte> array and then call
<write( toDiskArray, 0, 2)>, and then call
<read( fromDiskArray, 0, 2)>, and then convert <fromDiskArray> back
into a <short>, but this just seems like a really messy way to write
and read a <short>. Is there a better way?

Kevin Simonson

"You'll never get to heaven, or even to LA,
if you don't believe there's a way."
from _Why Not_

 

[Lothar Kimmeringer]

Remember the original question "How do you do that in Java, write binary
<short>s to disk and then later read in those <short>s from disk again?"

What advantage to you see, given that requirement, to writing the data
in little-endian format?

You don't like reading, do you? I also wrote

The OP didn't say anything about other programs but there is nothing
wrong in checking before if that is the case before you use a
specific encoding and regret it later.


Regards, Lothar
--
Lothar Kimmeringer E-Mail: (e-mail address removed)
PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
questions!

 

[Lothar Kimmeringer]

Thanks for the example snippet of code; I'm going to start using it
right away. But when I looked in the API for the description of
<BufferedOutputStream> and <BufferedInputStream>, I couldn't figure
out exactly how I'd use those classes to write Java <short>s to disk
and read them from disk.

I guess I could convert each <short> to a <byte> array and then call
<write( toDiskArray, 0, 2)>, and then call
<read( fromDiskArray, 0, 2)>, and then convert <fromDiskArray> back
into a <short>, but this just seems like a really messy way to write
and read a <short>. Is there a better way?

I'm sure he meant it that way:

FileOutputStream fos = new FileOutputStream(file);
BufferedOutputStream bos = new BufferedOutputStream(fos);
DataOutputStream dos = new DataOutputStream(bos);

Then you can still use writeShort without writing every single
directly to the disk which would lead to a drop in performance.


Regards, Lothar
--
Lothar Kimmeringer E-Mail: (e-mail address removed)
PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
questions!

 

[Qu0ll]

I'm planning on writing a program that stores a lot of binary informa-
tion, and I'd like to be able to write it to disk when I'm done with
it, so that later I can read it in from disk again. How do you do
that in Java, write binary <short>s to disk and then later read in
those <short>s from disk again?

The advice so far has been to use traditional java.io but I have found that
you'll get better performance if you use nio and especially if you use a
MappedByteBuffer which has methods to put shorts, ints etc.

--
And loving it,

-Qu0ll (Rare, not extinct)
_________________________________________________
(e-mail address removed)
[Replace the "SixFour" with numbers to email me]

 

[Lew]

Thanks for the example snippet of code; I'm going to start using it
right away. But when I looked in the API for the description of
<BufferedOutputStream> and <BufferedInputStream>, I couldn't figure
out exactly how I'd use those classes to write Java <short>s to disk
and read them from disk.

Dude, don't use angle brackets for Java conversations that way. Angle
brackets have a semantic - they mean "generic base type". That you don't mean
that is jarring. In your paragraph one wants to read "<BufferedOutputStream>"
as "generic type BufferedOutputStream". That is confusing.

I use single quotes around type names meant to be taken literally in an
otherwise natural-language paragraph. Very often people just ignore special
typography and let you understand that it's a Java type from context.

 

[Lew]

You don't like reading, do you?

That was uncalled-for and very snotty. It also didn't work - you don't get to
duck the question that way.

I also wrote

That doesn't answer Patricia's question. It is barely even related to her
question. Now why don't you be a polite little boy and just answer what she
asked?

The OP didn't say anything about other programs

Exactly so.

 

[Roedy Green]

I'm planning on writing a program that stores a lot of binary informa-
tion, and I'd like to be able to write it to disk when I'm done with
it, so that later I can read it in from disk again. How do you do
that in Java, write binary <short>s to disk and then later read in
those <short>s from disk again?

For simple arrays, use DataOutputStream. For complex objects, use
OutputObjectStream.

For sample code see

http://mindprod.com/applet/fileio.html
--
Roedy Green Canadian Mind Products
http://mindprod.com

"Civilisation advances by extending the number of important operations which we can perform without thinking about them."
~ Alfred North Whitehead (born: 1861-02-15 died: 1947-12-30 at age: 86)

 

[Roedy Green]

There another thread about little-endian, big-endian which you might
be conflating with this one. O.P. just wanted to write out some values
and read them back. That would naturally be done in big-endian with
DataOutputStream. DataOutputStream always writes big-endian,
independent of the endianness of the platform.
--
Roedy Green Canadian Mind Products
http://mindprod.com

"Civilisation advances by extending the number of important operations which we can perform without thinking about them."
~ Alfred North Whitehead (born: 1861-02-15 died: 1947-12-30 at age: 86)

 

[Eric Sosman]

Remember the original question "How do you do that in Java, write binary
<short>s to disk and then later read in those <short>s from disk again?"

What advantage to you see, given that requirement, to writing the data
in little-endian format?

Elsethread, the O.P. has explained that the data will be
consumed by a C program that expects to read a "native" format.

 

[Lew]

Elsethread, the O.P. has explained that the data will be
consumed by a C program that expects to read a "native" format.

I'm not finding that. I'm only finding:

How do you do that in Java,
write binary <short>s to disk and then later read in
those <short>s from disk again?
and

I guess I could convert each <short> to a <byte> array and then call
<write( toDiskArray, 0, 2)>, and then call
<read( fromDiskArray, 0, 2)>, and then convert <fromDiskArray> back
into a <short>,

after going back about a month's worth through clj.programmer and clj.help for
posts by the OP in any thread.

The only reference in this thread to reading by other programs was not from
the OP but from Lothar Kimmeringer, which is why Patricia asked him about the
advantages of little-endian storage, a question he has yet to answer.

As Roedy Green said in this thread:

There['s] another thread about little-endian, big-endian which you might
be conflating with this one. O.P. just wanted to write out some values
and read them back. That would naturally be done in big-endian with
DataOutputStream. DataOutputStream always writes big-endian,
independent of the endianness of the platform.

It looks like that other thread, the one you must be misremembering, Eric, is
in clj.help from Alexandre Ferrieux, "How to (efficiently) write an int array
into a file ?" [sic] 2009-09-25 at 21:14Z:

How do we accomplish this simple task of a direct write to disk of a
big (packed) array of ints, in native byte order ? [sic]

 

[Eric Sosman]

Elsethread, the O.P. has explained that the data will be
consumed by a C program that expects to read a "native" format.

I'm not finding that. [...]
It looks like that other thread, the one you must be misremembering,
Eric, is in clj.help from Alexandre Ferrieux, "How to (efficiently)
write an int array into a file ?" [sic] 2009-09-25 at 21:14Z:

Yes: I have woven two threads into a tangle.

 

[KevinSimonson]

On Sep 26, 3:54 pm, Lothar Kimmeringer <[email protected]>
wrote:

=I'm sure he meant it that way:
=
=FileOutputStream fos = new FileOutputStream(file);
=BufferedOutputStream bos = new BufferedOutputStream(fos);
=DataOutputStream dos = new DataOutputStream(bos);
=
=Then you can still use writeShort without writing every single
=directly to the disk which would lead to a drop in performance.

Thanks, Lothar; this is precisely what I wanted to know.

Kevin Simonson

"You'll never get to heaven, or even to LA,
if you don't believe there's a way."
from _Why Not_