Writing to file

[sara]

Hi All,

I have a program which generates many lines of data where each line
includes an integer and two floats. I want to know the fastest way to
write these generated lines to a file. Currently, I am writing to a
binary file as follows:
FileOutputStream fos = new FileOutputStream(fileName);
DataOutputStream dos = new DataOutputStream(fos);
for (n : N) {
dos.writeInt(n.id);
dos.writeFloat(x.floatValue());
dos.writeFloat(y.floatValue());
}

However it seems that this approach is very slow. Can I use any kind
of buffering technique to speed up writing to binary file?

Best
Sara

 

[Lew]

Hi All,

I have a program which generates many lines of data where each line
includes an integer and two floats. I want to know the fastest way to
write these generated lines to a file. Currently, I am writing to a
binary file as follows:
FileOutputStream fos = new FileOutputStream(fileName);
DataOutputStream dos = new DataOutputStream(fos);
for (n : N) {
dos.writeInt(n.id);
dos.writeFloat(x.floatValue());
dos.writeFloat(y.floatValue());
}

However it seems that this approach is very slow. Can I use any kind
of buffering technique to speed up writing to binary file?

Have you considered reading the API docs?
<http://docs.oracle.com/javase/7/docs/api/java/io/BufferedOutputStream.html>

Note that you will need to build the DataOutputStream on top of the BufferedOutputStream rather than the other way around.

As to whether this helps performance, what measurements have you done, and how do you know that the result is "very slow"? Compared to what?

What were the conditions of the measurements (other load on the system, hard drive configuration, etc.)?

 

[Roedy Green]

DataOutputStream dos = new DataOutputStream(fos);
for (n : N) {
dos.writeInt(n.id);
dos.writeFloat(x.floatValue());

See http://mindprod.com/applet/fileio.html

It will generate you code to do this. Make sure you ask for buffered
if you want speed.

Get rid of Float objects and just use pure float primitives if you
want speed.

There is not much overhead, just flipping the little endian Intel uses
internally to big endian.

--
Roedy Green Canadian Mind Products
http://mindprod.com
For me, the appeal of computer programming is that
even though I am quite a klutz,
I can still produce something, in a sense
perfect, because the computer gives me as many
chances as I please to get it right.

 

[Arne Vajhøj]

I have a program which generates many lines of data where each line
includes an integer and two floats. I want to know the fastest way to
write these generated lines to a file. Currently, I am writing to a
binary file as follows:
FileOutputStream fos = new FileOutputStream(fileName);
DataOutputStream dos = new DataOutputStream(fos);
for (n : N) {
dos.writeInt(n.id);
dos.writeFloat(x.floatValue());
dos.writeFloat(y.floatValue());
}

However it seems that this approach is very slow. Can I use any kind
of buffering technique to speed up writing to binary file?

You can use a BufferedOutputStream between the DataOutputStream
and the FileOutputStream.

But I am a bit skeptical about the choice of format for
persisting data.

Arne