OK, it's later now, so here I am again.
Testing reading and writing to file or to the LAN. The test involves reading
or writing 1GB either bytewise, or using a 1K, 8K, or 64K buffer.
Incidentally the size of the test is very important for file read/writes
otherwise OS buffering will invalidate the test. The test machine has 512Mb
RAM, of which most is free during the test, but the 1Gb is large enough that
just about all the data will have to be read/written in each case, and the
machine's OS caches are only (and legitimately) being used for
read-ahead/write-behind.
My test is running on a WinXP Pro laptop, using Sun's 1.4.2 JVM.
The network tests connect to a server that either reads or writes data as fast
as it can. In my test the server was running Sun's 1.4.1 on Win2K -- not that
it matters much, the only important thing is that the server machine can both
produce and consume data faster than the test machine can.
I'll add the code for the test and server process at the end of this post.
Here are the results. The last columns is the time in seconds, the numbers in
[square brackets] are the size of the array used for the read(byte[]) or
write(byte[]) calls.
File write bytewise: 1073741824 bytes: 86.464
File write[1024]: 1073741824 bytes: 92.713
File write[8192]: 1073741824 bytes: 81.377
File write[65536]: 1073741824 bytes: 73.215
Network write bytewise: 1073741824 bytes: 118.88
Network write[1024]: 1073741824 bytes: 94.015
Network write[8192]: 1073741824 bytes: 91.812
Network write[65536]: 1073741824 bytes: 91.151
File read bytewise: 1073741824 bytes: 45.385
File read[1024]: 1073741824 bytes: 45.105
File read[8192]: 1073741824 bytes: 45.135
File read[65536]: 1073741824 bytes: 45.065
Network read bytewise: 1073741824 bytes: 94.766
Network read[1024]: 1073741824 bytes: 90.861
Network read[8192]: 1073741824 bytes: 90.84
Network read[65536]: 1073741824 bytes: 90.791
In general the use of bytewise IO is costing very little -- at least not on my
system. Still, I'd expect similar results on other systems.
Note that these tests don't actually do anything with the data they read, or
take time creating the data to write. So in the real world most applications
more complicated than a simple file-copy would expect to see relatively less
benefit from using the array forms.
Another note is that both the array and the bytewise tests are using a
buffered
stream. That may be putting the array forms at a slight disadvantage compared
to a raw stream (since there's more complex code path, to no real advantage).
I expect that effect to be pretty small, but admit I haven't measured it.
-- chris
Rather manky code follows. Main takes the number of Mbytes to read/write as
its single agument. Server takes no arguments at all.
============= Main.java ============
import java.io.*;
import java.net.Socket;
public class Main
{
private static byte[] buffer1 = new byte[1024 * 1];
private static byte[] buffer8 = new byte[1024 * 8];
private static byte[] buffer64 = new byte[1024 * 64];
private static final String FILENAME = "C:/temp/crud.dat";
private static final String SERVER = "**THE OTHER MACHINE**";
public static void
main(String args[])
throws NumberFormatException
{
int todo = args.length > 0
? Integer.parseInt(args[0])
: 1; // deliberately tiny for testing
todo *= 1024 * 1024; // but not /that/ tiny...
initialiseBuffers();
timeWritesToFile(todo);
timeWritesToNetwork(todo);
timeReadsFromFile(todo);
timeReadsFromNetwork(todo);
}
private static void
initialiseBuffers()
{
initialiseBuffer(buffer1);
initialiseBuffer(buffer8);
initialiseBuffer(buffer64);
}
private static void
initialiseBuffer(byte[] buffer)
{
for (int i = 0; i < buffer.length; i++)
buffer
= (byte)(i-128);
}
private static void
timeWritesToFile(int todo)
{
timeWriteToFile(todo);
timeWriteToFile(todo, buffer1);
timeWriteToFile(todo, buffer8);
timeWriteToFile(todo, buffer64);
}
private static void
timeWritesToNetwork(int todo)
{
timeWriteToNetwork(todo);
timeWriteToNetwork(todo, buffer1);
timeWriteToNetwork(todo, buffer8);
timeWriteToNetwork(todo, buffer64);
}
private static void
timeWriteToFile(int todo)
{
try
{
OutputStream stream = openFileOutputStream();
timeWriteStream("File", stream, todo);
stream.close();
}
catch (Exception e)
{
reportError(e, "timeWriteToFile(" + todo + ")");
}
}
private static void
timeWriteToFile(int todo, byte[] buffer)
{
try
{
OutputStream stream = openFileOutputStream();
timeWriteStream("File", stream, todo, buffer);
stream.close();
}
catch (Exception e)
{
reportError(e, "timeWriteToFile(" + todo + ", " + buffer.length + ")");;
}
}
private static void
timeWriteToNetwork(int todo)
{
try
{
OutputStream stream = openNetworkOutputStream();
timeWriteStream("Network", stream, todo);
stream.close();
}
catch (Exception e)
{
reportError(e, "timeWriteToNetwork(" + todo + ")");;
}
}
private static void
timeWriteToNetwork(int todo, byte[] buffer)
{
try
{
OutputStream stream = openNetworkOutputStream();
timeWriteStream("Network", stream, todo, buffer);
stream.close();
}
catch (Exception e)
{
reportError(e, "timeWriteToNetwork(" + todo + ", " + buffer.length + ")");;
}
}
private static void
timeWriteStream(String name, OutputStream stream, int todo)
throws IOException
{
int done = 0;
long start = now();
while (done < todo)
{
stream.write((byte)(done-128));
done++;
}
stream.flush();
reportTime("write bytewise", done, name, now()-start);
}
private static void
timeWriteStream(String name, OutputStream stream, int todo, byte[] buffer)
throws IOException
{
int done = 0;
long start = now();
while (done < todo)
{
int write = Math.min(todo-done, buffer.length);
stream.write(buffer, 0, write);
done += write;
}
stream.flush();
reportTime("write[" + buffer.length + "]", done, name, now()-start);
}
private static void
timeReadsFromFile(int todo)
{
timeReadFromFile(todo);
timeReadFromFile(todo, buffer1);
timeReadFromFile(todo, buffer8);
timeReadFromFile(todo, buffer64);
}
private static void
timeReadsFromNetwork(int todo)
{
timeReadFromNetwork(todo);
timeReadFromNetwork(todo, buffer1);
timeReadFromNetwork(todo, buffer8);
timeReadFromNetwork(todo, buffer64);
}
private static void
timeReadFromFile(int todo)
{
try
{
InputStream stream = openFileInputStream();
timeReadStream("File", stream, todo);
stream.close();
}
catch (Exception e)
{
System.out.println("Did not test timeReadFromFile(" + todo + ")");;
}
}
private static void
timeReadFromFile(int todo, byte[] buffer)
{
try
{
InputStream stream = openFileInputStream();
timeReadStream("File", stream, todo, buffer);
stream.close();
}
catch (Exception e)
{
reportError(e, "timeReadFromFile(" + todo + ", " + buffer.length + ")");;
}
}
private static void
timeReadFromNetwork(int todo)
{
try
{
InputStream stream = openNetworkInputStream();
timeReadStream("Network", stream, todo);
stream.close();
}
catch (Exception e)
{
reportError(e, "timeReadFromNetwork(" + todo + ")");;
}
}
private static void
timeReadFromNetwork(int todo, byte[] buffer)
{
try
{
InputStream stream = openNetworkInputStream();
timeReadStream("Network", stream, todo, buffer);
stream.close();
}
catch (Exception e)
{
reportError(e, "timeReadFromNetwork(" + todo + ", " + buffer.length +
")");;
}
}
private static void
timeReadStream(String name, InputStream stream, int todo)
throws IOException
{
int done = 0;
long start = now();
while (done < todo)
{
// not actually interested in EOF or value read
stream.read();
done++;
}
reportTime("read bytewise", done, name, now()-start);
}
private static void
timeReadStream(String name, InputStream stream, int todo, byte[] buffer)
throws IOException
{
int done = 0;
long start = now();
while (done < todo)
{
// just overwrite buffer contents and ignore possible EOF
done += stream.read(buffer, 0, Math.min(todo-done, buffer.length));
}
reportTime("read[" + buffer.length + "]", done, name, now()-start);
}
private static OutputStream
openNetworkOutputStream()
throws IOException
{
return new BufferedOutputStream(
new Socket(SERVER, Server.WRITE_PORT)
.getOutputStream());
}
private static OutputStream
openFileOutputStream()
throws IOException
{
return new BufferedOutputStream(
new FileOutputStream(
FILENAME));
}
private static InputStream
openNetworkInputStream()
throws IOException
{
return new BufferedInputStream(
new Socket(SERVER, Server.READ_PORT)
.getInputStream());
}
private static InputStream
openFileInputStream()
throws IOException
{
return new BufferedInputStream(
new FileInputStream(
FILENAME));
}
private static long
now()
{
return System.currentTimeMillis();
}
private static void
reportTime(String test, int size, String name, long time)
{
System.out.print(name + " " + test + ":\t");
System.out.print(size + " bytes:\t");
System.out.println(time / 1000.0);
}
private static void
reportError(Exception e, String test)
{
System.err.print("Did not test ");
System.err.println(test);
e.printStackTrace();
}
}
========== end of Main.java ==========
============= Server.java ============
import java.io.*;
import java.net.Socket;
import java.net.*;
import java.io.*;
public abstract class Server
implements Runnable
{
public static final int READ_PORT = 3344; // port CLIENTS read from
public static final int WRITE_PORT = 4433; // port CLIENTS write to
protected final byte[] buffer = new byte[1024 * 256];
public static void
main(String[] args)
{
new Thread(new Reader()).start();
new Thread(new Writer()).start();
}
public void
run()
{
ServerSocket socket = null;
try
{
socket = new ServerSocket(serverPort());
}
catch (IOException e)
{
System.err.println(e);
return;
}
for (;
{
try
{
System.err.println("waiting on " + socket);
Socket client = socket.accept();
System.err.println("got " + client);
handle(client);
client.close();
System.err.println("finished with " + client);
}
catch (IOException e)
{
System.err.println(e);
}
}
}
protected abstract int serverPort();
protected abstract void handle(Socket client) throws IOException;
}
class Reader
extends Server
{
protected int serverPort()
{
return WRITE_PORT;
}
protected void
handle(Socket client)
throws IOException
{
InputStream in = client.getInputStream();
while (in.read(buffer) != -1)
;
in.close();
}
}
class Writer
extends Server
{
protected int serverPort()
{
return READ_PORT;
}
protected void
handle(Socket client)
throws IOException
{
OutputStream out = client.getOutputStream();
try
{
for (;
out.write(buffer);
}
catch (IOException e)
{
}
out.close();
}
}
========== end of Server.java ==========
