NIO slow (no buffering?)

[Timo Nentwig]

Hi!

I probably yet don't understand some fundamental things about NIO. The
following code is painfully slow (8sec. for a 6MiB file):

final FileChannel in = new FileInputStream( f ).getChannel();
final ByteBuffer token = ByteBuffer.allocate( 4 );

while( in.position() < in.size() )
{
in.read( token );
final String s = new String( token.array() );
token.flip();
}

What am I doing wrong?

 

[Andrey Kuznetsov]

I probably yet don't understand some fundamental things about NIO. The

following code is painfully slow (8sec. for a 6MiB file):

final FileChannel in = new FileInputStream( f ).getChannel();
final ByteBuffer token = ByteBuffer.allocate( 4 );

while( in.position() < in.size() )
{
in.read( token );
final String s = new String( token.array() );
token.flip();
}

What am I doing wrong?

creating 1.5 millions String Objects?

 

[Skip]

Hi!

I probably yet don't understand some fundamental things about NIO. The
following code is painfully slow (8sec. for a 6MiB file):

final FileChannel in = new FileInputStream( f ).getChannel();
final ByteBuffer token = ByteBuffer.allocate( 4 );

reading small packs of bytes is very slow, besides that, use direct-buffers:
ByteBuffer.allocateDirect( x );

while( in.position() < in.size() )
{
in.read( token );
final String s = new String( token.array() );
token.flip();
}

I'd change the design here. Read as much bytes as possible in read(), like
8192, then parse it and chop it in pieces.

int size = 8192;
ByteBuffer token = ByteBuffer.allocateDirect( size );
byte[] temp = new byte[size];

for(...)
{
in.read(token);
in.flip();
token.read(temp, 0, in.limit());
String raw = new String(temp, 0, in.limit(), "UTF-8");

// chop [raw] into 4-size parts
}

Further I'd suggest you to google a bit for tutorials about NIO to see which
actions are slow, or what's good and what's considered to be bad design.

HTH

 

[Skip]

for(...)
{ in.clear();
in.read(token);
in.flip();

<snip>

Or your buffer-limit would get smaller and smaller (sooner or later)

 

[Timo Nentwig]

reading small packs of bytes is very slow, besides that, use

I know, that's why I put "no buffering" in the subject...

direct-buffers: ByteBuffer.allocateDirect( x );

There's no ByteBuffer.array() in this case

 

[Timo Nentwig]

I'd change the design here. Read as much bytes as possible in read(), like
8192, then parse it and chop it in pieces.

Since it's "only" 6 MiB I thought about reading the entire file at once. But
this way I don't need NIO at all...

 

[Andrey Kuznetsov]

I probably yet don't understand some fundamental things about NIO. The

reading small packs of bytes is very slow, besides that, use
direct-buffers:
ByteBuffer.allocateDirect( x );

if you have nothing special with nio (e.g. multiplexing) then consider to
use Unified I/O.
Unified I/O makes transparent buffering, allows random access nearly to
everything and
can also read data direct from/to primitive arrays.
see http://uio.imagero.com and http://uio.imagero.com/performance.html for
more info.

 

[Skip]

I know, that's why I put "no buffering" in the subject...


There's no ByteBuffer.array() in this case

I typed the solution out for you, have you read it??

 

[Timo Nentwig]

creating 1.5 millions String Objects?

If you have a better idea, you are welcome! I do a Map lookup:

private static final Map MAP = new HashMap()
{
{
// put( "mhbd", new Foo() );
put( new byte[]{'m', 'h', 'b', 'd'}, new Foo() );
}
};

...


MAP.get(new String(token.array())); // will work
MAP.get(token.array()); // won't

 

[Timo Nentwig]

I typed the solution out for you, have you read it??

Yes, but in this case I'd rather read the entire file at once. Why should I
read 8k chunks?

 

[Skip]

creating 1.5 millions String Objects?

If you have a better idea, you are welcome! I do a Map lookup:

private static final Map MAP = new HashMap()
{
{
// put( "mhbd", new Foo() );
put( new byte[]{'m', 'h', 'b', 'd'}, new Foo() );
}
};

...


MAP.get(new String(token.array())); // will work
MAP.get(token.array()); // won't

If you want to play it dirty, you can encode 4 bytes in a single int, then
find an IntHashMap somewhere (or code it yourself) and it's much faster an
more efficient.

 

[Skip]

Yes, but in this case I'd rather read the entire file at once. Why should I
read 8k chunks?

Because you cannot read it at once. Because the read(arg) in both
InputStream and ByteBuffer do not state to fully fill the [arg] - they can,
but you cannot be sure.

You might want to look at BufferedInputStream.readFully(byte[], int, int),
that does the step-by-step filling for you, but then you lose NIO. So just
read it in chunks, because that happens anyway, in one way or another.

 

[Timo Nentwig]

that does the step-by-step filling for you, but then you lose NIO. So just

That's the topic of this thread: am I actually losing anything if I lose
NIO. When it comes to my tests: no. And I wonder wonder about the "N" in
NIO...

 

[Skip]

just

That's the topic of this thread: am I actually losing anything if I lose
NIO. When it comes to my tests: no. And I wonder wonder about the "N" in
NIO...

NIO is IO with another approach (allowing non-blocking IO)

In your case the performance will be the same when you use IO instead of
NIO.
I would use IO here, it's easier here, and you gain 'nothing' with NIO now.
Just be sure to read(byte[]) a lot of bytes per call.

You should read about NIO and the reason it was developed. If you don't care
enough, and do not find problems with your code / performance, just use IO,
it's ok.

 

[Timo Nentwig]

NIO is IO with another approach (allowing non-blocking IO) ^^^^^^^^^^^^

In your case the performance will be the same when you use IO instead of
NIO.
I would use IO here, it's easier here, and you gain 'nothing' with NIO
now. Just be sure to read(byte[]) a lot of bytes per call.

You should read about NIO and the reason it was developed. If you don't
care enough, and do not find problems with your code / performance, just
use IO, it's ok.

Well, IO is just as slow if you don't use a BufferedInputStream. There's
certainly a way to buffer NIO, isn't it?

 

[Timo Nentwig]

Yes, but in this case I'd rather read the entire file at once. Why should

...which is obviously the best choice: It takes 100ms, BufferedInputStream
takes 2,5s and NIO 8s.

Is there some utility class out there to act on the byte array as it it was
an input stream (ByteInputStream or something)?

 

[Timo Nentwig]

..which is obviously the best choice: It takes 100ms, BufferedInputStream
takes 2,5s and NIO 8s.

Is there some utility class out there to act on the byte array as it it
was an input stream (ByteInputStream or something)?

ByteArrayInputStream, sure....

 

[Timo Nentwig]

if you have nothing special with nio (e.g. multiplexing) then consider to
use Unified I/O.
Unified I/O makes transparent buffering, allows random access nearly to
everything and
can also read data direct from/to primitive arrays.
see http://uio.imagero.com and http://uio.imagero.com/performance.html for
more info.

I managed to read the file and iterate thru it in 4 byte chunks in about
60ms using NIO and a DirectByteBuffer. Do you think UIO can beat this?

 

[Timo Nentwig]

I managed to read the file and iterate thru it in 4 byte chunks in about
60ms using NIO and a DirectByteBuffer. Do you think UIO can beat this?

Oops, I forgot the flipping. 140ms. Ok, it's as fast a
BufferedInputStream...

 

[Skip]

NIO is IO with another approach (allowing non-blocking IO) ^^^^^^^^^^^^

In your case the performance will be the same when you use IO instead of
NIO.
I would use IO here, it's easier here, and you gain 'nothing' with NIO
now. Just be sure to read(byte[]) a lot of bytes per call.

You should read about NIO and the reason it was developed. If you don't
care enough, and do not find problems with your code / performance, just
use IO, it's ok.

Well, IO is just as slow if you don't use a BufferedInputStream.

It's not. BufferedInputStream does the buffering for you, you can do that
yourself, so it's not slow at all. You should really check the sourcecode of
BufferedInputStream.