nio limitations?

[Roedy Green]

Is this true, or have I overlooked something?

nio gives you the ability to read the entire file into ram or map it
into virtual ram. There is no ability to slide a viewing window over
the file the way you effectively do with an ordinary Stream.

Buffer is a misnomer. It is the contents of the file.

nio gives you the ability to look at binary file contents as big or
little endian fields, but only if the ints start on even 4-byte
boundaries relative to the start of the file.

 

[Thomas Hawtin]

Is this true, or have I overlooked something?

nio gives you the ability to read the entire file into ram or map it
into virtual ram. There is no ability to slide a viewing window over
the file the way you effectively do with an ordinary Stream.

You can map a section of a file. You can also read sections into
ByteBuffers.

The one big problem is that there is no way to unmap a file.

Tom Hawtin

 

[Roedy Green]

You can map a section of a file. You can also read sections into
ByteBuffers.

But would you read an entire file this way, mapping your way along?
Was this mechanism intended for sequentially reading files? It seems
clumsy compared with a stream. It does not have any lookahead
ability, but I guess neither do ordinary Java streams. This seems odd
given that double buffering with i/o lookahead was pulled off back in
the days of 16K machines.

 

[EJP]

Was this mechanism intended for sequentially reading files?

It's very useful for random access files e.g. databases. Very useful
indeed, and substantial performances have been observed e.g. on Lucene
indexes.

It seems clumsy compared with a stream.

Certainly, but then a stream is clumsy compared to random access with
seeks, and random access with seeks is clumsy compared with direct
addressing via a memory mapped file. So?

It does not have any lookahead
ability, but I guess neither do ordinary Java streams.

The facility gives you a virtual address for the file contents, or part
thereof. Being VM it is read on demand. One would hope that the
platform's VM and/or I/O subsystem or at least the disk controller would
be doing readahead automatically. Otherwise what indeed have we learnt
since the 1960s. Anyway not much point in implementing it at the Java
level if these lower-level guys aren't doing it.

 

[Thomas Weidenfeller]

nio gives you the ability to read the entire file into ram or map it
into virtual ram. There is no ability to slide a viewing window over
the file the way you effectively do with an ordinary Stream.

FileChannel allows you to map only a region of a file and not the whole
file into memory. You could use this to map one region after the other
into memory.

Buffer is a misnomer. It is the contents of the file.

It has to be, otherwise you couldn't see the file's contents. However,
the question is how does the contents end up there. If it is indeed
backed by memory mapped I/O, then mapping a section of a file to virtual
memory should be a rather cheap operation (some registers and values in
the OS and MMU are loaded and configured, and that's it). Only when you
start to access the memory it should trigger some page fault, upon which
part of the virtual memory should me mapped to real memory and the file
data being transfered to it. Disk I/O is probably the limiting factor here.

nio gives you the ability to look at binary file contents as big or
little endian fields, but only if the ints start on even 4-byte
boundaries relative to the start of the file.

Since when? ByteBuffer.getInt(offset) can take any kind of offset and
does not assume 4-byte alignment. Since you can also change the endian
of a byte buffer, this should bring you some way.

/Thomas

 

[Thomas Hawtin]

then mapping a section of a file to virtual
memory should be a rather cheap operation (some registers and values in
the OS and MMU are loaded and configured, and that's it). Only when you
start to access the memory it should trigger some page fault, upon which
part of the virtual memory should me mapped to real memory and the file
data being transfered to it. [...]

That doesn't sound too lightweight to me. Compare it against the
non-mapped way of just copying a section of memory (preferably ignoring
caches). Mustang on UNIX/Linux changed from memory mapping jars to
reading conventionally, which gained some performance.

File memory mapping is not a good fit for reading a file sequentially.

Tom Hawtin

 

[bugbear]

File memory mapping is not a good fit for reading a file sequentially.

Really? Surely if you simple read the (mapped to file) memory
sequentially, the data will be read (via page faults),
and LRU pages of physical memory reused as needed.

The only performance issues would come from
not "giving back" physical memory pages
soon enough, leading to overall shortage of RAM.

BugBear

 

[bugbear]

then mapping a section of a file to
virtual memory should be a rather cheap operation (some registers and
values in the OS and MMU are loaded and configured, and that's it).
Only when you start to access the memory it should trigger some page
fault, upon which part of the virtual memory should me mapped to real
memory and the file data being transfered to it. [...]

That doesn't sound too lightweight to me. Compare it against the
non-mapped way of just copying a section of memory (preferably ignoring
caches). Mustang on UNIX/Linux changed from memory mapping jars to
reading conventionally, which gained some performance.

File memory mapping is not a good fit for reading a file sequentially.

Tom Hawtin

This is interesting; haven't analysed his code or results in detail:

http://www.skytopia.com/project/articles/javaio.html

BugBear

 

[Thomas Hawtin]

This is interesting; haven't analysed his code or results in detail:

http://www.skytopia.com/project/articles/javaio.html

There are many things wrong with that article. For the most part he
isn't even measuring times for reading from disc. He uses a very
unlifelike hundreds of lines long method. If my mental calculations are
correct, he is claiming that MappedByteBuffer takes around 30 cycles to
read each byte that is already in RAM.

Cameron Purdy has some fun with non-I/O NIO benchmarks in his weblog.
You need to get rid of the problems caused by the unrealistic measuring
technique first. Or just benchmark real problems.

Tom Hawtin