nio limitations?

Discussion in 'Java' started by Roedy Green, Mar 2, 2006.

  1. Roedy Green

    Roedy Green Guest

    Is this true, or have I overlooked something?

    nio gives you the ability to read the entire file into ram or map it
    into virtual ram. There is no ability to slide a viewing window over
    the file the way you effectively do with an ordinary Stream.

    Buffer is a misnomer. It is the contents of the file.

    nio gives you the ability to look at binary file contents as big or
    little endian fields, but only if the ints start on even 4-byte
    boundaries relative to the start of the file.

    --
    Canadian Mind Products, Roedy Green.
    http://mindprod.com Java custom programming, consulting and coaching.
    Roedy Green, Mar 2, 2006
    #1
    1. Advertising

  2. Roedy Green wrote:
    > Is this true, or have I overlooked something?
    >
    > nio gives you the ability to read the entire file into ram or map it
    > into virtual ram. There is no ability to slide a viewing window over
    > the file the way you effectively do with an ordinary Stream.


    You can map a section of a file. You can also read sections into
    ByteBuffers.

    The one big problem is that there is no way to unmap a file.

    Tom Hawtin
    --
    Unemployed English Java programmer
    http://jroller.com/page/tackline/
    Thomas Hawtin, Mar 2, 2006
    #2
    1. Advertising

  3. Roedy Green

    Roedy Green Guest

    On Thu, 02 Mar 2006 18:00:02 +0000, Thomas Hawtin
    <> wrote, quoted or indirectly quoted someone
    who said :

    >You can map a section of a file. You can also read sections into
    >ByteBuffers.


    But would you read an entire file this way, mapping your way along?
    Was this mechanism intended for sequentially reading files? It seems
    clumsy compared with a stream. It does not have any lookahead
    ability, but I guess neither do ordinary Java streams. This seems odd
    given that double buffering with i/o lookahead was pulled off back in
    the days of 16K machines.

    --
    Canadian Mind Products, Roedy Green.
    http://mindprod.com Java custom programming, consulting and coaching.
    Roedy Green, Mar 3, 2006
    #3
  4. Roedy Green

    EJP Guest

    Roedy Green wrote:

    > Was this mechanism intended for sequentially reading files?


    It's very useful for random access files e.g. databases. Very useful
    indeed, and substantial performances have been observed e.g. on Lucene
    indexes.

    > It seems clumsy compared with a stream.


    Certainly, but then a stream is clumsy compared to random access with
    seeks, and random access with seeks is clumsy compared with direct
    addressing via a memory mapped file. So?

    > It does not have any lookahead
    > ability, but I guess neither do ordinary Java streams.


    The facility gives you a virtual address for the file contents, or part
    thereof. Being VM it is read on demand. One would hope that the
    platform's VM and/or I/O subsystem or at least the disk controller would
    be doing readahead automatically. Otherwise what indeed have we learnt
    since the 1960s. Anyway not much point in implementing it at the Java
    level if these lower-level guys aren't doing it.
    EJP, Mar 3, 2006
    #4
  5. Roedy Green wrote:
    > nio gives you the ability to read the entire file into ram or map it
    > into virtual ram. There is no ability to slide a viewing window over
    > the file the way you effectively do with an ordinary Stream.


    FileChannel allows you to map only a region of a file and not the whole
    file into memory. You could use this to map one region after the other
    into memory.

    > Buffer is a misnomer. It is the contents of the file.


    It has to be, otherwise you couldn't see the file's contents. However,
    the question is how does the contents end up there. If it is indeed
    backed by memory mapped I/O, then mapping a section of a file to virtual
    memory should be a rather cheap operation (some registers and values in
    the OS and MMU are loaded and configured, and that's it). Only when you
    start to access the memory it should trigger some page fault, upon which
    part of the virtual memory should me mapped to real memory and the file
    data being transfered to it. Disk I/O is probably the limiting factor here.

    > nio gives you the ability to look at binary file contents as big or
    > little endian fields, but only if the ints start on even 4-byte
    > boundaries relative to the start of the file.


    Since when? ByteBuffer.getInt(offset) can take any kind of offset and
    does not assume 4-byte alignment. Since you can also change the endian
    of a byte buffer, this should bring you some way.

    /Thomas
    --
    The comp.lang.java.gui FAQ:
    ftp://ftp.cs.uu.nl/pub/NEWS.ANSWERS/computer-lang/java/gui/faq
    http://www.uni-giessen.de/faq/archiv/computer-lang.java.gui.faq/
    Thomas Weidenfeller, Mar 3, 2006
    #5
  6. Thomas Weidenfeller wrote:
    > then mapping a section of a file to virtual
    > memory should be a rather cheap operation (some registers and values in
    > the OS and MMU are loaded and configured, and that's it). Only when you
    > start to access the memory it should trigger some page fault, upon which
    > part of the virtual memory should me mapped to real memory and the file
    > data being transfered to it. [...]


    That doesn't sound too lightweight to me. Compare it against the
    non-mapped way of just copying a section of memory (preferably ignoring
    caches). Mustang on UNIX/Linux changed from memory mapping jars to
    reading conventionally, which gained some performance.

    File memory mapping is not a good fit for reading a file sequentially.

    Tom Hawtin
    --
    Unemployed English Java programmer
    http://jroller.com/page/tackline/
    Thomas Hawtin, Mar 3, 2006
    #6
  7. Roedy Green

    bugbear Guest

    Thomas Hawtin wrote:
    > File memory mapping is not a good fit for reading a file sequentially.


    Really? Surely if you simple read the (mapped to file) memory
    sequentially, the data will be read (via page faults),
    and LRU pages of physical memory reused as needed.

    The only performance issues would come from
    not "giving back" physical memory pages
    soon enough, leading to overall shortage of RAM.

    BugBear
    bugbear, Mar 3, 2006
    #7
  8. Roedy Green

    bugbear Guest

    Thomas Hawtin wrote:
    > Thomas Weidenfeller wrote:
    >
    >> then mapping a section of a file to
    >> virtual memory should be a rather cheap operation (some registers and
    >> values in the OS and MMU are loaded and configured, and that's it).
    >> Only when you start to access the memory it should trigger some page
    >> fault, upon which part of the virtual memory should me mapped to real
    >> memory and the file data being transfered to it. [...]

    >
    >
    > That doesn't sound too lightweight to me. Compare it against the
    > non-mapped way of just copying a section of memory (preferably ignoring
    > caches). Mustang on UNIX/Linux changed from memory mapping jars to
    > reading conventionally, which gained some performance.
    >
    > File memory mapping is not a good fit for reading a file sequentially.
    >
    > Tom Hawtin


    This is interesting; haven't analysed his code or results in detail:

    http://www.skytopia.com/project/articles/javaio.html

    BugBear
    bugbear, Mar 3, 2006
    #8
  9. bugbear wrote:
    >
    > This is interesting; haven't analysed his code or results in detail:
    >
    > http://www.skytopia.com/project/articles/javaio.html


    There are many things wrong with that article. For the most part he
    isn't even measuring times for reading from disc. He uses a very
    unlifelike hundreds of lines long method. If my mental calculations are
    correct, he is claiming that MappedByteBuffer takes around 30 cycles to
    read each byte that is already in RAM.

    Cameron Purdy has some fun with non-I/O NIO benchmarks in his weblog.
    You need to get rid of the problems caused by the unrealistic measuring
    technique first. Or just benchmark real problems.

    Tom Hawtin
    --
    Unemployed English Java programmer
    http://jroller.com/page/tackline/
    Thomas Hawtin, Mar 3, 2006
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Saqib Ali

    SOAP:Lite Limitations re: https??

    Saqib Ali, Sep 12, 2003, in forum: Perl
    Replies:
    1
    Views:
    3,019
    Serge Dubrouski
    Sep 16, 2003
  2. 123
    Replies:
    0
    Views:
    334
  3. Wayne
    Replies:
    3
    Views:
    318
    John Saunders
    Nov 12, 2003
  4. Trevor Oakley

    ASP.NET v ASP v CGI - limitations

    Trevor Oakley, Feb 17, 2004, in forum: ASP .Net
    Replies:
    4
    Views:
    697
    Kevin Spencer
    Feb 17, 2004
  5. iksrazal

    NIO with timeouts != NIO?

    iksrazal, Jun 17, 2004, in forum: Java
    Replies:
    1
    Views:
    6,217
    iksrazal
    Jun 18, 2004
Loading...

Share This Page