MappedByteBuffer

A

Alan Gutierrez

I'm implementing file storage for a B+Tree for deployment on Linux,
development on OS X, and wanted to use MappedByteBuffer. This seems like
an easy way to address large pages of data and not have implement
buffering myself.

I've done a little testing...

import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.channels.FileChannel.MapMode;

public class MemoryMap {
public static void main(String[] args) throws IOException {
int count = Integer.parseInt(args[0]);
MappedByteBuffer[] mappedByteBuffers =
new MappedByteBuffer[count];
for (int i = 0; i < count; i++) {
mappedByteBuffers = opened(i);
}
}

public static MappedByteBuffer opened(int count)
throws IOException {
int size = 1024 * 1024 * 256; // * 1024; // 1 GB
RandomAccessFile raf =
new RandomAccessFile("hello" + count + ".raf", "rw");
try {
raf.setLength(size);
FileChannel channel = raf.getChannel();
try {
MappedByteBuffer map
= channel.map(MapMode.READ_WRITE, 0, size);
channel.close();
map.put(size - 1, (byte) 7);
} finally {
channel.close();
}
} finally {
raf.close();
}
raf = new RandomAccessFile("hello" + count + ".raf", "rw");
try {
FileChannel channel = raf.getChannel();
try {
MappedByteBuffer map
= channel.map(MapMode.READ_WRITE, 0, size);
channel.close();
if (map.get(size - 1) != 7) {
throw new IllegalStateException();
}
return map;
} finally {
channel.close();
}
} finally {
raf.close();
}
}
}

On a smallish Linode running Ubuntu 10.4 I can create an array of
hundreds of MappedByteBuffer. Here are some timings.

[alan@maribor ~]$ time java MemoryMap 24

real 0m0.119s
user 0m0.067s
sys 0m0.029s
[alan@maribor ~]$ time java MemoryMap 512

real 0m0.225s
user 0m0.150s
sys 0m0.092s
[alan@maribor ~]$

On OS X running Java 1.5 I can create an array of 5, the first pass is
very slow, the second pass is faster, I assume this is because OS X
isn't being clever about creating the empty file.

[alan@postojna ~]$ time java MemoryMap 5

real 0m20.920s
user 0m0.089s
sys 0m1.267s
[alan@postojna ~]$ time java MemoryMap 5

real 0m0.301s
user 0m0.091s
sys 0m0.044s
[alan@postojna ~]$

When I attempt to create an array of 6 250 MB `MappedByteBuffer`s I get:

[alan@postojna ~]$ java MemoryMap 6
Exception in thread "main" java.io.IOException: Cannot allocate memory
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:742)
at MemoryMap.opened(MemoryMap.java:23)
at MemoryMap.main(MemoryMap.java:12)
[alan@postojna ~]$

However:

[alan@postojna ~]$ time
/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java
MemoryMap 6

real 0m0.386s
user 0m0.354s
sys 0m0.087s
[alan@postojna ~]$

I've created up to 24 250MB buffers using Java 1.6 on OS X. It takes a
while to create them, but once created, performance seems reasonable.

[alan@postojna ~]$ time
/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java
MemoryMap 24

real 3m13.779s
user 0m0.358s
sys 0m5.543s
[alan@postojna ~]$ time
/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java
MemoryMap 24

real 0m1.646s
user 0m0.364s
sys 0m0.100s
[alan@postojna ~]$

I've tried the program running on Windows XP with JDK 1.6 running in
Virtual Box and I can't get an array of three pages.

All this leads me to the question: am I on the right track? It seems
like I'll be able to get the benefits I seek,

* paging managed by the operating system,
* a simple memory management strategy.

on Linux and OS X with JDK 1.6.

Or am I misunderstanding the usage of MappedByteBuffer?
 
K

Kevin McMurtrie

Alan Gutierrez said:
I'm implementing file storage for a B+Tree for deployment on Linux,
development on OS X, and wanted to use MappedByteBuffer. This seems like
an easy way to address large pages of data and not have implement
buffering myself.

I've done a little testing...

import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer; ...
Or am I misunderstanding the usage of MappedByteBuffer?


Java 1.5 is more commonly a 32 bit process while Java 1.6 is more
commonly a 64 bit process. You're running out of memory addresses in 32
bit mode.

Adding '-d64' as the first JVM option will force the 64 bit version or
fail with a message that it's not installed.
 
A

Alan Gutierrez

Kevin said:
Java 1.5 is more commonly a 32 bit process while Java 1.6 is more
commonly a 64 bit process. You're running out of memory addresses in 32
bit mode.

Adding '-d64' as the first JVM option will force the 64 bit version or
fail with a message that it's not installed.

Adding -d64 to JDK 1.5 does allow it to address more than 6 pages. Thank
you.

Can you help me understand what is going on here? Is it that, Even
through `MappedByteBuffer` will page in and out of memory, you still
need a 64 bits to address entire region, and the addresses cannot overlap?
 
A

Alan Gutierrez

Lew said:
I see nothing in
<http://download.oracle.com/docs/cd/E17409_01/javase/6/docs/technotes/
tools/solaris/java.html>
that requires the "-d64" option to be the first option.

It works though:

[alan@postojna ~]$ java -d64 MemoryMap 6
[alan@postojna ~]$ java MemoryMap 6
Exception in thread "main" java.io.IOException: Cannot allocate memory
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:742)
at MemoryMap.opened(MemoryMap.java:23)
at MemoryMap.main(MemoryMap.java:12)
[alan@postojna ~]$
 
N

Nigel Wade

Adding -d64 to JDK 1.5 does allow it to address more than 6 pages. Thank
you.

Can you help me understand what is going on here? Is it that, Even
through `MappedByteBuffer` will page in and out of memory, you still
need a 64 bits to address entire region, and the addresses cannot overlap?

In the dim and distant past I did some work using memory mapped files in
C, and I presume that the mechanism behind Java memory mapped files is
similar. A memory mapped file is mapped onto a specific address in the
virtual memory address space of the application. Although they are not
actually occupying the memory there they still require a unique, and
contiguous, address space equal to the size of the memory mapped region.
So if you want to map a region of 256MB you need 256MB of contiguous
address space.

A value is read from a memory mapped region by accessing a particular
address in the virtual memory of the application e.g. if the file is
mapped at address 0x20000, and you want to read a value at offset 0x500
from that start of the file then the data at 0x20500 is read. This
address must be unique, so the mapped regions cannot overlap, they must
be contiguous, and all mapped regions must lie within the addressable
space of the application.

In a 32bit application your maximum address space is 4GB, and could be
a good deal less than that due to system dependent memory limitations.
Thus, your attempt to allocate multiple regions of that address space in
blocks of 256MB will have a limited amount of success, depending on what
contiguous regions of the address space are currently available.

When you run the 64bit version of the JVM the available address space
has increased by a factor of 1 billion. Therefore you are able to
allocate those regions of contiguous address space for your application,
even though there is still the same amount of physical RAM and system VM.
 
A

Alan Gutierrez

Patricia said:
Nigel said:
[...]
In a 32bit application your maximum address space is 4GB, and could be
a good deal less than that due to system dependent memory limitations.
Thus, your attempt to allocate multiple regions of that address space in
blocks of 256MB will have a limited amount of success, depending on what
contiguous regions of the address space are currently available.

Indeed, it's worth pointing out that on Windows, the maximum address
space for Java programs is only 2GB, and really only about 1.5GB (give
or take) after various overhead involved in a Windows process and Java.
And of course, any interesting program will have to allocate other
memory objects, reducing that further (through both just the use of
memory and fragmentation of the virtual address space).

I assume when you say "on Windows" you mean "on 32 bit Windows"?

I assume you're implying that 64 bit Windows has the same advantages as
64 bit JVM as noted by Nigel Wade?
 
A

Arne Vajhøj

Java 1.5 is more commonly a 32 bit process while Java 1.6 is more
commonly a 64 bit process.

For Sun Java then:
Windows Linux Solaris not SPARC
1.6 latest both avilable both available both available
1.5 latest both avilable both available both available
1.4.2 latest only 32 bit only 32 bit only 32 bit

So I can not see why it should necessarily be that way.

Arne
 
A

Arne Vajhøj

Patricia said:
Nigel Wade wrote:
[...]
In a 32bit application your maximum address space is 4GB, and could be
a good deal less than that due to system dependent memory limitations.
Thus, your attempt to allocate multiple regions of that address
space in
blocks of 256MB will have a limited amount of success, depending on
what
contiguous regions of the address space are currently available.

Indeed, it's worth pointing out that on Windows, the maximum address
space for Java programs is only 2GB, and really only about 1.5GB (give
or take) after various overhead involved in a Windows process and Java.
And of course, any interesting program will have to allocate other
memory objects, reducing that further (through both just the use of
memory and fragmentation of the virtual address space).

I assume when you say "on Windows" you mean "on 32 bit Windows"?

No. Without "large address aware" (Java is not, as delivered, "large
address aware"), 32-bit processes are limited to 2GB even on 64-bit
Windows.

If you enable "large address aware" (which you can do after the fact…so
in theory someone could hack Java's .exe to have that flag set), then on
32-bit Windows, the limit goes up to 3GB, and on 64-bit Windows the
limit goes up to 4GB.

But out of the box, 32-bit Java is restricted to 2GB whether running on
32-bit Windows or 64-bit Windows.

The IMAGE_FILE_LARGE_ADDRESS_AWARE is not the real problem.

SUN Java requires contiguous heap space. And even with /3GB there
are something around the 2 GB marker.

BEA (now Oracle) JRockit allows non-contiguous heap and allows
almost 3 GB heap with /3GB.

And almost 4 GB for 32 bit JVM on 64 bit Windows.

http://blogs.oracle.com/jrockit/2008/09/how_to_get_almost_3_gb_heap_on_windows.html

Arne
 
A

Alan Gutierrez

Peter said:
Arne said:
[...]
But out of the box, 32-bit Java is restricted to 2GB whether running on
32-bit Windows or 64-bit Windows.

The IMAGE_FILE_LARGE_ADDRESS_AWARE is not the real problem.

On 64-bit Windows, I think it is.
SUN Java requires contiguous heap space. And even with /3GB there
are something around the 2 GB marker.

The /3GB switch (that is, the flag set for Windows itself…this works
alongside 32-bit apps with the "large address aware" flag set) applies
only to 32-bit Windows.

On 64-bit Windows (where the /3GB switch is moot), I'm not aware of any
address space boundaries that would cause a problem.

Even on 32-bit Windows with /3GB in the boot.ini file, one may get more
memory for their Java process if they hack the Java .exe to have the
"large address aware" flag, because less of the first 2GB of virtual
address space is taken up by overhead (i.e. a lot of it can go in the
third GB of address space).

But you're right that on 32-bit Windows, a 32-bit process isn't going to
get a Java heap larger than 2GB (and likely not even as large as).

What I'm understanding from this is:

I can create a great many large MappedByteBuffers (~250MB) on 64 bit OS
X, SPARC and Linux, but this strategy is not going to work on 64 bit
Windows.

I'm also eager to hear someone destroy the idea entirely, to tell me,
"no you're not supposed to use `MappedByteBuffer` that way," or better
still, to say I'm on the right track if I'm willing to accept the
deployment limitations. A yeah or nay would suffice. Essentially, I
always see `MappedByteBuffer` hinted at for this sort of problem, but no
one has blogged or otherwise recorded their experiences with anything
other than reading a single file. (Maybe that is because 64 bits has
only become prevalent more recently?)
 
L

Lew

Alan said:
What I'm understanding from this is:

I can create a great many large MappedByteBuffers (~250MB) on 64 bit OS
X, SPARC and Linux, but this strategy is not going to work on 64 bit
Windows.

I don't think that's a problem if you use a 64-bit JVM. The restrictions
people have discussed here apply only to 32-bit JVMs.
 
K

Kevin McMurtrie

[QUOTE="Lew said:
Java 1.5 is more commonly a 32 bit process while Java 1.6 is more
commonly a 64 bit process.

How do you figure that?[/QUOTE]

Just an observation of which is more likely to launch by default when
there's no selector provided. It may depend on the Java bundle used
too. I count 5 primary bundles on Oracle's page. Java DB documents
also reference "Server SDK" and "GlassFish Server" bundles.
 
L

Lew

Kevin said:
Just an observation of which is more likely to launch by default when
there's no selector provided. It may depend on the Java bundle used
too. I count 5 primary bundles on Oracle's page. Java DB documents
also reference "Server SDK" and "GlassFish Server" bundles.

Whichever one launches by default depends on the one installed. You must have
observed 32-bit Java 5 installations and 64-bit Java 6 installations, but it's
always dangerous to infer universality from a small non-random sample.

It would be just as dangerous for me to conclude that Java 5 64-bit is more
common than Java 5 32-bit just because the large government projects that have
hired me over the last few years happen to use big iron.
 
M

markspace

Alan said:
I'm also eager to hear someone destroy the idea entirely, to tell me,
"no you're not supposed to use `MappedByteBuffer` that way,"


I think you're well on your way to becoming the expert on memory mapped
B-trees around here. ;)

I don't have a lot of experience using memory mapped files like this.
What I would question is why you want to do this at all. Why not just
use JDB or SQLite and not implement your own B-trees at all? I assume
you must have your reasons.

I think I'd like to see the difference in performance between hand
rolled B-trees and JDB. Should be useful to know if the extra effort
results in a worthwhile performance gain, or if JDB is fast enough as-is.
 
A

Alan Gutierrez

markspace said:
I think you're well on your way to becoming the expert on memory mapped
B-trees around here. ;)

Green light then. Trial and error, here I come.
I don't have a lot of experience using memory mapped files like this.
What I would question is why you want to do this at all. Why not just
use JDB or SQLite and not implement your own B-trees at all? I assume
you must have your reasons.

Reasons? Why, of course I have my reasons! I'm building a Skyscraper to
the Moon!

For this I will need a write ahead log, a B+tree, and, eventually, a
Paxos implementation.

And by Skyscraper to the Moon, I mean, a personal project that seems
absurdly ambitious, even to myself, that only I could possibly care
about, especially at this stage.

I'm asserting my freedom to tinker. I find the topic interesting. I've
got a web application I'd like to build. Its not a secret, but the
details are off topic. The application is an end to justify the means.
The means is a database assembled from database primitives. The
motivation is that I've come to suspect the value of the myriad of
abstraction layers I've come employ in my Java programming, especially
for solo programmer projects.
I think I'd like to see the difference in performance between hand
rolled B-trees and JDB. Should be useful to know if the extra effort
results in a worthwhile performance gain, or if JDB is fast enough as-is.

When I work with Hibernate, the notion that Hibernate abstracts the
database so I don't have to think about it is something that I'm
subscribing to in order to humor Hibernate, to make Hibernate feel like
my super best helper, when in my mind, I know that I've had to
de-normalize, time, tune, index and configure, configure, configure to
get the performance I seek. I'm very conscious of each index, and the
strategy for each query, and even which indexes are written to which
disk, and which disk hosts my write ahead log.

Actually, Hibernate *is* my super best helper, but I don't consider it
an abstraction. Everything I do with Hibernate is very deliberate.

In fact, Hibernate has talked me out of pessimistic concurrency
controls, so I've begun to structure my web applications with very
deliberate concurrency controls, optimistic locking that catches
concurrency errors and retries operations, usually in worker threads,
with some form of queuing, so I can reduce collisions. The Hibernate
community favors optimistic concurrency control, and now so do I.

Recently, I set out to implement incremental backups on some MySQL
databases that had gotten rather large for to dump whole. The
experience, not yet complete, of implementing incremental backups on
INNODB was the last straw. It shook my confidence right down the point
where I thought, if the only thing keeping me pursuing my interest in
distributed databases is a fear that I won't get durability right, well,
here I am learning about the inner workings of INNODB to get durability
right.

Thus, when I work with Hibernate, the database is in my head, from
concurrency, to schema, to indexes, to the backups. My experiment is to
see if a rugged write ahead log and generic B+tree is *less* effort than
all this.

I sense an opportunity for arbitrage.

If you nose around my GitHub account (bigeasy) you'll find it with its
MIT license, but its not setup for distribution yet, and I'm not
flogging it here, and never will. There are some incredible open source
projects producing powerful alternatives to ORM+RDBMS, to which I could
contribute, but that's not what I want to do.

I want to build a Skyscraper to the Moon.
 
E

Esmond Pitt

Green light then. Trial and error, here I come.

The Lucene project found about a 20% speed improvement using mapped byte
buffers for reading Lucene indexes.

They couldn't use it for output because you have to predetermine the
size, so you can't extend the file.

One *major* gotcha you need to be aware of is that there is no GC of the
address space of a MappedByteBuffer, so you can't just open them
willy-nilly. As in the Lucene case they are best used when you have a
known number of files of known size.
 
M

markspace

Esmond said:
One *major* gotcha you need to be aware of is that there is no GC of the
address space of a MappedByteBuffer, so you can't just open them
willy-nilly. As in the Lucene case they are best used when you have a
known number of files of known size.


So, if a memory mapped file is closed, its address space is never
reclaimed? That seems really wrong....
 
E

Esmond Pitt

So, if a memory mapped file is closed, its address space is never
reclaimed? That seems really wrong....

I told you it was a gotcha. See the Bug Parade, endless discussion about it.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top