java nio, memory mapping limits

[shahbazc]

I'm writing a prototype of a database management system (I know plenty
exist, I'm writing one anyway ), so I've decided to use NIO (jdk
1.5). I have two questions regarding memory mapping a file:

1. Is there anything wrong with mmaping a HUGE file (the complete limit
allowed in 32 or 64 bit systems)? For 32 bit systems, I'll just break
up my table among several files and map all of those.

2. Is there a limit to how many files I can have mapped at once (what
if I have 500 tables, all of which have a file channel pointing to
them)...I seem to recall that there was a OS imposed limit.

Basically I want to let the OS manage memory for me. I'm not really
concerned about issues of compatibility accross platforms (although I
would appreciate reading about caveats for windows, linux, sun, ibm,
etc.) I also realize that os managed memory will not be as effecient
as what I could do myself...but I very quickly get a working
prototype...so I don't care if it is not the absolute best solution.

I haven't read anything comprehensive regarding memory mapping using
Java's NIO. I suspect there is more material in OS or C/C++ books, if
someone has any good pointers, I'll appreciate getting them. Thanks!

-Falcon

 

[PRadyut]

if u get any clue pls let me know
I'm also interested.
Pointers or memory management in raw form are not used in database
management.

thanks

Pradyut
http://pradyut.tk
http://groups.yahoo.com/group/d_dom/
http://groups-beta.google.com/group/oop_programming
India

 

[John C. Bollinger]

I'm writing a prototype of a database management system (I know plenty
exist, I'm writing one anyway ), so I've decided to use NIO (jdk
1.5). I have two questions regarding memory mapping a file:

1. Is there anything wrong with mmaping a HUGE file (the complete limit
allowed in 32 or 64 bit systems)? For 32 bit systems, I'll just break
up my table among several files and map all of those.

2. Is there a limit to how many files I can have mapped at once (what
if I have 500 tables, all of which have a file channel pointing to
them)...I seem to recall that there was a OS imposed limit.

From the 1.4 docs (unlikely to have changed much in 1.5):

"Many of the details of memory-mapped files are inherently dependent
upon the underlying operating system and are therefore unspecified. The
behavior of this method when the requested region is not completely
contained within this channel's file is unspecified. Whether changes
made to the content or size of the underlying file, by this program or
another, are propagated to the buffer is unspecified. The rate at which
changes to the buffer are propagated to the file is unspecified."

Basically I want to let the OS manage memory for me.

Are you planning to use the mapped file as a big memory pool, then?
That's not pushing memory management onto the OS at all -- it's making
your program take on the task itself. What's wrong with letting Java's
built-in memory management do the job for you instead?

I'm not really
concerned about issues of compatibility accross platforms (although I
would appreciate reading about caveats for windows, linux, sun, ibm,
etc.)

You should be. Especially when you can get it cheaply by using what
comes free and without special effort as a built-in part of Java.

I also realize that os managed memory will not be as effecient
as what I could do myself...but I very quickly get a working
prototype...so I don't care if it is not the absolute best solution.

If the time to produce a working prototype is the driving issue then why
are you trying to make the prototype more complicated than it needs to be?

I haven't read anything comprehensive regarding memory mapping using
Java's NIO. I suspect there is more material in OS or C/C++ books, if
someone has any good pointers, I'll appreciate getting them. Thanks!

As the NIO docs say, the details are highly OS-dependent. You would
want to look at the docs for the target OS' system-level I/O interfaces.
(For instance, the C mmap() function or equivalent.) But really,
forget about this until you actually need it, which, for the purpose of
general memory management, will be never.

 

[shahbazc]

John,
I'm actually not using a memory mapped file as a single pool. I'm
wondering if there is a problem with mmaping every disk based table in
the database. In other words, I want to be able to do
insertions/deletions/query on a filechannel without having to worry
about keeping the memory data structure in sync with the disk file
(with an assumption that the server will not crash...good enough
assumption for the current prototype). Now these files may end up
being hundreds of megs larger than my system memory.

I've already started reading docs for mmap (beginning with 'man mmap').
Basically I'm hoping that memory mapped files will allow me to produce
a more functional prototype, in a limited amount of time.

 

[John C. Bollinger]

I'm actually not using a memory mapped file as a single pool. I'm
wondering if there is a problem with mmaping every disk based table in
the database. In other words, I want to be able to do
insertions/deletions/query on a filechannel without having to worry
about keeping the memory data structure in sync with the disk file
(with an assumption that the server will not crash...good enough
assumption for the current prototype). Now these files may end up
being hundreds of megs larger than my system memory.

In that case, what you are describing is not "memory management" in the
sense that I usually use the term. You seem more to be describing file
manipulation tasks, for which you are seeking a convenient interface.
In that case, I think you will find RandomAccessFile to be more suited
to your needs, at least inasmuch as its behavior and limitations are
better defined than those of a memory-mapped ByteBuffer.

 

[Cindi]

http://blogs.ittoolbox.com/eai/leadership/archives/003046.asp