many files and NIO

[Ethan Metsger]

Hello, all. I'm new to the group, but I've been doing Java programming for a
couple years fairly consistently.

First, the specs: I'm running a Fedora Core 3 Linux system with JRE 1.4.2
(Athlon XP 1800+/512MB RAM).


I am working on a research project for my master's thesis which requires that
I load several seconds of video into memory for processing. (We're attempting
to track human torsos.) Each frame is stored on disk as a PPM (for ease of
use and portability purposes). I am using binary PPMs, so there is some data
compression, but nothing terribly significant, so they consume around 250kB
apiece. My test data set is 110 frames (about 25MB). This is a small data
set, representing around 3.6s of video. I would not expect loading 110 PPM
objects to stress the JVM or the system.

However, I get several java.lang.OutOfMemory errors when attempting to load
them all together. At present, I am trying to optimize the choice of the
torso segment, so it is possible for me to look at each frame individually and
see if I have acquired a better choice this time around. I have threaded this
process using a thread pool (five threads, 200ms poll). Each thread also
spawns another Process (which blocks the current thread execution).

Unfortunately, the time will come when I will need to have the pixmap data
handy. I could reread it, but this would incur a performance hit that I'm not
really willing to take unless I absolutely have to. I have done some reading
on NIO, particularly using MappedByteBuffers, which proponents say can
increase performance of reads and writes.

But I am particularly interested in their properties of memory management.
Since a MappedByteBuffer allocates memory on the native heap instead of the
JVM, will it clear up this problem? (Obviously, others may arise.) Can I use
a MappedByteBuffer to improve not only R/W performance, but also memory issues
related to reading in data?

If you could, please cc: me along with any replies to the newsgroup.

Sincerely,

Ethan Metsger

 

[Stefan Schulz]

But I am particularly interested in their properties of memory
management. Since a MappedByteBuffer allocates memory on the native heap
instead of the JVM, will it clear up this problem? (Obviously, others
may arise.) Can I use a MappedByteBuffer to improve not only R/W
performance, but also memory issues related to reading in data?

These JVM options might be helpful:
-Xms<size> set initial Java heap size -Xmx<size> set
maximum Java heap size -Xss<size> set java thread stack size

Also, be sure you know how many times you buffer your data. Each buffer
eats memory, and thereby you can probably slice quite a bit of memory off
(at the cost of speed) by not buffering, or ideally re-use the same buffer
without making copies.

 

[Ethan Metsger]

These JVM options might be helpful:
-Xms<size> set initial Java heap size -Xmx<size> set
maximum Java heap size -Xss<size> set java thread stack size

Also, be sure you know how many times you buffer your data. Each buffer
eats memory, and thereby you can probably slice quite a bit of memory off
(at the cost of speed) by not buffering, or ideally re-use the same buffer
without making copies.

Stefan,

Thanks for the tips. I've played with some of the heap size settings, though
not extensively. I'll definitely do some more tweaking to see what happens.

Your second paragraph is somewhat intriguing. Are you recommending that I use
a single Reader to read in each file, rather than creating a new one for each
(which is what I do right now)? I assumed that the biggest consumer of
resources was not the actual buffer, but storing the pixmap in memory (an
array of RGB values).

Thanks again for your time; it's much appreciated!

-Ethan

 

[Stefan Schulz]

Your second paragraph is somewhat intriguing. Are you recommending that
I use a single Reader to read in each file, rather than creating a new
one for each (which is what I do right now)? I assumed that the biggest
consumer of resources was not the actual buffer, but storing the pixmap
in memory (an array of RGB values).

It was my first stab at the problem, yes. Since you explicitly mention
NIO, i looked at my experience, and saw that NIO "encourages" you to use
many Buffer objects.

Re-Using a Reader is nearly impossible, since your FileChannel is tied to
the Reader instance. However, if you are willing to spend a little more
time, don't use Channels.getReader(), but instead use one
ByteBuffer/CharBuffer pair you allocated yourself. Since the binary image
of your PPM files will likely be _much_ smaller then the on-disk version,
that should at least help reduce the memory footprint. It also has the
added benefit of enabling you to use the same buffer for each file (albeit
forcing you to read the files sequentially, or at least partially so).

Other then that, be sure to not leave references "hanging around" by
accident. I had a similar problem once, and found out that a Map that was
only needed in the startup of my application stayed for all time because
one of my helper objects retained a reference in a field. This is not to
say that you should explicitly null each and every reference you come
across (that is just overkill), but that you should take a good look at
your design, and decide if each field is really needed for the entire
lifetime of the object, or better passed as an argument for each
invocation.