Running out of memory using FileChannel.map()

P

Paul J. Lucas

I'm using FileChannel.map() to read through large image files.
I have a utility function to return a ByteBuffer from a file:

public static ByteBuffer map( File file ) throws IOException {
final FileInputStream fis = new FileInputStream( file );
final FileChannel fc = fis.getChannel();
try {
return fc.map(FileChannel.MapMode.READ_ONLY, 0, file.length());
}
finally {
fis.close();
}
}

So as not to keep the ByteBuffer around indefinitely, I have an
object that uses a WeakReference to it:

public synchronized ByteBuffer getByteBuffer() throws IOException {
ByteBuffer strongRef;
if ( m_weakRef == null ||
(strongRef = (ByteBuffer)m_weakRef.get()) == null )
{
strongRef = ByteBufferUtil.map( m_imageFile );
m_weakRef = new WeakReference( strongRef );
}
return strongRef;
}

And yet my program, after reading many image files, runs out of
memory. Assuming I don't hang on to the strong references to
the ByteBuffer by mistake elsewhere, is there anything wrong
with the above code? Is this the right way to used mapped
ByteBuffers?

- Paul
 
A

Andrew Thompson

And yet my program, after reading many image files, runs out of
memory.

...images? Images are notorious for being cached.

A couple of requests..
- please don't indent text to usenet unless there is
a specific purpose for it, such as code formatting.
- please do not post tab characters to usenet, they are
often expanded to ridiculous widths by news clients.
 
P

Paul J. Lucas

Andrew Thompson said:
..images? Images are notorious for being cached.

The just so happen to be image files. As far as the code is
concerned, they're just a bag of bytes. The files are not being
opened *as* images.
A couple of requests..
- please don't indent text to usenet unless there is
a specific purpose for it, such as code formatting.
- please do not post tab characters to usenet, they are
often expanded to ridiculous widths by news clients.

Get better clients.

- Paul
 
T

Thomas Hawtin

Paul said:
And yet my program, after reading many image files, runs out of
memory. Assuming I don't hang on to the strong references to
the ByteBuffer by mistake elsewhere, is there anything wrong
with the above code? Is this the right way to used mapped
ByteBuffers?

Do you know what's in memory and how full it is when you get the exception?

I'm guess mapped files behave much the same as direct allocated buffers.

The mechanism of garbage collecting direct allocated buffers is not
straightforward as they are not part of the heap area. There is some NIO
code to request a GC, but it's a bit of a hack. You can get an
OutOfMemoryError if memory used by direct buffers exceed a limit (the
way the default is calculated is version dependent).

Tom Hawtin
 
A

Andrew Thompson

Get better clients.

get plonked.

--
Andrew Thompson
physci.org 1point1c.org javasaver.com lensescapes.com athompson.info
"..I pick up all the pieces and make an island. Might even raise just a
little sand"
Jimi Hendrix 'Voodoo Chile (Slight Return)'
 
J

jan V

Hi Paul, do yourself (or your company) a favour: invest in a profiler.
Memory problems are tackled so powerfully by looking at the detailed state
of your heap as your program runs.

Your approach should be to look at the profiler's output and views, and then
see whether that new knowledge fits totally with what you would expect from
your program. The inevitable mismatch is the path to your solution...
 
P

Paul J. Lucas

Thomas Hawtin said:
Do you know what's in memory and how full it is when you get the exception?

No. I first wanted to see if there was anything wrong with my
code, i.e., method of dealing with MappedByteBuffers.
The mechanism of garbage collecting direct allocated buffers is not
straightforward as they are not part of the heap area. There is some NIO
code to request a GC, but it's a bit of a hack.

What is it?

- Paul
 
A

Antti S. Brax

A couple of requests..
- please don't indent text to usenet unless there is
a specific purpose for it, such as code formatting.
- please do not post tab characters to usenet, they are
often expanded to ridiculous widths by news clients.

Please provide references that indicate that these requests
follow a common practise.
 
C

Chris Uppal

Antti said:
Please provide references that indicate that these requests
follow a common practise.

I don't know what percentage of Usenet readers use Outlook Express, but I think
the number is probably pretty high. OE does not render tabs in postings at
all. It's up to the poster to decide whether it's worth the effort of
"de-tabbing", but if they want people (like me for instance) to read the code
they post, then its worth knowing that using tabs for indentation makes it
harder for a (substantial?) portion of the reading population.

Oddly, the two problems cancelled out in the OPs post. He'd (unecessarily)
indented the whole post with tabs, and then further indented the real code with
spaces. The result was that it was formatted "correctly" when viewd in OE ;-)

As for the OP's question. I haven't used the NIO stuff myself, so the fact
that I don't see anything wrong with the code is probably not all that helpful
;-) However, I do remember that the buffers are only cleaned up by GC, even
though the resource they represent is scarcer than heap-allocated memory (a
/bad/ design error). If an attempt to allocate a buffer fails (the OS
refuses), then (IIRC) that is reported as an out-of-memory exception, even
though the JVM may have plenty of memory on hand for allocating "real" objects.
The result is that you can run out of (OS-level) buffers without the GC
realising that it has to wake up and do some work (which, if it did it, would
have a chance to release unused buffers back to the OS). It might be
sufficient (for your purposes) to force a GC from time to time, or whenever the
buffer creation fails. But that -- of course -- is not guaranteed to work.

-- chris
 
E

E.J. Pitt

Paul said:
So as not to keep the ByteBuffer around indefinitely, I have an
object that uses a WeakReference to it:

I don't understand the benefit of this, as opposed to just letting GC
work the way it always works when the rest of your code releases all its
references to the buffer. This way, you run the risk of multiple very
large buffers existing at the same time, which is worse than the problem
you are trying to cure.
 
P

Paul J. Lucas

E.J. Pitt said:
I don't understand the benefit of this, as opposed to just letting GC
work the way it always works when the rest of your code releases all its
references to the buffer.

There's an encapsulating object that may be held onto for a
long time (that contains an aggregation of other information as
well) whereas the large buffer that's being encapsulated is
used only initially. Hence, there's no reason to keep the large
buffer around.
This way, you run the risk of multiple very large buffers existing at the
same time, which is worse than the problem you are trying to cure.

That will never happen: when a WeakReference is cleared, it
means the thing referred to is going to be reclaimed. There
will never be several of them around.

If you were right, then there would be no point to using
WeakReferences ever.

- Paul
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top