Java memory management question

[Chris Ott]

Hi,

We're working on a Java application at my company and we're having some
memory issues. Let me point out first: I am not a Java programmer, myself,
but I've programmed in lots of other languages, so I understand things like
memory allocation, heaps, stacks, etc.

Unlike most Java applications (with which I'm familiar), ours doesn't have a
GUI. It just sits in the background, listening on a TCP socket. Other
programs connect on that socket and give the application a chunk of data,
which it processes and adds to our database. The application allocates the
memory it needs to process the dataset, then (supposedly) releases the
memory when it's done.

Normally, the datasets are in the 1-5 meg range but, every so often,
there'll be a huge dataset that causes an enormous increase in Java's
memory usage. We've seen it get up over 1/2 a gig. This isn't a problem,
short-term, but Java never releases that memory back to the OS. This is
true for all VMs we've tried: Tru64, AIX, Linux, and Windows.

The Java programmers claim they're releasing their objects properly and I'd
be inclined to agree because, as long as the application doesn't receive
any more large datasets, Java won't allocate any more memory. They tell me
the only way to free the memory is to restart the VM, but we'd rather not
do that, since this is a server-side application that's supposed to be
running 24/7.

I find it hard to believe that memory management for a language as popular
as Java could be - well - that broken, so I'm feeling the need for some
verification. Anyone know what's going on here? At this point, I'm
beginning to wonder if it was even appropriate to use Java for a
server-side application.

Thanks,
Chris Ott
<first initial, last name at acclamation dot com>

 

[Gordon Beaton]

The application allocates the memory it needs to process the
dataset, then (supposedly) releases the memory when it's done.

Normally, the datasets are in the 1-5 meg range but, every so often,
there'll be a huge dataset that causes an enormous increase in
Java's memory usage. We've seen it get up over 1/2 a gig. This isn't
a problem, short-term, but Java never releases that memory back to
the OS. This is true for all VMs we've tried: Tru64, AIX, Linux, and
Windows.

This is typical and expected behaviour. It isn't related to Java per
se, rather to the way the underlying allocation mechanism works that
the JVM uses.

Java has most likely released the memory associated with your large
objects. However as a rule, malloc libraries don't release the freed
memory back to the operating system since doing so would complicate
and slow down the allocation mechanism (such versions do exist
though). Instead, the memory stays part of the process and is
available for future calls to malloc(). Until then it's paged out and
doesn't occupy any physical memory.

If this truly is a problem, one way your application could deal with
it would be to create a temporary child process (with Runtime.exec())
to handle with the huge dataset. When the child exits, all of its
memory is released back to the OS, and the main process avoids growing
for the exceptional case.

/gordon

 

[Chris Uppal]

If this truly is a problem, one way your application could deal with
it would be to create a temporary child process (with Runtime.exec())
to handle with the huge dataset.

Or another way would be to hold the dataset in memory allocated directly from
the OS via JNI.

A lot of messing around, though unless the data is particularly simple.

Why not just restart the server automatically from time to time ?

Come to that, why is 1/2 Gig a problem at all, it's not as if its growing
beyond that ?

-- chris

 

[Tim Jowers]

Or another way would be to hold the dataset in memory allocated directly from
the OS via JNI.

A lot of messing around, though unless the data is particularly simple.

Why not just restart the server automatically from time to time ?

Come to that, why is 1/2 Gig a problem at all, it's not as if its growing
beyond that ?

-- chris

You guys had me scared. Just call System.gc() twice.
ref: http://www-106.ibm.com/developerwor...0656&forum=177&cat=10&message=1495457#1495457

TimJowers

tested with 1.4.2_03, java -cp . -Xmx256m -verbosegc
-XX:+PrintGCDetails LargeAlloc, and with the -XX:+AggressiveHeap
option as well


/**
* Created on Feb 20, 2004
* @author tjowers
*
* Simple test of news net posting on poor Java memory handling.
*

http://groups.google.com/[email protected]&prev=/groups?

q%3Dcomp.lang.java.programmer%26ie%3DUTF-8%26oe%3DUTF-8%26hl%3Den
*/
import java.util.*;

public class LargeAlloc {


public static void main(String[] args) {
try{ Thread.sleep( 7000 );}catch(Exception e){} // sleep enough time
to start perfmon etc.
ArrayList al = new ArrayList();
int cntObjects=64;
int OneMB = (int)Math.pow(2D,20D);
while( cntObjects-- > 0 )
{ System.out.print( cntObjects );
try{ Thread.sleep( 100 );}catch(Exception e){} // sleep to allow OS
to chart memory usage

so we can see it on the chart
if( 0 == cntObjects%16 ) // 48,32,16
{ // allocate a large object, say 32MB
byte b[] = new byte[32 * OneMB ];
al.add( b );
} else {// allocate normal, small blocks
byte b[] = new byte[OneMB ];
al.add( b );
}
}
System.out.println( "Determine that memory is allocated to the
process" );
try{ Thread.sleep( 7000 );}catch(Exception e){} // sleep a few
seconds, see perf tool
// now we can force the GC or could drop all references and do other
processing and wait for GC
// let's force the garbage collection to prove Java will release the
memory.
al = null;
System.out.println( "Validate that memory is NOT released from the
process" );
for( int tries=12; tries-->0; )
{ System.out.println( "try " + tries );
System.gc();
try{ Thread.sleep( 1000 );}catch(Exception e){} // sleep a second
}
// done and memory should have been released
// run with -verbosegc to see memory management
// use OS's process/memory management tools to observe
//

http://www-106.ibm.com/developerwor...=30656&forum=177&cat=10&message=1495457#14954

57
}
}

 

[Chris Smith]

The Java programmers claim they're releasing their objects properly and I'd
be inclined to agree because, as long as the application doesn't receive
any more large datasets, Java won't allocate any more memory.

This statement has got me a little worried. If a second large data set
causes the heap to grow by another bound, then you've got an application
problem. If you only mean that a second *even larger* data set causes
the heap to grow further, then read Gordon's reply.

--
www.designacourse.com
The Easiest Way to Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation

 

[Doug Pardee]

Normally, the datasets are in the 1-5 meg range but, every so often,
there'll be a huge dataset that causes an enormous increase in Java's
memory usage. We've seen it get up over 1/2 a gig. This isn't a problem,
short-term, but Java never releases that memory back to the OS. This is
true for all VMs we've tried: Tru64, AIX, Linux, and Windows.

Most of the time we can ignore the details of the Java memory
management and garbage collection systems. However, when you have
memory usage spikes of half a gig, and at the same time need to keep
the memory usage down, you're going to have to dive in and learn how
your particular JVM manages memory, and then play with whatever
adjustments that are available.

The following assumes that you're using a Sun 1.4.2 JVM...

Sun's tuning document: http://java.sun.com/docs/hotspot/gc1.4.2/
Sun's JVM options: http://java.sun.com/docs/hotspot/VMOptions.html
Sun's GC FAQ: http://java.sun.com/docs/hotspot/gc1.4.2/faq.html

Notice section 3.1 in the tuning document:

"By default, the virtual machine grows or shrinks the heap at each
collection to try to keep the proportion of free space to live objects
at each collection within a specific range. This target range is set
as a percentage by the parameters -XX:MinHeapFreeRatio=<minimum> and
-XX:MaxHeapFreeRatio=<maximum>, and the total size is bounded below by
-Xms and above by -Xmx ."

So, we see that the JVM *is* capable of releasing unused memory back
to the operating system. That will not happen until the next time that
garbage collection is triggered for the generation, which will be some
time after the memory being freed is no longer in use by the Java
program. The amount that the JVM will release is controlled by the JVM
parameters indicated.

Since you're not seeing this happen, something is up.

I could make a guess: the huge block of memory is way too large for
the Young generation, and so is allocated in (or quickly promoted to)
the tenured generation. But not much else is getting moved to the
tenured generation, so an "old generation" collection isn't triggered
until the *next* time that you need a huge amount of memory. Your JVM
memory usage might be dropping for a short time after the collection,
but going right back up again as the new giant dataset comes in.

Another possibility (not really likely, since you don't seem to have
messed with the GC options) is that you've selected the
Concurrent-Mark-and-Sweep (CMS) collector for the old generation. That
collector doesn't compact unless a full collection is triggered, which
should be rare if it ever happens at all. YOu probably won't see
memory being returned to the operating system from the old generation
if you use the CMS collector.

Speculation about what's happening is a poor way to go, though. What
you need to do is to add some JVM GC display options and run your
application, then analyze the GC behavior to see when GCs are running
and what they're accomplishing (or not accomplishing). Then you can
start to play with options to tune your garbage collectors to get the
results that you want.

 

[Doug Pardee]

If a second large data set causes the heap to grow by another
bound, then you've got an application problem.

Probably, but not necessarily.

1. Memory is not reclaimed at the time that objects are "freed", but
when a garbage collection cycle is run after the objects have become
unreachable. If no GC cycle has been triggered, heap usage will
continue to climb as new objects are created.

2. Even if a GC cycle has been triggered, the Java Language
Specification and the JVM Specification do not require "accurate"
garbage collection, where all of the memory used by unreachable
objects is reclaimed (some folks argue that the specs do not require
garbage collection at all). Early Sun JVMs did not have accurate
garbage collection. In Sun's most recent JVMs, the optional CMS
(Concurrent Mark and Sweep) collector for the old generation is
effectively inaccurate because no compaction is done unless the JVM is
forced to do a "stop the world" full collection, something that is
usually avoided at all costs in any system that needs the reduced
pause time of CMS collection.

3. In a generational system such as in Sun's modern JVMs, memory usage
is independent between the generations. Reclaiming a chunk of old
generation memory does not make more room in the young generation, and
vice versa.