-Xmx out of memory errors

[catphive]

That java gets out of memory errors, and that the maximum heap size
needs to be controlled by a flag has always bugged me. Why is this so?
Most systems already have mechanisms in place to set hard limits
(getrlimit, setrlimit) on resource usage, and have mechanisms to
increase the number of pages available for memory management (brk,
sbrk). Why doesn't java just rely on these instead of putting extra
hard limits on memory?

Also, is it at all possible to turn this behavior off? I find that
many java programs eventually run into and crash on their default max
heap size. It would be nice to have a way to either tell individual
java instances, or preferably all java programs to have no maximum
heap size except that defined by rlimit.

Thanks

 

[=?ISO-8859-1?Q?Arne_Vajh=F8j?=]

That java gets out of memory errors, and that the maximum heap size
needs to be controlled by a flag has always bugged me. Why is this so?
Most systems already have mechanisms in place to set hard limits
(getrlimit, setrlimit) on resource usage, and have mechanisms to
increase the number of pages available for memory management (brk,
sbrk). Why doesn't java just rely on these instead of putting extra
hard limits on memory?

Also, is it at all possible to turn this behavior off? I find that
many java programs eventually run into and crash on their default max
heap size. It would be nice to have a way to either tell individual
java instances, or preferably all java programs to have no maximum
heap size except that defined by rlimit.

Java is intended to be cross platform. So it makes some sense to
control something that is not guaranteed to be part of all deployment
OS'es.

You can use the maximum supported value as "unlimited". They could
also have defined that -Xmx0 meant maximum, but there are a certain
documentative effect of having the actual number, because it is
not unlimited.

Arne

 

[catphive]

Java is intended to be cross platform. So it makes some sense to
control something that is not guaranteed to be part of all deployment
OS'es.

You can use the maximum supported value as "unlimited". They could
also have defined that -Xmx0 meant maximum, but there are a certain
documentative effect of having the actual number, because it is
not unlimited.

Arne

unlimited eh? Thanks.

 

[catphive]

Java is intended to be cross platform. So it makes some sense to
control something that is not guaranteed to be part of all deployment
OS'es.

You can use the maximum supported value as "unlimited". They could
also have defined that -Xmx0 meant maximum, but there are a certain
documentative effect of having the actual number, because it is
not unlimited.

Arne

actually, upon checking unlimited doesn't seem to be acceptable...
this again raises the problem with -Xmx, that it forces me to put a
hard limit on a dynamically allocated resource...

Now that you mention it, old mac os 9 used to require a max heap size
set for every application (get info, maximum memory). However, that
was a horribly primitive system. I understand that they'd want mac
compatibility, but it seems unfortunate that they did it in such a way
that made everyone else suffer... I'd like my programs not to die
every time they get an unexpectedly large input. However high I set
the -Xmx value it's always possible that more heap space will be
necessary.

Given virtual memory, and the fact the fact that on a reasonable
application only a small subset of memory need to be accessible in
ram, thrashing isn't really a problem, there's really no practical
limit to a maximum heap size.

 

[Chris Dollin]

Given virtual memory, and the fact the fact that on a reasonable
application only a small subset of memory need to be accessible in
ram, thrashing isn't really a problem, there's really no practical
limit to a maximum heap size.

Those who have experienced systems thrashing may disagree.

--
"We did not have time to find out everything /A Clash of Cymbals/
we wanted to know." - James Blish

Hewlett-Packard Limited registered no:
registered office: Cain Road, Bracknell, Berks RG12 1HN 690597 England

 

[=?ISO-8859-1?Q?Arne_Vajh=F8j?=]

actually, upon checking unlimited doesn't seem to be acceptable...
this again raises the problem with -Xmx, that it forces me to put a
hard limit on a dynamically allocated resource...

Given virtual memory, and the fact the fact that on a reasonable
application only a small subset of memory need to be accessible in
ram, thrashing isn't really a problem, there's really no practical
limit to a maximum heap size.

????

The highest limit you can set with Xmx is the limit imposed
by the limits in the virtual address space.

Java can not get more from the operating system.

Arne

 

[Stan]

One thing to consider, the larger the heap the harder the java system
gets hit when it needs to do a garbage collection. I know it sounds
crazy but some systems actually behave more efficiently with smaller
heaps. Most people tune their java apps and monitor the gc behavior
with he verbose gc parameter. It will let you monitor how often a gc
event occurs, and how long each gc event takes. Usually while a gc
event occurs all threads may stop working.

Other thing to consider is that java uses more then just heap memory.
It also uses system memory from jni calls. You usually need to balance
both. Having the ability to control the max and min size is not
necessarily a bad thing.

Stan

 

[catphive]

The issue is maximum heap... just because the heap is bounded to a
certain volume, or unbounded doesn't mean that java needs to sbrk that
much heap space. Making the default unbounded, which is all I would
like to see change, wouldn't mean that java by default sbrks the whole
address space, and if it does, that's a java implementation problem
that shouldn't be there.

I don't see how -Xmx is useful for balancing "system memory" (by which
I think you mean memory allocated by malloc... which is also on the
heap btw, just not the gc'd section). As near as I can tell
applications pretty much just die when they run into the -Xmx barrier.
I guess it probably throws an exception which could be caught; however
it seems unlikely that such an exception could be recovered from in a
meaningful way given the number of places that could throw such an
exception.

Someone else mentioned that I could make -Xmx the size of the entire
address space; however I would note that isn't portable, as for 32 bit
systems that is merely 4 gigs, whereas 64 bit machines can handle a
lot more depending on the vm's implementation. The problem is that
right now, java users who know nothing about java implementation
details have to go in and mess with the java launching script to make
their java applications work, and they would still have to do that
occasionally if the script writer guessed a common address space size.
It's not the correct solution.

Someone else mentioned that thrashing can happen... but that is the
result of a poorly written program or gc, not a high -Xmx value.
Extremely large heaps can not thrash, and significantly smaller heaps
can depending on memory access patterns.

In conclussion -Xmx seems like a total hack to me, and I'd like to see
it set to an unlimited value by default. It could still be there for
people who need it, and don't know how to do the exact same thing
through the shell with rlimit. The default action is just a nuisance
though.