how to solve this problem : out of memory

[Simon Brooke]

Lew wrote:

[me:]

But it's not a requirement (of the OS) that the user should have to
tell it how much memory the app is going to use in advance. The JVM's
inflexibility in this is somewhat puzzling -- I have no other
implementations of GC-ed languages which share this odd
requirement[*].

I have watched the 1.4 JVM grab memory until it uses well over 1GB,
absent
setting a limit with -Xmx. That indicates to me that the JVM will take
*all* available memory by default.

If it did work like that then it would never be possible cure an
out-of-memory condition by setting an explicit max -- only to create one.

BTW, the amount of memory the OS reports the JVM to be using is not
particularly closely tied to how much it is actually using (and
especially not to how much it is using for actual objects), and is even
less tightly tied to the min/max parameters (whether the defaults or set
explicitly).

However, and more seriously, if you are hitting out of memory problems you
need to understand why. If something in your application is chewing up
memory, then ultimately however much memory you allocate it is going to
run out. Some programs - or their working data sets - simply are large,
and that's OK. But as long as something in your application is holding a
pointer to a piece of data the garbage collector cannot collect it, so,
e.g., when building complex persistent recursive structures it's important
to remember to set null things you no longer need.

 

[Simon Brooke]

Why?

I'm with you, Patricia. If you're holding onto a data structure you no
longer need, the quickest way of advertising to the garbage collector that
you don't need it any more is simply to set the variable to null. This is
particularly true in recursive data structures.

 

[Thomas Kellerer]

Lew wrote on 03.10.2006 14:01:

But it's not a requirement (of the OS) that the user should have to
tell it how
much memory the app is going to use in advance. The JVM's
inflexibility in
this is somewhat puzzling -- I have no other implementations of GC-ed
languages
which share this odd requirement[*].

I have watched the 1.4 JVM grab memory until it uses well over 1GB,
absent setting a limit with -Xmx. That indicates to me that the JVM
will take *all* available memory by default.

The -Xmx parameter only restricts the *heap* memory available to the Java
program. It does not restrict the overall memory usage of the JVM which is more
than the heap. The additional memory that the JVM allocates is used to store
class definitions and compiled bytecode. Using reflection increases this memory
usage even more (which can be controlled to a certain extend with a JVM parameter)

The additional memory is used for holding class definitions and compiled
bytecode, which can never be restricted by a JVM parameter. The more classes you
have and if you use reflection very intensively you will see the memory grow
even though your application is not using a big deal of heap memory.

Thomas

 

[Tor Iver Wilhelmsen]

But it's not a requirement (of the OS) that the user should have to
tell it how much memory the app is going to use in advance. The
JVM's inflexibility in this is somewhat puzzling -- I have no other
implementations of GC-ed languages which share this odd
requirement[*].

This is a virtual machine, so telling the system how much memory to
allocate for the heap is akin to the motherboard telling the OS how
much RAM there is.

 

[Chris Uppal]

Tor Iver Wilhelmsen wrote:

[me:]

But it's not a requirement (of the OS) that the user should have to
tell it how much memory the app is going to use in advance. The
JVM's inflexibility in this is somewhat puzzling -- I have no other
implementations of GC-ed languages which share this odd
requirement[*].

This is a virtual machine, so telling the system how much memory to
allocate for the heap is akin to the motherboard telling the OS how
much RAM there is.

A false analogy, IMO.

This is a /program/ -- and one which is sophisticated about memory allocation
at that (at the OS level, I don't mean the GC algorithms). It is perfectly
capable (in principle) of adapting to dynamic memory requirements as they
change. There is nothing even theoretically wrong with allowing the user the
option of bounding its memory use -- that's the kind of thing one would expect
from a sophisticated program (to operate in tandem with the controls provided
to the user by the OS). But imposing that limit /as the default/, with no way
even to turn it off, seems more than just clunky to me.

-- chris

 

[=?ISO-8859-1?Q?Arne_Vajh=F8j?=]

This is a /program/ -- and one which is sophisticated about memory allocation
at that (at the OS level, I don't mean the GC algorithms). It is perfectly
capable (in principle) of adapting to dynamic memory requirements as they
change. There is nothing even theoretically wrong with allowing the user the
option of bounding its memory use -- that's the kind of thing one would expect
from a sophisticated program (to operate in tandem with the controls provided
to the user by the OS). But imposing that limit /as the default/, with no way
even to turn it off, seems more than just clunky to me.

Java has not wa of detecting was the users would
consider a smart distribution of memory on the system.

It is possible to de facto turn it off by setting
it to the maximum value.

We can discuss whether default should be as it is,
somewhat higher or maximum (=not set).

I think I would vote for somewhat higher like 256m, but ...

Arne

 

[Tim Ward]

Java has not wa of detecting was the users would
consider a smart distribution of memory on the system.

Funny how this isn't a problem with applications written in other languages.
I run all sorts of things on various machines, and only for those written in
Java do I have to worry about telling them how much memory to use - the
others just get on with doing their jobs.

 

[Thomas Kellerer]

Chris Uppal wrote on 05.10.2006 14:03:

A false analogy, IMO.

This is a /program/ -- and one which is sophisticated about memory allocation
at that (at the OS level, I don't mean the GC algorithms). It is perfectly
capable (in principle) of adapting to dynamic memory requirements as they
change. There is nothing even theoretically wrong with allowing the user the
option of bounding its memory use -- that's the kind of thing one would expect
from a sophisticated program (to operate in tandem with the controls provided
to the user by the OS). But imposing that limit /as the default/, with no way
even to turn it off, seems more than just clunky to me.

I agree.

The JVM should not restrict the memory in any way unless the user explicitely
asks for it. After all the JVM already allocates as much memory as necessary for
class definitions. But then again limits something like the "Permanent size"
which aparently stores metainformation for reflection. Which cost us about three
days to hunt down an OutOfMemoryError in a huge BEA based portal, because the
reported free heap was large enought.

I think the other way round should be the default: if the user wants to limit
the memory it should be possible, but "out-of-the-box" the JVM should use as
much as available (just like any other program)

But I doubt that Sun will ever change that

Regards
Thomas

 

[Chris Uppal]

Java has not wa of detecting was the users would
consider a smart distribution of memory on the system.

All other systems I know of work on the assumption that if the user hasn't told
them not to, then it's OK to use as much memory as the OS will let them have
(as long as they aren't greedy ;-)

The scheme implemented by the Sun JVMs is neither so flexible, nor so
convenient. And that is -- in my opinion -- one of the deployment "niggles"
which have hindered Java's use on the desktop.

-- chris

 

[=?ISO-8859-1?Q?Arne_Vajh=F8j?=]

Funny how this isn't a problem with applications written in other languages.
I run all sorts of things on various machines, and only for those written in
Java do I have to worry about telling them how much memory to use - the
others just get on with doing their jobs.

It is a problem with other languages. Why do you think
it is general recommended not to run IIS and SQLServer
on the same server ? They can not find out how to divide
memory properly. If they were written in Java that would
not be a problem.

Arne

 

[=?ISO-8859-1?Q?Arne_Vajh=F8j?=]

All other systems I know of work on the assumption that if the user hasn't told
them not to, then it's OK to use as much memory as the OS will let them have
(as long as they aren't greedy ;-)

The scheme implemented by the Sun JVMs is neither so flexible, nor so
convenient. And that is -- in my opinion -- one of the deployment "niggles"
which have hindered Java's use on the desktop.

Java's way is the safe way.

But yes it can sometimes be inconvenient.

I think they could lift some of the inconvenience by increasing
the default so that most desktop apps could run with it.

Arne