limit java heap size to available physical memory

A

Alessandro

Hi all,

I have a very bug memory eater java application (it takes in input large csv
files with milions of records), running on a server with 32 GB ram.
I have set -Xms32m and -Xmx1024m and it seems good, anyway I can't forecast
how many records will be in csv files and I'm not so quiet.

Is it possibile to limit the max heap size to available physical memory or
anyway to max Windows allocation memory limit for a single process ?

I suppose that defining only -Xms I will have the standard heap size limit
(64MB).

Any idea ? Am I saying bad ?

Thanks and best regards,
Ale
 
A

Arne Vajhøj

Alessandro said:
I have a very bug memory eater java application (it takes in input large csv
files with milions of records), running on a server with 32 GB ram.
I have set -Xms32m and -Xmx1024m and it seems good, anyway I can't forecast
how many records will be in csv files and I'm not so quiet.

Is it possibile to limit the max heap size to available physical memory or
anyway to max Windows allocation memory limit for a single process ?

I suppose that defining only -Xms I will have the standard heap size limit
(64MB).

Any idea ?

If 32 bit OS then use:

-Xmx1650m

You will get OutOfMemory if that is exceeded but you can not set it
much higher.

If 64 bit OS then use:

-Xmx30g

Arne

PS: Xms is Xmx is virtual memory not physical memory.
 
M

Martin Gregorie

Is it possibile to limit the max heap size to available physical memory
or anyway to max Windows allocation memory limit for a single process ?
I don't know about Windows, but under Linux -Xmx can be bigger than
physical memory. The app will just chug on if it exceeds free memory,
though rather slowly. I imagine, though, that it would crash if it ran
out of swap space as well..
 
M

Martin Gregorie

It's allowed in Windows as well. But as a previous respondent said,
32-bit Windows only allows a setting of about 1650m, and it's getting
relatively rare for Windows systems to have less than 2GB. If it does
happen, though, it could indeed result in thrashing if both physical
memory and available swap space get tight.

Another gotcha with both OSes is that its getting fashionable to limit
swap space to the same size as physical memory rather than the
traditional RAM * 2 - I don't know why: disk space is getting bigger and
cheaper every day.
 
A

Arne Vajhøj

Patrick said:
That's under Windows.

OP stated Windows.
You can go to three or three and a half gig
under various Unixes.

Yup. But runs 32 bit Unix today.
Well, you could. In practice I've found that much more than eight
or ten gig will result in GC becoming a problem (as in seconds rather
than milliseconds).

Isn't that a matter of tuning GC via all the X and XX's ?

Arne
 
A

Arne Vajhøj

Martin said:
I don't know about Windows, but under Linux -Xmx can be bigger than
physical memory.

Xmx is virtual memory, so I would expect that behavior on practically
any OS.

Arne
 
A

Arne Vajhøj

Martin said:
Another gotcha with both OSes is that its getting fashionable to limit
swap space to the same size as physical memory rather than the
traditional RAM * 2 - I don't know why: disk space is getting bigger and
cheaper every day.

I think there is a myth that less page/swap space => less paging/swapping.

Arne
 
L

Lew

Isn't that a matter of tuning GC via all the X and XX's ?

That is absolutely correct. I work on projects where there is 16 to 64 GB of
RAM given to the JVMs, and careful tuning of GC parameters makes it work.

With generational garbage collectors and correct idiomatic use of Java there
shouldn't be that much garbage to collect in a full GC. With multi-processor
systems and, for example, concurrent GC the pauses can be amortized to a point
where they do not impede.

As with big-iron database systems, tuning the JVM for big systems is a Dark Art.
 
M

Martin Gregorie

Martin Gregorie wrote:

I think there is a myth that less page/swap space => less
paging/swapping.
Yes, that makes a mad sort of sense.

I notice that my car has fewer punctures if I run it on three wheels, too.
 
A

Arne Vajhøj

Patrick said:
In my experience, even with the best tuning parameters, the border
lies somewhere around 10 to 15 gig. The company I work for has a number
of customers running happily in the eight gig range, but one customer
has pushed to 15 gig and is experiencing GC pauses two decimal orders of
magnitude higher than they did when they were below ten gig.

Obviously this is application specific to a degree, but I suspect
that it may have something to do with the JVM garbage collector itself.
While I am under NDA, I can say that this particular system involves
running pattern matching algorithms over a large data set. I'd be
interested in hearing about other results.

I would be surprised if it would not be possible to change
that GC behavior.

But one should never say never. Weirder things have been seen before.

If it indeed is the case, then it is something that need to be
addressed in future JVM, because >8 GB will soon become the norm.

And since those boxes will have lots of cores, then the CPU/mem
ration has not decreased and it should be possible not to have
huge GC hicups.

Arne
 
L

Lew

How long is that?

We run larger heaps than that and have GC pauses down to acceptable levels.
I would be surprised if it would not be possible to change
that GC behavior.

Apparently it is.
But one should never say never. Weirder things have been seen before.

If it indeed is the case, then it is something that need to be
addressed in future JVM, because >8 GB will soon become the norm.

And since those boxes will have lots of cores, then the CPU/mem
ration has not decreased and it should be possible not to have
huge GC hicups.

If the system uses RMI on Sun's JVM, there's a hidden trigger for full GCs
once a minute. Reconfigure that to once per hour and GC overhead drops
dramatically.

Part of the trick is to use idioms that allow young-generation GCs to
dominate, which are quicker than full (tenured-generation) GCs. Avoid
references from tenured objects to short-lived objects, don't foolishly reuse
objects so that they get tenured, change RMI's forced GC to be once per hour
instead of once per minute, use -XX params - it's a Dark Art but doable.

Consider using different vendors' JVMs.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top