CPU increasing slowly up to 100%

[SLL]

Hi all.
We have 3 JVM installed on a 4 CPU server running Windows 2000 server.
JVM is Sun 1.4.2_04.
Under constant load, the cpu usage is SLOWLY increasing from 5% up to
100% in 20-30 hours. The cpu increase is not linear, but parabolic.
The cpu usage is equally distributed on the 3 JVM (33% each).
Memory usage per JMV remains constant (300-400 MB).
We already have investigated the Garbage Collector behavior and seams
not to be an issue, because the GC interval and the time used fo GC
are constant and <100 ms.
When we restart 1 of the 3 JVMs, the CPU usage suddenly decrease to 5%
and the observed behavoir restart again.


Any idea?

Thanks.

-=SLL=-

 

[Filip Larsen]

SLL wrote

We have 3 JVM installed on a 4 CPU server running Windows 2000 server.
JVM is Sun 1.4.2_04.
Under constant load, the cpu usage is SLOWLY increasing from 5% up to
100% in 20-30 hours. The cpu increase is not linear, but parabolic.
The cpu usage is equally distributed on the 3 JVM (33% each).
Memory usage per JMV remains constant (300-400 MB).
We already have investigated the Garbage Collector behavior and seams
not to be an issue, because the GC interval and the time used fo GC
are constant and <100 ms.
When we restart 1 of the 3 JVMs, the CPU usage suddenly decrease to 5%
and the observed behavoir restart again.

Perhaps your processes are making linear processing through ever
increasing lists? If I were in your place I would probably pull out a
profiler about now and start measuring where time was spend. With your
system setup it sounds like profiling should be an exercise done on
regular basis, even if everything seems to run smoothly.


Regards,

 

[Paul Lutus]

Hi all.
We have 3 JVM installed on a 4 CPU server running Windows 2000 server.
JVM is Sun 1.4.2_04.
Under constant load, the cpu usage is SLOWLY increasing from 5% up to
100% in 20-30 hours. The cpu increase is not linear, but parabolic.

I believe you mean exponential. "Parabolic" would mean the usage would begin
by climbing very steeply, level off, and then fall, like the path of a
thrown ball.

The cpu usage is equally distributed on the 3 JVM (33% each).
Memory usage per JMV remains constant (300-400 MB).
We already have investigated the Garbage Collector behavior and seams
not to be an issue, because the GC interval and the time used fo GC
are constant and <100 ms.

This strongly implies that some object references are not being released for
garbage collection.

When we restart 1 of the 3 JVMs, the CPU usage suddenly decrease to 5%
and the observed behavoir restart again.

Tell us if this is a new problem with existing software, or a new problem
with new software. Tell us if you have access to, and control over, the
source code for the software.

 

[Tony Morris]

This strongly implies that some object references are not being released for
garbage collection.

eh? How does memory remaining constant imply that?

I'm speculating that there is some sort of linear structure that is growing
over time e.g. a linear search cache (doesn't sound nice, but I've seen it
done).
One can only speculate.

 

[Paul Lutus]

eh? How does memory remaining constant imply that?

Sorry. I orignally read this as an example of parasitic processor usage by
objects and/or threads that were not terminating properly. Then I said
something seemingly unrelated, ruining any chance for clear expression.

I'm speculating that there is some sort of linear structure that is
growing over time e.g. a linear search cache (doesn't sound nice, but I've
seen it done).
One can only speculate.

I agree.

 

[SLL]

SLL wrote


Perhaps your processes are making linear processing through ever
increasing lists? If I were in your place I would probably pull out a
profiler about now and start measuring where time was spend. With your
system setup it sounds like profiling should be an exercise done on
regular basis, even if everything seems to run smoothly.


Regards,

Thanks for your reply.
I forgot to mention that we already done profiling, and CPU seams
-not- to used heavyly by internal JVM threads: the CPU usage
distribution on the various threads is similar when cpu is at 5% and
when is at 100%.
As you state, we use an internal cache for requests processed and our
first investigation was for measuring time spent in validating if
objects in cache are valid or expired, but this is not an issue: our
internal cache garbace collector run every 300-600 seconds and the
time to scan the cache is 5-10 seconds; the number objs to delete and
the frequency of cache GC is controlled by fuzzy logic algorithm, so
the number of objs in cache is constant (more or less).

The "strange thing" is that, as I mention above, stopping 1 JVM when
cpu is at 100% reverts the situation to a total 5% CPU usage in few
seconds. So there should be something "external" factor to JVMs (at OS
level?) that creates the CPU increase. We don't use -XX:+AggresiveHeap
for java GC tuning, so RAM should not to be in contention by JVMs.
We have also investigated the number of network socket used, that
remains constant and so should not be the main cause for the cpu
rising.

We are run out of ideas.... :-(

-=SLL=-

 

[SLL]

Tell us if this is a new problem with existing software, or a new problem
with new software. Tell us if you have access to, and control over, the
source code for the software.

This is a new problem in new software, manifested only on our
production system (4 CPU machines).
We are not able to reproduce such a behavoir in our development (1 CPU
machines) and test bed (2 CPU machines) environments.
We have total control over source code. The application logging system
reveals no anomalies during CPU increase, the number of threads used
is more or less constant (50-60).
Any ideas for further investigation?

 

[John B. Matthews]

This is a new problem in new software, manifested only on our
production system (4 CPU machines).
We are not able to reproduce such a behavoir in our development (1 CPU
machines) and test bed (2 CPU machines) environments.
We have total control over source code. The application logging system
reveals no anomalies during CPU increase, the number of threads used
is more or less constant (50-60).
Any ideas for further investigation?

Is there any reason to suspect the CPU measurement tool? For
example, top initially reports CPU usage over a long period and then
reports interval usage after the first update. Until that first
update, the apparent usage may be distorted by recent heavy usage.

 

[Paul Lutus]

SLL wrote:

/ ...

Any ideas for further investigation?

After my first post, I thought about this some more and realized two other
possibilities.

1. I think you should look for examples of one process waiting for another
to complete, but waiting in inefficient ways.

Generic example:

while(otherProcessNotComplete);

Sometimes people think if this sort of wait is placed in a thread, there's
no harm in it -- but this sort of wait can eat CPU time like crazy.

Alternative:

while(otherProcessNotComplete) {
Thread.sleep(1000);
}

I hope this is not too terribly obvious. I raise it because of your report
that if you kill off one JVM, the problem goes away. This made me think of
various kinds of time-eaters and deadlocks.

2. On the topic of deadlocks, and again at the risk of saying the obvious,
never stop a thread externally. If you must stop a thread, set a flag that
is visible to the thread, have the thread's code detect the state of the
flag and exit on its own.

This is also a classic way to create the kind of problem you are seeing, and
it is also possibly entirely too obvious.

I apologize in advance if these are simply not sophisticated enough to rank
as possibilities in your situation.

 

[Tony Morris]

SLL wrote:

/ ...


After my first post, I thought about this some more and realized two other
possibilities.

1. I think you should look for examples of one process waiting for another
to complete, but waiting in inefficient ways.

Generic example:

while(otherProcessNotComplete);

Sometimes people think if this sort of wait is placed in a thread, there's
no harm in it -- but this sort of wait can eat CPU time like crazy.

Alternative:

while(otherProcessNotComplete) {
Thread.sleep(1000);
}

I hope this is not too terribly obvious. I raise it because of your report
that if you kill off one JVM, the problem goes away. This made me think of
various kinds of time-eaters and deadlocks.

2. On the topic of deadlocks, and again at the risk of saying the obvious,
never stop a thread externally. If you must stop a thread, set a flag that
is visible to the thread, have the thread's code detect the state of the
flag and exit on its own.

This is also a classic way to create the kind of problem you are seeing, and
it is also possibly entirely too obvious.

I apologize in advance if these are simply not sophisticated enough to rank
as possibilities in your situation.

Good point.
This is a known abuse - the term coined "the busy/wait anti-pattern".
However, I disagree with your suggested solution.
Almost certainly the desired solution is the use of a wait/notify as opposed
to polling.

Polling can be defined as "check for state changes at specific time
intervals".
The problem with this approach is that you don't know what interval to set
your polling period to. What if the state changes to and back again within
an interval? What if we want real-time notification? Just to name a couple.

The wait/notify mechanism provides a 'callback' for a thread to register
itself for notifications.
Sometimes this is impossible; for example with JDBC, you cannot find out if
a database table changes, so the only alternative it to poll. Or, if you are
not restricted to using JDBC, your database might permit the use of database
triggers, which is analogous to wait/notify.