CPU increasing slowly up to 100%

Discussion in 'Java' started by SLL, Oct 16, 2004.

  1. SLL

    SLL Guest

    Hi all.
    We have 3 JVM installed on a 4 CPU server running Windows 2000 server.
    JVM is Sun 1.4.2_04.
    Under constant load, the cpu usage is SLOWLY increasing from 5% up to
    100% in 20-30 hours. The cpu increase is not linear, but parabolic.
    The cpu usage is equally distributed on the 3 JVM (33% each).
    Memory usage per JMV remains constant (300-400 MB).
    We already have investigated the Garbage Collector behavior and seams
    not to be an issue, because the GC interval and the time used fo GC
    are constant and <100 ms.
    When we restart 1 of the 3 JVMs, the CPU usage suddenly decrease to 5%
    and the observed behavoir restart again.


    Any idea?

    Thanks.

    -=SLL=-
     
    SLL, Oct 16, 2004
    #1
    1. Advertising

  2. SLL

    Filip Larsen Guest

    SLL wrote

    > We have 3 JVM installed on a 4 CPU server running Windows 2000 server.
    > JVM is Sun 1.4.2_04.
    > Under constant load, the cpu usage is SLOWLY increasing from 5% up to
    > 100% in 20-30 hours. The cpu increase is not linear, but parabolic.
    > The cpu usage is equally distributed on the 3 JVM (33% each).
    > Memory usage per JMV remains constant (300-400 MB).
    > We already have investigated the Garbage Collector behavior and seams
    > not to be an issue, because the GC interval and the time used fo GC
    > are constant and <100 ms.
    > When we restart 1 of the 3 JVMs, the CPU usage suddenly decrease to 5%
    > and the observed behavoir restart again.


    Perhaps your processes are making linear processing through ever
    increasing lists? If I were in your place I would probably pull out a
    profiler about now and start measuring where time was spend. With your
    system setup it sounds like profiling should be an exercise done on
    regular basis, even if everything seems to run smoothly.


    Regards,
    --
    Filip Larsen
     
    Filip Larsen, Oct 17, 2004
    #2
    1. Advertising

  3. SLL

    Paul Lutus Guest

    SLL wrote:

    > Hi all.
    > We have 3 JVM installed on a 4 CPU server running Windows 2000 server.
    > JVM is Sun 1.4.2_04.
    > Under constant load, the cpu usage is SLOWLY increasing from 5% up to
    > 100% in 20-30 hours. The cpu increase is not linear, but parabolic.


    I believe you mean exponential. "Parabolic" would mean the usage would begin
    by climbing very steeply, level off, and then fall, like the path of a
    thrown ball.

    > The cpu usage is equally distributed on the 3 JVM (33% each).
    > Memory usage per JMV remains constant (300-400 MB).
    > We already have investigated the Garbage Collector behavior and seams
    > not to be an issue, because the GC interval and the time used fo GC
    > are constant and <100 ms.


    This strongly implies that some object references are not being released for
    garbage collection.

    > When we restart 1 of the 3 JVMs, the CPU usage suddenly decrease to 5%
    > and the observed behavoir restart again.


    Tell us if this is a new problem with existing software, or a new problem
    with new software. Tell us if you have access to, and control over, the
    source code for the software.

    --
    Paul Lutus
    http://www.arachnoid.com
     
    Paul Lutus, Oct 17, 2004
    #3
  4. SLL

    Tony Morris Guest


    > > The cpu usage is equally distributed on the 3 JVM (33% each).
    > > Memory usage per JMV remains constant (300-400 MB).
    > > We already have investigated the Garbage Collector behavior and seams
    > > not to be an issue, because the GC interval and the time used fo GC
    > > are constant and <100 ms.

    >
    > This strongly implies that some object references are not being released

    for
    > garbage collection.


    eh? How does memory remaining constant imply that?

    I'm speculating that there is some sort of linear structure that is growing
    over time e.g. a linear search cache (doesn't sound nice, but I've seen it
    done).
    One can only speculate.

    --
    Tony Morris
    http://xdweb.net/~dibblego/
     
    Tony Morris, Oct 17, 2004
    #4
  5. SLL

    Paul Lutus Guest

    Tony Morris wrote:

    >
    >> > The cpu usage is equally distributed on the 3 JVM (33% each).
    >> > Memory usage per JMV remains constant (300-400 MB).
    >> > We already have investigated the Garbage Collector behavior and seams
    >> > not to be an issue, because the GC interval and the time used fo GC
    >> > are constant and <100 ms.

    >>
    >> This strongly implies that some object references are not being released

    > for
    >> garbage collection.

    >
    > eh? How does memory remaining constant imply that?


    Sorry. I orignally read this as an example of parasitic processor usage by
    objects and/or threads that were not terminating properly. Then I said
    something seemingly unrelated, ruining any chance for clear expression.

    > I'm speculating that there is some sort of linear structure that is
    > growing over time e.g. a linear search cache (doesn't sound nice, but I've
    > seen it done).
    > One can only speculate.


    I agree.

    --
    Paul Lutus
    http://www.arachnoid.com
     
    Paul Lutus, Oct 17, 2004
    #5
  6. SLL

    SLL Guest

    "Filip Larsen" <> wrote in message news:<ckse26$13mh$>...
    > SLL wrote
    >
    > > We have 3 JVM installed on a 4 CPU server running Windows 2000 server.
    > > JVM is Sun 1.4.2_04.
    > > Under constant load, the cpu usage is SLOWLY increasing from 5% up to
    > > 100% in 20-30 hours. The cpu increase is not linear, but parabolic.
    > > The cpu usage is equally distributed on the 3 JVM (33% each).
    > > Memory usage per JMV remains constant (300-400 MB).
    > > We already have investigated the Garbage Collector behavior and seams
    > > not to be an issue, because the GC interval and the time used fo GC
    > > are constant and <100 ms.
    > > When we restart 1 of the 3 JVMs, the CPU usage suddenly decrease to 5%
    > > and the observed behavoir restart again.

    >
    > Perhaps your processes are making linear processing through ever
    > increasing lists? If I were in your place I would probably pull out a
    > profiler about now and start measuring where time was spend. With your
    > system setup it sounds like profiling should be an exercise done on
    > regular basis, even if everything seems to run smoothly.
    >
    >
    > Regards,



    Thanks for your reply.
    I forgot to mention that we already done profiling, and CPU seams
    -not- to used heavyly by internal JVM threads: the CPU usage
    distribution on the various threads is similar when cpu is at 5% and
    when is at 100%.
    As you state, we use an internal cache for requests processed and our
    first investigation was for measuring time spent in validating if
    objects in cache are valid or expired, but this is not an issue: our
    internal cache garbace collector run every 300-600 seconds and the
    time to scan the cache is 5-10 seconds; the number objs to delete and
    the frequency of cache GC is controlled by fuzzy logic algorithm, so
    the number of objs in cache is constant (more or less).

    The "strange thing" is that, as I mention above, stopping 1 JVM when
    cpu is at 100% reverts the situation to a total 5% CPU usage in few
    seconds. So there should be something "external" factor to JVMs (at OS
    level?) that creates the CPU increase. We don't use -XX:+AggresiveHeap
    for java GC tuning, so RAM should not to be in contention by JVMs.
    We have also investigated the number of network socket used, that
    remains constant and so should not be the main cause for the cpu
    rising.

    We are run out of ideas.... :-(

    -=SLL=-
     
    SLL, Oct 17, 2004
    #6
  7. SLL

    SLL Guest

    Paul Lutus <Paul Lutus <>> wrote in message news:<>...


    >
    > Tell us if this is a new problem with existing software, or a new problem
    > with new software. Tell us if you have access to, and control over, the
    > source code for the software.


    This is a new problem in new software, manifested only on our
    production system (4 CPU machines).
    We are not able to reproduce such a behavoir in our development (1 CPU
    machines) and test bed (2 CPU machines) environments.
    We have total control over source code. The application logging system
    reveals no anomalies during CPU increase, the number of threads used
    is more or less constant (50-60).
    Any ideas for further investigation?
     
    SLL, Oct 17, 2004
    #7
  8. In article <>,
    (SLL) wrote:

    > Paul Lutus <Paul Lutus <>> wrote in message
    > news:<>...
    >
    >
    > >
    > > Tell us if this is a new problem with existing software, or a new problem
    > > with new software. Tell us if you have access to, and control over, the
    > > source code for the software.

    >
    > This is a new problem in new software, manifested only on our
    > production system (4 CPU machines).
    > We are not able to reproduce such a behavoir in our development (1 CPU
    > machines) and test bed (2 CPU machines) environments.
    > We have total control over source code. The application logging system
    > reveals no anomalies during CPU increase, the number of threads used
    > is more or less constant (50-60).
    > Any ideas for further investigation?


    Is there any reason to suspect the CPU measurement tool? For
    example, top initially reports CPU usage over a long period and then
    reports interval usage after the first update. Until that first
    update, the apparent usage may be distorted by recent heavy usage.

    --
    John
    ----
    jmatthews at wright dot edu
    www dot wright dot edu/~john.matthews/
     
    John B. Matthews, Oct 17, 2004
    #8
  9. SLL

    Paul Lutus Guest

    SLL wrote:

    / ...

    > Any ideas for further investigation?


    After my first post, I thought about this some more and realized two other
    possibilities.

    1. I think you should look for examples of one process waiting for another
    to complete, but waiting in inefficient ways.

    Generic example:

    while(otherProcessNotComplete);

    Sometimes people think if this sort of wait is placed in a thread, there's
    no harm in it -- but this sort of wait can eat CPU time like crazy.

    Alternative:

    while(otherProcessNotComplete) {
    Thread.sleep(1000);
    }

    I hope this is not too terribly obvious. I raise it because of your report
    that if you kill off one JVM, the problem goes away. This made me think of
    various kinds of time-eaters and deadlocks.

    2. On the topic of deadlocks, and again at the risk of saying the obvious,
    never stop a thread externally. If you must stop a thread, set a flag that
    is visible to the thread, have the thread's code detect the state of the
    flag and exit on its own.

    This is also a classic way to create the kind of problem you are seeing, and
    it is also possibly entirely too obvious.

    I apologize in advance if these are simply not sophisticated enough to rank
    as possibilities in your situation.

    --
    Paul Lutus
    http://www.arachnoid.com
     
    Paul Lutus, Oct 18, 2004
    #9
  10. SLL

    Tony Morris Guest

    "Paul Lutus" <> wrote in message
    news:...
    > SLL wrote:
    >
    > / ...
    >
    > > Any ideas for further investigation?

    >
    > After my first post, I thought about this some more and realized two other
    > possibilities.
    >
    > 1. I think you should look for examples of one process waiting for another
    > to complete, but waiting in inefficient ways.
    >
    > Generic example:
    >
    > while(otherProcessNotComplete);
    >
    > Sometimes people think if this sort of wait is placed in a thread, there's
    > no harm in it -- but this sort of wait can eat CPU time like crazy.
    >
    > Alternative:
    >
    > while(otherProcessNotComplete) {
    > Thread.sleep(1000);
    > }
    >
    > I hope this is not too terribly obvious. I raise it because of your report
    > that if you kill off one JVM, the problem goes away. This made me think of
    > various kinds of time-eaters and deadlocks.
    >
    > 2. On the topic of deadlocks, and again at the risk of saying the obvious,
    > never stop a thread externally. If you must stop a thread, set a flag that
    > is visible to the thread, have the thread's code detect the state of the
    > flag and exit on its own.
    >
    > This is also a classic way to create the kind of problem you are seeing,

    and
    > it is also possibly entirely too obvious.
    >
    > I apologize in advance if these are simply not sophisticated enough to

    rank
    > as possibilities in your situation.
    >
    > --
    > Paul Lutus
    > http://www.arachnoid.com
    >


    Good point.
    This is a known abuse - the term coined "the busy/wait anti-pattern".
    However, I disagree with your suggested solution.
    Almost certainly the desired solution is the use of a wait/notify as opposed
    to polling.

    Polling can be defined as "check for state changes at specific time
    intervals".
    The problem with this approach is that you don't know what interval to set
    your polling period to. What if the state changes to and back again within
    an interval? What if we want real-time notification? Just to name a couple.

    The wait/notify mechanism provides a 'callback' for a thread to register
    itself for notifications.
    Sometimes this is impossible; for example with JDBC, you cannot find out if
    a database table changes, so the only alternative it to poll. Or, if you are
    not restricted to using JDBC, your database might permit the use of database
    triggers, which is analogous to wait/notify.

    --
    Tony Morris
    http://xdweb.net/~dibblego/
     
    Tony Morris, Oct 19, 2004
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Andreas Klemt
    Replies:
    2
    Views:
    433
    Marina
    Jul 28, 2003
  2. Not4u
    Replies:
    9
    Views:
    1,060
    Not4u
    Feb 27, 2004
  3. Replies:
    8
    Views:
    6,733
    Neredbojias
    Dec 9, 2005
  4. fred
    Replies:
    3
    Views:
    303
    Zifud
    Mar 17, 2005
  5. Replies:
    5
    Views:
    914
Loading...

Share This Page