Memory leak due to too many threads?

[bernd]

Hi folks,

my application polls a SysV-message queue from within a thread (class
Mqueue extending Thread). The queue is polled repeatedly using a
native function (which is basically a java-port of msgrcv()). Between
two polling actions I wait a few hundred milliseconds using

Thread.sleep( pollint ) ; # within Mqueue.run()

poll_int being the number of milliseconds which should elapse after an
invocation of the polling function.

If the number of messages which are sent over the IPC-queue is not too
high (up to 100 messages), i.e., if the sending procress does not last
too long, everything works fine, but if a higher number of messages is
generated the application hangs and sometimes it crashes with the
following error message:

Exception in thread "CompilerThread0" java.lang.OutOfMemoryError:
requested 32756 bytes for ChunkPool::allocate. Out of swap space?
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at IPC.MQueue.run(MQueue.java:253)
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at IPC.MQueue.run(MQueue.java:253)
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at IPC.MQueue.run(MQueue.java:253)

Exception in thread "CompilerThread1" java.lang.OutOfMemoryError:
requested 32756 bytes for ChunkPool::allocate. Out of swap space?

My questions to the experts now:

1. Is it just by chance that (and possibly due to the “waiting” nature
of the sleep-Thread) that the interruption occurs during the sleep()
or does this point to a too high number of threads in my application
which lead to the memory leak?

2. Up to now I did not explicitly stop() any thread in my application.
Is this strictly necessary or may I rely on the GC to not run into
trouble with too a high number of threads? Is it possible to destroy
the Mqueue-object, which from my point of view is the possible
originator of the leak, after it is not used any more to avoid the
leak?

3. How do I find out the exact reason for the memory leak? I am just
speculating concerning the threads being the reason for the problem.
Is this possible by only using the JVM and appropriate command line
option?

4. Is it worth to play around with the –Xmx option of the JVM?

Cheers


Bernd

 

[Gordon Beaton]

Exception in thread "CompilerThread0" java.lang.OutOfMemoryError:
requested 32756 bytes for ChunkPool::allocate. Out of swap space?

I'd suspect something you're doing within the native method itself.

/gordon

--

 

[Gordon Beaton]

1. Is it just by chance that (and possibly due to the ?waiting? nature
of the sleep-Thread) that the interruption occurs during the sleep()
or does this point to a too high number of threads in my application
which lead to the memory leak?

How many threads is "a high number"? Are you creating a new one each
time you poll the msgqueue?

/gordon

--

 

[Tom Anderson]

Exception in thread "CompilerThread0" java.lang.OutOfMemoryError:
requested 32756 bytes for ChunkPool::allocate. Out of swap space?

This seems to be a known bug:

http://dschneller.blogspot.com/2007/04/exception-in-compilerthread0.html
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6487381
http://www.codingthearchitecture.com/2008/01/14/jvm_lies_the_outofmemory_myth.html

(ping Mark Space - i draw your attention in particular to the last of
those links!)

Try upgrading your JVM. Failing that, look for leaks in the native code -
what's happening is that native space is being gobbled up, and then the
JVM's compiler gets upset.

1. Is it just by chance that (and possibly due to the ?waiting? nature
of the sleep-Thread) that the interruption occurs during the sleep() or
does this point to a too high number of threads in my application which
lead to the memory leak?

It happens because HotSpot is running when all the app threads are asleep.

2. Up to now I did not explicitly stop() any thread in my application.
Is this strictly necessary or may I rely on the GC to not run into
trouble with too a high number of threads?

Should be fine.

Is it possible to destroy the Mqueue-object, which from my point of view
is the possible originator of the leak, after it is not used any more to
avoid the leak?

You can't explicitly destroy any java object. You can ensure that there
are no references to it, and let it be collected. But in this case, i
don't think that's where the leak is.

3. How do I find out the exact reason for the memory leak?

If there was a general answer to that, the world would be a much happier
place!

I am just speculating concerning the threads being the reason for the
problem. Is this possible by only using the JVM and appropriate command
line option?

No. And i think the leak is likely to be in your native message queue
code, not anything to do with threads. Are you mallocing a buffer and then
not freeing it when you do a receive?

4. Is it worth to play around with the ?Xmx option of the JVM?

One of those links suggests *decreasing* the size of the heap.

tom

 

[bernd]

How many threads is "a high number"? Are you creating a new one each
time you poll the msgqueue?

/gordon

--

Hi Gordon,

to answer Your question I have go into more detail:

In a loop the main prog instantiates an MQueue-object which creates
the IPC-message queue first and then polls it from within run() of the
Thread. Afterwards a class different from MQueue (AppCtrl) fires up a
c-program which generates the messages (frequency: 1 Hz).

When the c-program has finished I terminate Mqueue’s thread using
Mqueue.interrupt(). I guess that the MQueue-object is GCed then, or?

Then, since all takes place in the above mentioned loop, a new
instance of MQueue is generated and the c-prog is started with
different command line parameters and the whole thing starts again.

Within MQueue.run() the only thing I do with the Thread-class is
pausing it for the mentioned pollint-Interval.

I am not sure any more: Do I generate a new thread by using

Thread.sleep( pollint ) ; ?

I do not think so.

But even if I would: after pollint secs this should be vanished again,
or?

Cheers

Bernd

 

[Gordon Beaton]

When the c-program has finished I terminate Mqueue?s thread using
Mqueue.interrupt(). I guess that the MQueue-object is GCed then, or?

That depends on what MQueue.run() contains (more than just
a call to Thread.sleep(), I'm guessing).

/gordon

--

 

[Roedy Green]

It is highly unlikely you will find any sort of memory leak. You
might find some packratting. See
http://mindprod.com/jgloss/packratting.html

Running out of memory can have causes other than improperly holding
onto objects. One simple thing to check is just how many times have
you said "new Thread" vs how many times one of those thread
terminated.

Threads are whacking huge objects. You will soon saturate ram if you
create them without bound.

 

[bernd]

This seems to be a known bug:

http://dschneller.blogspot.com/2007...itecture.com/2008/01/14/jvm_lies_the_outofmem...

(ping Mark Space - i draw your attention in particular to the last of
those links!)

Try upgrading your JVM. Failing that, look for leaks in the native code -
what's happening is that native space is being gobbled up, and then the
JVM's compiler gets upset.


It happens because HotSpot is running when all the app threads are asleep..


Should be fine.


You can't explicitly destroy any java object. You can ensure that there
are no references to it, and let it be collected. But in this case, i
don't think that's where the leak is.


If there was a general answer to that, the world would be a much happier
place!


No. And i think the leak is likely to be in your native message queue
code, not anything to do with threads. Are you mallocing a buffer and then
not freeing it when you do a receive?


One of those links suggests *decreasing* the size of the heap.

tom

Hi Tom,

thx for Your thorough reply. I studied Daniel Schneller's
contribution.

One solution to this seems to be what he called the "switch to the
client JVM". What does he mean by this and would this be feasible for
me as well?

Cheers


Bernd

 

[Tom Anderson]

I studied Daniel Schneller's contribution.

One solution to this seems to be what he called the "switch to the
client JVM". What does he mean by this and would this be feasible for me
as well?

This refers to whether you're using the -client or -server flag to the
JVM. If you're using neither, you're already running in client mode.

tom

 

[bernd]

This seems to be a known bug:

http://dschneller.blogspot.com/2007...itecture.com/2008/01/14/jvm_lies_the_outofmem...

(ping Mark Space - i draw your attention in particular to the last of
those links!)

Try upgrading your JVM. Failing that, look for leaks in the native code -
what's happening is that native space is being gobbled up, and then the
JVM's compiler gets upset.


It happens because HotSpot is running when all the app threads are asleep..


Should be fine.


You can't explicitly destroy any java object. You can ensure that there
are no references to it, and let it be collected. But in this case, i
don't think that's where the leak is.


If there was a general answer to that, the world would be a much happier
place!


No. And i think the leak is likely to be in your native message queue
code, not anything to do with threads. Are you mallocing a buffer and then
not freeing it when you do a receive?


One of those links suggests *decreasing* the size of the heap.

tom

Hi Tom,

different question:

I am not a very experienced c-programmer, so I was of the opinion that
freeing the buffer for the message is not strictly necessary, since I
invoke msgrcv() (or better: it's native java-equivalent) only once for
each message on the queue (and the size of each message is around 4K).
Is the memory not cleaned up after the function returned?

Cheers


Bernd

 

[Gordon Beaton]

I am not a very experienced c-programmer, so I was of the opinion that
freeing the buffer for the message is not strictly necessary, since I
invoke msgrcv() (or better: it's native java-equivalent) only once for
each message on the queue (and the size of each message is around 4K).

What is the "native java equivalent" if not simply a JNI wrapper
around the msgrcv() system function itself?

Is the memory not cleaned up after the function returned?

If you've explicitly allocated anything in C you must free it as well,
it isn't optional.

If you created Java objects using the JNI API from C, then in most
(but not all) cases Java will take care of them for you.

Why don't you post some code, not just descriptions of it.

/gordon

--