Linux java.lang.OutOfMemoryError: unable to create new native thread

M

Mit Majithia

Hi,
We are running a web application which is a stateless system and
gets a lot of hits a day per server.
We are running the following configuration :
Red Hat Linux 7.3
Linux Kernel 2.4.20-19.7smp
Sun JVM JDK 1.4.1_02
The machine has 2GB of RAM and we have set the -XMx=1500M.

Pretty much everyday after about 200,000 - 250,000 hits, the
server get the error shown below. We have looked at the memory usage
and it looks ok.The application does load up some data and use a
substantial amount of memory(about 500M). But we looking through both
top and Runtime.getRuntime().totalMemory() we see that it has still
not close to the 1500M limit that we have set. Also we use the
following code on a jsp page to montior the threads and they don't
seem to be spiralling out of control

Thread[] threads = new Thread[1000];
ThreadGroup tg = Thread.currentThread().getThreadGroup();

while (tg.getParent() != null)
{
tg = tg.getParent();
}

tg.enumerate( threads );
for (int i = 0; i < threads.length; i++)
{
if (threads != null)
{
out.print(i + ": ");
out.print( "pri=" + threads.getPriority() + " ");
out.print( "ac=" + threads.activeCount() + " ");
out.println(threads);
}
}

Does anyone have any idea what the problem could be ? What the
possible issues could be ? We have looked at Memory and threads and
they don't seem to be the issue. Is here any documentation about the
cases that would through up this error ? Any help in this matter would
be greatly appreciated.

Thanks,
Mit








javax.servlet.ServletException: Servlet execution threw an exception
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Unknown
Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(Unknown
Source)
at org.apache.catalina.core.StandardWrapperValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.catalina.core.StandardContextValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.catalina.core.StandardContext.invoke(Unknown Source)
at org.apache.catalina.core.StandardHostValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.valves.ErrorReportValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.catalina.core.StandardEngineValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:594)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:392)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:565)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:619)
at java.lang.Thread.run(Thread.java:536)
----- Root Cause -----
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start(Native Method)
at com.i411.search.controller.LowLevelSearch.performSocketSearch(LowLevelSearch.java:428)
at com.i411.search.controller.LowLevelSearch.performNetworkSearch(LowLevelSearch.java:348)
at com.i411.search.controller.LowLevelSearch.ExecuteSearch(LowLevelSearch.java:145)
at com.i411.search.controller.SearchController.catItemsSearch(SearchController.java:407)
at com.i411.search.controller.SearchController.getSearchResults(SearchController.java:84)
at com.i411.dex.searchflow.SearchExecutor.run(SearchExecutor.java:45)
at com.i411.dex.searchflow.BusinessNameSearchExecutor.run(BusinessNameSearchExecutor.java:82)
at com.i411.dex.searchflow.DexSearchFlow.performSearchFlow(DexSearchFlow.java:335)
at com.i411.web.common.WebController.executeSearchWithSearchFlow(WebController.java:137)
at com.i411.web.common.WebController.executeSearchWithSearchFlow(WebController.java:182)
at com.i411.dex.xml.XMLSearchServlet.PerformQuerySearch(XMLSearchServlet.java:260)
at com.i411.dex.xml.XMLSearchServlet.doGet(XMLSearchServlet.java:178)
at com.i411.dex.xml.XMLSearchServlet.doPost(XMLSearchServlet.java:366)
at javax.servlet.http.HttpServlet.service(HttpServlet.java)
at javax.servlet.http.HttpServlet.service(HttpServlet.java)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Unknown
Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(Unknown
Source)
at org.apache.catalina.core.StandardWrapperValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.catalina.core.StandardContextValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.catalina.core.StandardContext.invoke(Unknown Source)
at org.apache.catalina.core.StandardHostValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.valves.ErrorReportValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.catalina.core.StandardEngineValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:594)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:392)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:565)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:619)
at java.lang.Thread.run(Thread.java:536)
 
A

Arnaud Berger

Hi !

We also encountered this problem before.

You will probably have to play with the "ulimit" command.

Regards,

Arnaud

Mit Majithia said:
Hi,
We are running a web application which is a stateless system and
gets a lot of hits a day per server.
We are running the following configuration :
Red Hat Linux 7.3
Linux Kernel 2.4.20-19.7smp
Sun JVM JDK 1.4.1_02
The machine has 2GB of RAM and we have set the -XMx=1500M.

Pretty much everyday after about 200,000 - 250,000 hits, the
server get the error shown below. We have looked at the memory usage
and it looks ok.The application does load up some data and use a
substantial amount of memory(about 500M). But we looking through both
top and Runtime.getRuntime().totalMemory() we see that it has still
not close to the 1500M limit that we have set. Also we use the
following code on a jsp page to montior the threads and they don't
seem to be spiralling out of control

Thread[] threads = new Thread[1000];
ThreadGroup tg = Thread.currentThread().getThreadGroup();

while (tg.getParent() != null)
{
tg = tg.getParent();
}

tg.enumerate( threads );
for (int i = 0; i < threads.length; i++)
{
if (threads != null)
{
out.print(i + ": ");
out.print( "pri=" + threads.getPriority() + " ");
out.print( "ac=" + threads.activeCount() + " ");
out.println(threads);
}
}

Does anyone have any idea what the problem could be ? What the
possible issues could be ? We have looked at Memory and threads and
they don't seem to be the issue. Is here any documentation about the
cases that would through up this error ? Any help in this matter would
be greatly appreciated.

Thanks,
Mit








javax.servlet.ServletException: Servlet execution threw an exception
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Unknown
Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(Unknown
Source)
at org.apache.catalina.core.StandardWrapperValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.catalina.core.StandardContextValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.catalina.core.StandardContext.invoke(Unknown Source)
at org.apache.catalina.core.StandardHostValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
Source)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
Source)
at org.apache.catalina.valves.ErrorReportValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.catalina.core.StandardEngineValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:594)
at
Click to expand...
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConne
ction(Http11Protocol.java:392)
at
Click to expand...
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:565)
at
Click to expand...
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:619)
at java.lang.Thread.run(Thread.java:536)
----- Root Cause -----
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start(Native Method)
at com.i411.search.controller.LowLevelSearch.performSocketSearch(LowLevelSearch
..java:428)
at
Click to expand...
com.i411.search.controller.LowLevelSearch.performNetworkSearch(LowLevelSearc
h.java:348)
at
Click to expand...
com.i411.search.controller.LowLevelSearch.ExecuteSearch(LowLevelSearch.java:
145)
at
Click to expand...
com.i411.search.controller.SearchController.catItemsSearch(SearchController.
java:407)
at
Click to expand...
com.i411.search.controller.SearchController.getSearchResults(SearchControlle
r.java:84)
at com.i411.dex.searchflow.SearchExecutor.run(SearchExecutor.java:45)
at com.i411.dex.searchflow.BusinessNameSearchExecutor.run(BusinessNameSearchExe
cutor.java:82)
at
Click to expand...
com.i411.dex.searchflow.DexSearchFlow.performSearchFlow(DexSearchFlow.java:3
35)
at
Click to expand...
com.i411.web.common.WebController.executeSearchWithSearchFlow(WebController.
java:137)
at
Click to expand...
com.i411.web.common.WebController.executeSearchWithSearchFlow(WebController.
java:182)
at
Click to expand...
com.i411.dex.xml.XMLSearchServlet.PerformQuerySearch(XMLSearchServlet.java:2
60)
at com.i411.dex.xml.XMLSearchServlet.doGet(XMLSearchServlet.java:178)
at com.i411.dex.xml.XMLSearchServlet.doPost(XMLSearchServlet.java:366)
at javax.servlet.http.HttpServlet.service(HttpServlet.java)
at javax.servlet.http.HttpServlet.service(HttpServlet.java)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Unknown
Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(Unknown
Source)
at org.apache.catalina.core.StandardWrapperValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.catalina.core.StandardContextValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.catalina.core.StandardContext.invoke(Unknown Source)
at org.apache.catalina.core.StandardHostValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
Source)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
Source)
at org.apache.catalina.valves.ErrorReportValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.catalina.core.StandardEngineValve.invoke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:594)
at
Click to expand...
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConne
ction(Http11Protocol.java:392)
at
Click to expand...
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:565)
at
Click to expand...
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:619)
at java.lang.Thread.run(Thread.java:536)
Click to expand...
 
J

Jayaram

Have encouneterd a similar problem on HP-UX 11. Check for values of
the soft and hard limits (kernel settings) on the number of threads
allowed per process on your system. These limits may have been set to
a low value on your system.
Reagrds,
Jay
 
M

Matthias Ernst

Mit said:
Hi,
We are running a web application which is a stateless system and
gets a lot of hits a day per server.
We are running the following configuration :
Red Hat Linux 7.3
Linux Kernel 2.4.20-19.7smp
Sun JVM JDK 1.4.1_02
The machine has 2GB of RAM and we have set the -XMx=1500M.
Click to expand...

The higher you set -mx, the less threads you can create (why does a
stateless system need 1.5G anyway ...)

I think a very naive formula is max threads = (2G - mx) / stacksize.
You may try to work with a smaller heap size or smaller stack sizes.

Matthias
 
M

Mit Majithia

Update:
We have been playing around with the settings.One of the
settings that we changed was to reduce the memory allocated to the JVM
(-Xmx1500m) to be lesser to -Xmx1000m. This seems to have alleviated
the situation a little for the time being. It does not go down and
does not give the error as before. Also we do have to restart the
server on a daily basis to update the data.But with the new settings
it makes it easily through the entire day without giving an error. Can
anyone explain this ? Due to the growing traffic and data we will
probably need the expanded memory sometime in the future. We feel that
at the moment we have just pushed the problem to surface at some point
in the future.

Thanks,
Mit
 
C

Chris Bailey

Mit Majithia said:
Update:
We have been playing around with the settings.One of the
settings that we changed was to reduce the memory allocated to the JVM
(-Xmx1500m) to be lesser to -Xmx1000m. This seems to have alleviated
the situation a little for the time being. It does not go down and
does not give the error as before. Also we do have to restart the
server on a daily basis to update the data.But with the new settings
it makes it easily through the entire day without giving an error. Can
anyone explain this ? Due to the growing traffic and data we will
probably need the expanded memory sometime in the future. We feel that
at the moment we have just pushed the problem to surface at some point
in the future.

Thanks,
Mit
Click to expand...

The Java process has two memory areas: the Java heap, and the "native heap",
which combine total the memory usage of the process.
The Java heap is controlled via the -Xms and -Xmx setting, and the space
available to the native heap is that which isn't used by the Java heap.

The act of reducing the maximum Java heap size has made the "native heap"
bigger, and this is the area that was memory constrained.
We know this because the OutOfMemoryError was generated the message informed
you that the JVM was unable to allocate a new native stack, this is
allocated onto the native heap (there is also a Java thread object which is
created and allocated onto the Java heap).

It is entirely possible that the amount of "native heap" available to the
JVM was insufficient to allocate the underlying resources to run the Java
process under the load that was being driven through it. The native heap is
now 500MB bigger, and unless there is a memory leak or the load is
significantly increased, this change should prevent any OutOfMemoryErrors
based on the native heap.
 
M

Moazam Raja

This could be a couple of different things.

What are the full JVM options you are running with? Do you have -Xms
specified also, and are you using any custom PermGen settings? You may
want to try setting -Xms to the same value as -Xmx.

Try using JVMSTAT on the process and check to see what is actually
happening in the VM (as per young generation, old generation, etc.).
JVMSTAT will help you visualize what is happening with memory inside the
VM. This will help you figure out what you need to tune.

To get jvmstat, http://developers.sun.com/dev/coolstuff/jvmstat/ .

Otherwise, if you just want to manually look at the garbage collection
information you can add flags such as '-verbose:gc' and
'XX:+PrintGCDetails'. You can also output the GC information to a file
using '-Xloggc:<file>'.

-Moazam

Mit said:
Hi,
We are running a web application which is a stateless system and
gets a lot of hits a day per server.
We are running the following configuration :
Red Hat Linux 7.3
Linux Kernel 2.4.20-19.7smp
Sun JVM JDK 1.4.1_02
The machine has 2GB of RAM and we have set the -XMx=1500M.

Pretty much everyday after about 200,000 - 250,000 hits, the
server get the error shown below. We have looked at the memory usage
and it looks ok.The application does load up some data and use a
substantial amount of memory(about 500M). But we looking through both
top and Runtime.getRuntime().totalMemory() we see that it has still
not close to the 1500M limit that we have set. Also we use the
following code on a jsp page to montior the threads and they don't
seem to be spiralling out of control

Thread[] threads = new Thread[1000];
ThreadGroup tg = Thread.currentThread().getThreadGroup();

while (tg.getParent() != null)
{
tg = tg.getParent();
}

tg.enumerate( threads );
for (int i = 0; i < threads.length; i++)
{
if (threads != null)
{
out.print(i + ": ");
out.print( "pri=" + threads.getPriority() + " ");
out.print( "ac=" + threads.activeCount() + " ");
out.println(threads);
}
}

Does anyone have any idea what the problem could be ? What the
possible issues could be ? We have looked at Memory and threads and
they don't seem to be the issue. Is here any documentation about the
cases that would through up this error ? Any help in this matter would
be greatly appreciated.

Thanks,
Mit








----- Root Cause -----
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start(Native Method)
at com.i411.search.controller.LowLevelSearch.performSocketSearch(LowLevelSearch.java:428)
at com.i411.search.controller.LowLevelSearch.performNetworkSearch(LowLevelSearch.java:348)
Click to expand...

...............
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

java.lang.NoClassDefFoundError: org/aspectj/lang/Signature 2
jsp error 0
servlet exception 5
jdbc on tomcat 0
JSP: org.apache.jasper.JasperException: Unable to compile class 0
java servlet newbie problem 3
Unable to load class using jstl on Tomcat 1
app vendor telling porkies? 6

Members online

Total: 37 (members: 1, guests: 36)
Robots: 209

Forum statistics

Threads
473,766
Messages
2,569,569
Members
45,045
Latest member
DRCM

Latest Threads

Top