Help: Tomcat crashes on java.lang.OutOfMemoryError

[Cathy Hui]

I am running an application (i.e. Remedy Midtier) on Tomcat. Currently
I run into the "java.lang.OutOfMemoryError" when performing some
activities from the browser when accessing this application.
Basically, the Tomcat crashes when the java process takes about 160MB -
190MB memory usage (I did monitor the "top" for system resources)

Background information:

OS: Solaris 8
System physical memory: 2048MB
Tomcat version: 4.1.29
JDK version: 1.4.1

/etc/system

set rlim_fd_cur = 65536
set rlim_fd_max = 65536

This is what I have tried so far:

1) Double the heap size in catalina.sh to allocate to java from
"-Xms256m -Xmx512m" to "-Xms512m -Xmx1024m"

2) install jdk 1.4.2

However, the result is: JAVA still fails after taking 160MB-190MB
memory and Tomcat crashes on java.lang.OutOfMemoryError.

Does anyone know what might be the cause of it and how to fix it?

Thanks!

 

[Gerbrand van Dieijen]

Cathy Hui schreef:

I am running an application (i.e. Remedy Midtier) on Tomcat. Currently
I run into the "java.lang.OutOfMemoryError" when performing some
activities from the browser when accessing this application.
Basically, the Tomcat crashes when the java process takes about 160MB -
190MB memory usage (I did monitor the "top" for system resources)

Hello,

You might try to use a profiler.
I'm recently used JProfiler (http://www.jprofiler.com) that was quite
easy to use.
You can view the heap-memory and see what objects are created and/or
garbage collected. If there's steady increase, this might indicate a
memory-leak.

Also, you might look into the source code for code that might use a lot
of memory without freeing it (well pretty obvious maybe). Static objects
(lists) might be a problem. Also, objects that are stored in sessions
could be a problem, if there are a lot of users to the webapplication.

 

[Cathy Hui]

More info: This is the catalina.out with the error log. Any idea of
why it's failing?

====================================================

loaded libraryException in thread "ARTID~0" java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError
Exception in thread "ARTID~0" java.lang.OutOfMemoryError

Unexpected Signal : 11 occurred at PC=0xFE1A03FC
Function=[Unknown. Nearest: JVM_Clone+0xD9DC]
Library=/usr/j2se/jre/lib/sparc/client/libjvm.so

Current Java thread:
at com.remedy.arsys.api.Proxy.ARGetListActiveLinkObjects(Native
Method)
at
com.remedy.arsys.api.ActiveLinkFactory.findObjects(ActiveLinkFactory.java:142)
at
com.remedy.arsys.cache.definition.ActiveLinkCache.getActiveLinkListFromServer(Unknown
Source)
at
com.remedy.arsys.cache.definition.ActiveLinkCache.populateCacheWithDefinitions(Unknown
Source)
at
com.remedy.arsys.cache.definition.ActiveLinkCache.getActiveLinkList(Unknown
Source)
at
com.remedy.arsys.cache.definition.ActiveLinkCache.preLoadObjects(Unknown
Source)
at com.remedy.arsys.cache.Cache.preLoadObjects(Unknown Source)
at
com.remedy.arsys.cache.definition.SchemaCache.CreateSchemaDefinitions(Unknown
Source)
at
com.remedy.arsys.cache.definition.SchemaCache.getSchemaDefinition(Unknown
Source)
at
com.remedy.arsys.cache.definition.SchemaCache.getSchemaViewDescriptorMap(Unknown
Source)
at
com.remedy.arsys.cache.Cache.getSchemaViewDescriptorMap(Unknown Source)
at com.remedy.arsys.appmgr.Application.getLocaleList(Unknown
Source)
at com.remedy.arsys.appmgr.Application.deployFromCache(Unknown
Source)
- locked <f2f77248> (a com.remedy.arsys.appmgr.Application)
at com.remedy.arsys.appmgr.AppManager.deployApp(Unknown Source)
- locked <f2d75508> (a com.remedy.arsys.appmgr.AppManager)
at
com.remedy.arsys.config.ConfigServlet.deployApplications(Unknown
Source)
at
com.remedy.arsys.config.ConfigServlet.deployApplication(Unknown Source)
at com.remedy.arsys.config.ConfigServlet.doPost(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at
org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:457)
at
org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:576)
at java.lang.Thread.run(Thread.java:536)

Dynamic libraries:
0x10000 /usr/j2se/bin/java
0xff360000 /usr/lib/libthread.so.1
0xff3a0000 /usr/lib/libdl.so.1
0xff200000 /usr/lib/libc.so.1
0xff330000 /usr/platform/SUNW,Serverblade1/lib/libc_psr.so.1
0xfe000000 /usr/j2se/jre/lib/sparc/client/libjvm.so
0xff2e0000 /usr/lib/libCrun.so.1
0xff1e0000 /usr/lib/libsocket.so.1
0xff100000 /usr/lib/libnsl.so.1
0xff0d0000 /usr/lib/libm.so.1
0xff310000 /usr/lib/libw.so.1
0xff0b0000 /usr/lib/libmp.so.2
0xff080000 /usr/j2se/jre/lib/sparc/native_threads/libhpi.so
0xff050000 /usr/j2se/jre/lib/sparc/libverify.so
0xfe7c0000 /usr/j2se/jre/lib/sparc/libjava.so
0xff030000 /usr/j2se/jre/lib/sparc/libzip.so
0xf1f90000 /usr/j2se/jre/lib/sparc/libnet.so
0xede00000
/export/home/local/jakarta-tomcat-4.1.29/webapps/arsys/WEB-INF/lib/libarutiljni60.so
0xefb60000 /usr/lib/libpthread.so.1
0xef180000
/export/home/local/jakarta-tomcat-4.1.29/webapps/arsys/WEB-INF/lib/libarjni60.so

Local Time = Tue May 31 23:13:55 2005
Elapsed Time = 1052
#
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002E6 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Java VM: Java HotSpot(TM) Client VM (1.4.1_02-b06 mixed mode)
#
# An error report file has been saved as hs_err_pid1184.log.
# Please refer to the file for further information.
#

 

[Cathy Hui]

Another piece of useful information is. While the java memory usage is
trap in range 160-190MB, the CPU usage is going up to from very minimal
to > 95% before Tomcat crashes.

 

[.]

I am running an application (i.e. Remedy Midtier) on Tomcat. Currently
I run into the "java.lang.OutOfMemoryError" when performing some
activities from the browser when accessing this application.
Basically, the Tomcat crashes when the java process takes about 160MB -
190MB memory usage (I did monitor the "top" for system resources)

Try profiling it. JProbe from Quest Software has a profiler, memory
debugger and coverage tool. It integrates with Tomcat 4.x. You can get a
trial version at http://www.quest.com/jprobe/try.asp

 

[Gerbrand van Dieijen]

Cathy Hui schreef:

Another piece of useful information is. While the java memory usage is
trap in range 160-190MB, the CPU usage is going up to from very minimal
to > 95% before Tomcat crashes.

I think the best way to go is using a profiler and examine the objects
created as well as reviewing the code.

Do you use Threads inside the servlets code too? These might be the
cause of the problem as well.

 

[Wibble]

I am running an application (i.e. Remedy Midtier) on Tomcat. Currently
I run into the "java.lang.OutOfMemoryError" when performing some
activities from the browser when accessing this application.
Basically, the Tomcat crashes when the java process takes about 160MB -
190MB memory usage (I did monitor the "top" for system resources)

Background information:

OS: Solaris 8
System physical memory: 2048MB
Tomcat version: 4.1.29
JDK version: 1.4.1

/etc/system

set rlim_fd_cur = 65536
set rlim_fd_max = 65536

This is what I have tried so far:

1) Double the heap size in catalina.sh to allocate to java from
"-Xms256m -Xmx512m" to "-Xms512m -Xmx1024m"

2) install jdk 1.4.2

However, the result is: JAVA still fails after taking 160MB-190MB
memory and Tomcat crashes on java.lang.OutOfMemoryError.

Does anyone know what might be the cause of it and how to fix it?

Thanks!

We had these too and none of the profilers found it... it was the Perm
memory. Use the following command line switch (you may need more or less)

-XX:MaxPermSize=64m

 

[John C. Bollinger]

Another piece of useful information is. While the java memory usage is
trap in range 160-190MB, the CPU usage is going up to from very minimal
to > 95% before Tomcat crashes.

It all sounds consistent with the VM hitting the heap size bound and
going into GC thrashing before being forced to give up. It is possible
that the 160-190 MB is coincidental. For instance, if your program
happens for some reason to reliably try to allocate a 1 GB array when
the memory consumption is at that level then raising the heap limit from
512 MB to 1024 MB would not change the symptoms. This would also be
consistent with the observation that a 512 MB heap limit was not enough,
despite the fact that the observed consumption was less than 200 MB.