Java 1.5_22 crash my server, please for help

[Lothar Kimmeringer]

After one day, two days of working in production, my server crash with
the error:

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0xb7962d4f, pid=3307, tid=1355598736
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_22-b03 mixed mode)
# Problematic frame:
# V [libjvm.so+0x402d4f] [...]

2 days ago I had java newest java 1.6 with same problem.
I get still this message: # Problematic frame: # V [libjvm.so
+0x402d4f]

Really? The value after libjvm.so is AFAIR the offset inside
the library. That would mean that the two shared libraries were
more or less identical after compilation.

When searching the web for Cent OS and JVM-crashes, you can
find a couple of discussions going on. Reinstalling the JDK
might be worth a try. Download it directly from SUN (not the
RPM but the selfextracting file) and set the JAVA_HOME-variable
to that directory explicitly to make sure that it's really used.


Regards, Lothar
--
Lothar Kimmeringer E-Mail: (e-mail address removed)
PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
questions!

 

[Roedy Green]

# An unexpected error has been detected by HotSpot Virtual Machine:

This sounds like the sort of thing only Sun can fix.

However, if this happens after only a few days, it could be some sort
of slow memory leak. Is there a way you could monitor free memory to
see if is gradually reducing?

Do you have an exerciser -- something that simulates a curious user
who exercises all your code. If it something unusual your code is
doing, that way you could out what it was, and find some alternative
way of doing it.

What is centos? Who the made the JVM?

 

[John B. Matthews]

What is centos?

<http://www.centos.org/> is a RedHat clone with spotty maintenance:

<http://linux.slashdot.org/story/09/08/01/1443221/CentOS-Administrator-Reappears>

IMO, RedHat's low annual fees are a relative bargain:

<http://gigaom.com/2010/01/22/how-red-hat-routed-the-recession/>

 

[Lew]

Sherkan wrote, quoted or indirectly quoted someone who said :

# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0xb7962d4f, pid=3307, tid=1355598736
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_22-b03 mixed mode)
# Problematic frame:
# V [libjvm.so+0x402d4f]

This sounds like the sort of thing only Sun can fix.

However, if this happens after only a few days, it could be some sort
of slow memory leak. Is there a way you could monitor free memory to
see if is gradually reducing?

SIGSEGV is not usually a result of a slow memory leak. It's an illegal access
to a section of memory, in this case by the JVM itself. Monitoring free
memory as seen by the application itself probably won't help, and I really
can't see how monitoring free memory in the OS would help either.

If the problem were a memory leak or insufficient memory allocated to the
heap, you wouldn't get a SIGSEGV.

The problem is nearly impossible to diagnose from Usenet, although if we knew
the code layout of libjvm.so and had source we could probably figure out which
instruction was trying to access memory illegally.

Next time make it easier to read this by formatting it for Usenet. It's hard
to follow all these complicated options when line breaks interrupt in the
middle of an argument. (I tried to neaten up here for citation.)

Some interesting choices here for your JVM arguments. Why these values? In
particular, aren't 40 an 70 the default values for the min and max HeapFreeRatios?

Why is CompileThreshold here twice? Or at all?

"serwer" is an interesting spelling.

You don't actually need to specify $CATALINA_BASE if it's the same as
$CATALINA_HOME.

Moving on:

Environment Variables:
LD_LIBRARY_PATH=/serwer/install/jdk1.5.0_22/jre/lib/i386/server:
/serwer/install/jdk1.5.0_22/jre/lib/i386:
/serwer/install/jdk1.5.0_22/jre/../lib/i386

I don't see the relevance of LD_LIBRARY_PATH. That's something for C
programs, not Java. Are you using JNI? If so, why, in a Tomcat context?

2 days ago I had java [sic] newest java 1.6 with same problem.
I get still this message:
# Problematic frame:
# V [libjvm.so+0x402d4f]

(Line wrap fixed up - you really need to manage that.)

Somehow that's just waaaaay too much of a coincidence. You might be accessing
the wrong libjvm.so.

 

[Kevin McMurtrie]

-XX:CompileThreshold=100

To my knowledge, there is no version of Java 1.5 that is free of serious
HotSpot bugs. Setting the compilation threshold so low almost
guarantees that it's going to crash or unroll loops incorrectly.

Java 1.6 seems better (so far) but I still wouldn't set the compilation
threshold lower than 10000 for a server. With a value of 100, your 512MB
Perm Heap may be completely consumed during initialization, leaving your
main code paths interpreted.

 

[Sherkan]

Hello guys,
Thank you for reply for my email.
I need to explain one think.
I got 3 apps on tomcat 6, based on:

first and second: Seam 2.0.1GA, Facelets, JSF, RichFaces, jdbc
third is on Seam 2.2.0GA, facelets, JSF, Richfaces, jdbc

All app jar libs are in each app WEB-INF/lib directory.
One month ago my first and second apps worked alone withouth third, on
newest centos 64bit, Core 2 Duo 3GHz, 4 gb ram, and postgres 8.3,
I had to change datacenter and it all works right now, on newest 32bit
centos on Quad Core 2,5Ghz and 8GB ram.
Don't ask me why not 64bit (this is long story)

The first time my apps worked on this configuration for over 2 years.
Server worked fine for months without rebooting. Right now on the same
configuration, same tomcat, same postgres and even driver this f***
server can't hold 2 days.
It crash randomly, 20minutes after start, 5 hours, 10 hours. I even
don't know what crashes my whole server so randomly.

Yesterday I loose whole day to profile my all apps, with yourkit
profiler, and it all looks fine without memory leaks, and also if
there could be memory leak, why server crash 20 minutes after start,
there could be huge leak, but it should get an error message like:
java heap space or perm gen space, not this one.

OK here are my changes:
1. I have checked it there is no new libraries of seam, jdbc etc.
2. I've changed JSF state in web.xml from server in all aps to client.
3. I've raised -XX:NewSize=512m -XX:MaxNewSize=512m"
from 256 to 512 and I see little difference.
Today server works for all day without problems, and I see in jvm
logs, my machine start FULL GC...

68504.347: [Full GC 68504.347: [ParNew: 1334K->24K(471872K), 0.0056640
secs] 215448K->214137K(996160K), 0.0058180 secs]

In previous logs I couldn't find such an FULL GC log.
but there is also new problem, why I get this FULL gc log sometimes?

72356.821: [Full GC 72356.821: [ParNew: 419456K->419456K(471872K),
0.0000080 secs] 634061K->634061K(996160K), 0.0000940 secs]
GC locker: Trying a full collection because scavenge failed
72356.821: [Full GC 72356.821: [Tenured: 214604K->217917K(524288K),
0.7640090 secs] 634061K->217917K(996160K), [Perm : 69328K->69328K
(262144K)], 0.7641500 secs]


Here are my complete server parameters:
JAVA_OPTS="$JAVA_OPTS -server -Xms1024m -Xmx1024m -XXermSize=256m -
XX:MaxPermSize=512m"
JAVA_OPTS="$JAVA_OPTS -XX:MinHeapFreeRatio=40 -
XX:MaxHeapFreeRatio=70"
JAVA_OPTS="$JAVA_OPTS -XX:NewRatio=3 -XX:NewSize=512m -
XX:MaxNewSize=512m"
JAVA_OPTS="$JAVA_OPTS -Dsun.rmi.dgc.server.gcInterval=3600000"
JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC -XX:
+CMSPermGenSweepingEnabled"
JAVA_OPTS="$JAVA_OPTS -XX:+ClassUnloading -XX:
+CMSClassUnloadingEnabled -XX:CompileThreshold=100"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails -verbose:gc -Xloggc:/
serwer/jvm"
JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"

Yes there was twice compile. I set this to 100 because I found on one
web site author offer this to set to minimum to force to compile to
native code very fast classes.

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails -verbose:gc -Xloggc:/
serwer/jvm"

/serwer/jvm - this is my centos jvm.log file path for gc logs.
OK i will remove compilethreeshold in the next time, and we so how it
works.
I hope all java opts don't break next time.

 

[jolz]

If the problem were a memory leak or insufficient memory allocated to

the heap, you wouldn't get a SIGSEGV.

If memory leaks in native code malloc will eventually return NULL (or
more likely bad_alloc will be thrown). In 32 bit system it may be really
fast, especially if Xmx is set to maximum possible value. Sun's JVM and
JRockit and possibly all other JVM ignore malloc failures and will
crash. So yes, memory leak may cause SIGSEGV.

 

[jolz]

It crash randomly, 20minutes after start, 5 hours, 10 hours. I even

don't know what crashes my whole server so randomly.

Are you sure it's a software problem? Very often faulty hardware
(possibly RAM) causes similar behaviour.

 

[Sherkan]

Are you sure it's a software problem? Very often faulty hardware
(possibly RAM) causes similar behaviour.

I really don't know
So why on 32bit system there will not be thrown exception java heap
space if there is memory leak instead of SIGSEGV?

 

[jolz]

So why on 32bit system there will not be thrown exception java heap

space if there is memory leak instead of SIGSEGV?

Sun's JVM will throw OutOfMemoryError if memory limit for java was hit
but will crash if memory limit for the process will be reached. On 64
bit system it's hard to reach that limit (unless the disc is full and
there's no place for swap), but on 32 bit system native code may have
very little memory to use (for example yours 1024m (-Xmx) + 512m
(-XX:MaxPermSize) + memory for used by JVM will leave very little from
2G (depends on the OS) available for a process). If memory leaks all
available memory will be used very soon.

 

[Sherkan]

OK,
2 days ago I've started server and it worked for about 1,5 day....10
minutes ago I had to change something application, after reboot my
server worked for 4 minutes!
Why server crash after 4 minutes of work?

 

[Lew]

2 days ago I've started server and it worked for about 1,5 day....10
minutes ago I had to change something application, after reboot my
server worked for 4 minutes!
Why server crash after 4 minutes of work?

Because the conditions that cause the crash manifested after four minutes.

I'd be more specific, but my crystal ball is in the shop.

 

[Arne Vajhøj]

What is centos? Who the made the JVM?

May I suggest there two wonderful services:

http://www.google.com/
http://www.wikipedia.org/

They are absolutely excellent to answer that type
of questions.

Arne

 

[Kevin McMurtrie]

So why on 32bit system there will not be thrown exception java heap
space if there is memory leak instead of SIGSEGV?

Sun's JVM will throw OutOfMemoryError if memory limit for java was hit
but will crash if memory limit for the process will be reached. On 64
bit system it's hard to reach that limit (unless the disc is full and
there's no place for swap), but on 32 bit system native code may have
very little memory to use (for example yours 1024m (-Xmx) + 512m
(-XX:MaxPermSize) + memory for used by JVM will leave very little from
2G (depends on the OS) available for a process). If memory leaks all
available memory will be used very soon.[/QUOTE]

I don't know about CentOS, but many OSes log malloc/free errors. It
helps debug a problem that may have deferred symptoms.

 

[Sherkan]

Very funny Arne,

I've added remove client to my server of yourkit profiler and I
starded profiling my app remotly.
It seems that something is going on with memory or system.

Sometimes if I force full GC from the YP server crash.
Yourkit profler also shows that is less than 200mb of 1gb in use of
heap and 51mb of non heap memory.
So this is NOT MEMORY LEAK i know how to use profiler.

Yesterday I've tryed to start tomcat with PararelGc, CMS, serial types
of GC.
This happen in all GC's.

 

[Sherkan]

sorry remote* not remove