Java 1.5_22 crash my server, please for help

Discussion in 'Java' started by Lothar Kimmeringer, Jan 26, 2010.

  1. Sherkan wrote:

    > After one day, two days of working in production, my server crash with
    > the error:
    >
    > #
    > # An unexpected error has been detected by HotSpot Virtual Machine:
    > #
    > # SIGSEGV (0xb) at pc=0xb7962d4f, pid=3307, tid=1355598736
    > #
    > # Java VM: Java HotSpot(TM) Server VM (1.5.0_22-b03 mixed mode)
    > # Problematic frame:
    > # V [libjvm.so+0x402d4f]

    [...]
    >
    > 2 days ago I had java newest java 1.6 with same problem.
    > I get still this message: # Problematic frame: # V [libjvm.so
    > +0x402d4f]


    Really? The value after libjvm.so is AFAIR the offset inside
    the library. That would mean that the two shared libraries were
    more or less identical after compilation.

    When searching the web for Cent OS and JVM-crashes, you can
    find a couple of discussions going on. Reinstalling the JDK
    might be worth a try. Download it directly from SUN (not the
    RPM but the selfextracting file) and set the JAVA_HOME-variable
    to that directory explicitly to make sure that it's really used.


    Regards, Lothar
    --
    Lothar Kimmeringer E-Mail:
    PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

    Always remember: The answer is forty-two, there can only be wrong
    questions!
    Lothar Kimmeringer, Jan 26, 2010
    #1
    1. Advertising

  2. Lothar Kimmeringer

    Roedy Green Guest

    On Mon, 25 Jan 2010 09:26:42 -0800 (PST), Sherkan <>
    wrote, quoted or indirectly quoted someone who said :

    ># An unexpected error has been detected by HotSpot Virtual Machine:


    This sounds like the sort of thing only Sun can fix.

    However, if this happens after only a few days, it could be some sort
    of slow memory leak. Is there a way you could monitor free memory to
    see if is gradually reducing?

    Do you have an exerciser -- something that simulates a curious user
    who exercises all your code. If it something unusual your code is
    doing, that way you could out what it was, and find some alternative
    way of doing it.

    What is centos? Who the made the JVM?
    --
    Roedy Green Canadian Mind Products
    http://mindprod.com
    Computers are useless. They can only give you answers.
    ~ Pablo Picasso (born: 1881-10-25 died: 1973-04-08 at age: 91)
    Roedy Green, Jan 28, 2010
    #2
    1. Advertising

  3. In article <>,
    Roedy Green <> wrote:

    > What is centos?


    <http://www.centos.org/> is a RedHat clone with spotty maintenance:

    <http://linux.slashdot.org/story/09/08/01/1443221/CentOS-Administrator-Reappears>

    IMO, RedHat's low annual fees are a relative bargain:

    <http://gigaom.com/2010/01/22/how-red-hat-routed-the-recession/>

    --
    John B. Matthews
    trashgod at gmail dot com
    <http://sites.google.com/site/drjohnbmatthews>
    John B. Matthews, Jan 28, 2010
    #3
  4. Lothar Kimmeringer

    Lew Guest

    Sherkan wrote, quoted or indirectly quoted someone who said :
    >> # An unexpected error has been detected by HotSpot Virtual Machine:
    >> #
    >> # SIGSEGV (0xb) at pc=0xb7962d4f, pid=3307, tid=1355598736
    >> #
    >> # Java VM: Java HotSpot(TM) Server VM (1.5.0_22-b03 mixed mode)
    >> # Problematic frame:
    >> # V [libjvm.so+0x402d4f]


    Roedy Green wrote:
    > This sounds like the sort of thing only Sun can fix.
    >
    > However, if this happens after only a few days, it could be some sort
    > of slow memory leak. Is there a way you could monitor free memory to
    > see if is gradually reducing?


    SIGSEGV is not usually a result of a slow memory leak. It's an illegal access
    to a section of memory, in this case by the JVM itself. Monitoring free
    memory as seen by the application itself probably won't help, and I really
    can't see how monitoring free memory in the OS would help either.

    If the problem were a memory leak or insufficient memory allocated to the
    heap, you wouldn't get a SIGSEGV.

    The problem is nearly impossible to diagnose from Usenet, although if we knew
    the code layout of libjvm.so and had source we could probably figure out which
    instruction was trying to access memory illegally.

    Sherkan wrote:
    >> jvm_args:


    Next time make it easier to read this by formatting it for Usenet. It's hard
    to follow all these complicated options when line breaks interrupt in the
    middle of an argument. (I tried to neaten up here for citation.)

    >> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
    >> -Xms1024m -Xmx1024m -XX:permSize=256m -XX:MaxPermSize=512m
    >> -XX:MinHeapFreeRatio=40 -XX:MaxHeapFreeRatio=70 -XX:NewRatio=3
    >> -XX:MaxNewSize=512m


    Some interesting choices here for your JVM arguments. Why these values? In
    particular, aren't 40 an 70 the default values for the min and max HeapFreeRatios?

    >> -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseParNewGC
    >> -XX:+CMSPermGenSweepingEnabled
    >> -XX:+ClassUnloading -XX:+CMSClassUnloadingEnabled
    >> -XX:CompileThreshold=100 -XX:CompileThreshold=100


    Why is CompileThreshold here twice? Or at all?

    >> -XX:+PrintGCDetails -verbose:gc -Xloggc:/serwer/jvm


    "serwer" is an interesting spelling.

    >> -Djava.util.logging.config.file=/serwer/tomcat6/conf/logging.properties
    >> -Djava.endorsed.dirs=/serwer/tomcat6/endorsed
    >> -Dcatalina.base=/serwer/tomcat6 -Dcatalina.home=/serwer/tomcat6


    You don't actually need to specify $CATALINA_BASE if it's the same as
    $CATALINA_HOME.

    >> -Djava.io.tmpdir=/serwer/tomcat6/temp


    Moving on:
    > Environment Variables:
    > LD_LIBRARY_PATH=/serwer/install/jdk1.5.0_22/jre/lib/i386/server:
    > /serwer/install/jdk1.5.0_22/jre/lib/i386:
    > /serwer/install/jdk1.5.0_22/jre/../lib/i386


    I don't see the relevance of LD_LIBRARY_PATH. That's something for C
    programs, not Java. Are you using JNI? If so, why, in a Tomcat context?

    > 2 days ago I had java [sic] newest java 1.6 with same problem.
    > I get still this message:
    > # Problematic frame:
    > # V [libjvm.so+0x402d4f]

    (Line wrap fixed up - you really need to manage that.)

    Somehow that's just waaaaay too much of a coincidence. You might be accessing
    the wrong libjvm.so.

    --
    Lew
    Lew, Jan 29, 2010
    #4
  5. In article
    <>,
    Sherkan <> wrote:

    > -XX:CompileThreshold=100


    To my knowledge, there is no version of Java 1.5 that is free of serious
    HotSpot bugs. Setting the compilation threshold so low almost
    guarantees that it's going to crash or unroll loops incorrectly.

    Java 1.6 seems better (so far) but I still wouldn't set the compilation
    threshold lower than 10000 for a server. With a value of 100, your 512MB
    Perm Heap may be completely consumed during initialization, leaving your
    main code paths interpreted.
    --
    I won't see Google Groups replies because I must filter them as spam
    Kevin McMurtrie, Jan 29, 2010
    #5
  6. Lothar Kimmeringer

    Sherkan Guest

    Hello guys,
    Thank you for reply for my email.
    I need to explain one think.
    I got 3 apps on tomcat 6, based on:

    first and second: Seam 2.0.1GA, Facelets, JSF, RichFaces, jdbc
    third is on Seam 2.2.0GA, facelets, JSF, Richfaces, jdbc

    All app jar libs are in each app WEB-INF/lib directory.
    One month ago my first and second apps worked alone withouth third, on
    newest centos 64bit, Core 2 Duo 3GHz, 4 gb ram, and postgres 8.3,
    I had to change datacenter and it all works right now, on newest 32bit
    centos on Quad Core 2,5Ghz and 8GB ram.
    Don't ask me why not 64bit (this is long story) :(

    The first time my apps worked on this configuration for over 2 years.
    Server worked fine for months without rebooting. Right now on the same
    configuration, same tomcat, same postgres and even driver this f***
    server can't hold 2 days.
    It crash randomly, 20minutes after start, 5 hours, 10 hours. I even
    don't know what crashes my whole server so randomly.

    Yesterday I loose whole day to profile my all apps, with yourkit
    profiler, and it all looks fine without memory leaks, and also if
    there could be memory leak, why server crash 20 minutes after start,
    there could be huge leak, but it should get an error message like:
    java heap space or perm gen space, not this one.

    OK here are my changes:
    1. I have checked it there is no new libraries of seam, jdbc etc.
    2. I've changed JSF state in web.xml from server in all aps to client.
    3. I've raised -XX:NewSize=512m -XX:MaxNewSize=512m"
    from 256 to 512 and I see little difference.
    Today server works for all day without problems, and I see in jvm
    logs, my machine start FULL GC...

    68504.347: [Full GC 68504.347: [ParNew: 1334K->24K(471872K), 0.0056640
    secs] 215448K->214137K(996160K), 0.0058180 secs]

    In previous logs I couldn't find such an FULL GC log.
    but there is also new problem, why I get this FULL gc log sometimes?

    72356.821: [Full GC 72356.821: [ParNew: 419456K->419456K(471872K),
    0.0000080 secs] 634061K->634061K(996160K), 0.0000940 secs]
    GC locker: Trying a full collection because scavenge failed
    72356.821: [Full GC 72356.821: [Tenured: 214604K->217917K(524288K),
    0.7640090 secs] 634061K->217917K(996160K), [Perm : 69328K->69328K
    (262144K)], 0.7641500 secs]


    Here are my complete server parameters:
    JAVA_OPTS="$JAVA_OPTS -server -Xms1024m -Xmx1024m -XX:permSize=256m -
    XX:MaxPermSize=512m"
    JAVA_OPTS="$JAVA_OPTS -XX:MinHeapFreeRatio=40 -
    XX:MaxHeapFreeRatio=70"
    JAVA_OPTS="$JAVA_OPTS -XX:NewRatio=3 -XX:NewSize=512m -
    XX:MaxNewSize=512m"
    JAVA_OPTS="$JAVA_OPTS -Dsun.rmi.dgc.server.gcInterval=3600000"
    JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC -XX:
    +CMSPermGenSweepingEnabled"
    JAVA_OPTS="$JAVA_OPTS -XX:+ClassUnloading -XX:
    +CMSClassUnloadingEnabled -XX:CompileThreshold=100"
    JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails -verbose:gc -Xloggc:/
    serwer/jvm"
    JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"

    Yes there was twice compile. I set this to 100 because I found on one
    web site author offer this to set to minimum to force to compile to
    native code very fast classes.

    JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails -verbose:gc -Xloggc:/
    serwer/jvm"

    /serwer/jvm - this is my centos jvm.log file path for gc logs.
    OK i will remove compilethreeshold in the next time, and we so how it
    works.
    I hope all java opts don't break next time.
    Sherkan, Jan 29, 2010
    #6
  7. Lothar Kimmeringer

    jolz Guest

    > If the problem were a memory leak or insufficient memory allocated to
    > the heap, you wouldn't get a SIGSEGV.


    If memory leaks in native code malloc will eventually return NULL (or
    more likely bad_alloc will be thrown). In 32 bit system it may be really
    fast, especially if Xmx is set to maximum possible value. Sun's JVM and
    JRockit and possibly all other JVM ignore malloc failures and will
    crash. So yes, memory leak may cause SIGSEGV.
    jolz, Jan 29, 2010
    #7
  8. Lothar Kimmeringer

    jolz Guest

    > It crash randomly, 20minutes after start, 5 hours, 10 hours. I even
    > don't know what crashes my whole server so randomly.


    Are you sure it's a software problem? Very often faulty hardware
    (possibly RAM) causes similar behaviour.
    jolz, Jan 29, 2010
    #8
  9. Lothar Kimmeringer

    Sherkan Guest

    On 30 Sty, 00:30, jolz <> wrote:
    > > It crash randomly, 20minutes after start, 5 hours, 10 hours. I even
    > > don't know what crashes my whole server so randomly.

    >
    > Are you sure it's a software problem? Very often faulty hardware
    > (possibly RAM) causes similar behaviour.


    I really don't know:(
    So why on 32bit system there will not be thrown exception java heap
    space if there is memory leak instead of SIGSEGV?
    Sherkan, Jan 30, 2010
    #9
  10. Lothar Kimmeringer

    jolz Guest

    > So why on 32bit system there will not be thrown exception java heap
    > space if there is memory leak instead of SIGSEGV?


    Sun's JVM will throw OutOfMemoryError if memory limit for java was hit
    but will crash if memory limit for the process will be reached. On 64
    bit system it's hard to reach that limit (unless the disc is full and
    there's no place for swap), but on 32 bit system native code may have
    very little memory to use (for example yours 1024m (-Xmx) + 512m
    (-XX:MaxPermSize) + memory for used by JVM will leave very little from
    2G (depends on the OS) available for a process). If memory leaks all
    available memory will be used very soon.
    jolz, Jan 30, 2010
    #10
  11. Lothar Kimmeringer

    Sherkan Guest

    OK,
    2 days ago I've started server and it worked for about 1,5 day....10
    minutes ago I had to change something application, after reboot my
    server worked for 4 minutes!
    Why server crash after 4 minutes of work?
    Sherkan, Jan 30, 2010
    #11
  12. Lothar Kimmeringer

    Lew Guest

    Sherkan wrote:
    > 2 days ago I've started server and it worked for about 1,5 day....10
    > minutes ago I had to change something application, after reboot my
    > server worked for 4 minutes!
    > Why server crash after 4 minutes of work?


    Because the conditions that cause the crash manifested after four minutes.

    I'd be more specific, but my crystal ball is in the shop.

    --
    Lew
    Lew, Jan 30, 2010
    #12
  13. Lothar Kimmeringer

    Arne Vajhøj Guest

    On 28-01-2010 07:40, Roedy Green wrote:
    > What is centos? Who the made the JVM?


    May I suggest there two wonderful services:

    http://www.google.com/
    http://www.wikipedia.org/

    They are absolutely excellent to answer that type
    of questions.

    Arne
    Arne Vajhøj, Jan 31, 2010
    #13
  14. In article <hk21og$hsk$>,
    jolz <> wrote:

    > > So why on 32bit system there will not be thrown exception java heap
    > > space if there is memory leak instead of SIGSEGV?

    >
    > Sun's JVM will throw OutOfMemoryError if memory limit for java was hit
    > but will crash if memory limit for the process will be reached. On 64
    > bit system it's hard to reach that limit (unless the disc is full and
    > there's no place for swap), but on 32 bit system native code may have
    > very little memory to use (for example yours 1024m (-Xmx) + 512m
    > (-XX:MaxPermSize) + memory for used by JVM will leave very little from
    > 2G (depends on the OS) available for a process). If memory leaks all
    > available memory will be used very soon.


    I don't know about CentOS, but many OSes log malloc/free errors. It
    helps debug a problem that may have deferred symptoms.
    --
    I won't see Google Groups replies because I must filter them as spam
    Kevin McMurtrie, Jan 31, 2010
    #14
  15. Lothar Kimmeringer

    Sherkan Guest

    Very funny Arne,

    I've added remove client to my server of yourkit profiler and I
    starded profiling my app remotly.
    It seems that something is going on with memory or system.

    Sometimes if I force full GC from the YP server crash.
    Yourkit profler also shows that is less than 200mb of 1gb in use of
    heap and 51mb of non heap memory.
    So this is NOT MEMORY LEAK i know how to use profiler.

    Yesterday I've tryed to start tomcat with PararelGc, CMS, serial types
    of GC.
    This happen in all GC's.
    Sherkan, Feb 1, 2010
    #15
  16. Lothar Kimmeringer

    Sherkan Guest

    sorry remote* not remove
    Sherkan, Feb 1, 2010
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John
    Replies:
    3
    Views:
    461
  2. Replies:
    4
    Views:
    506
    Chris Uppal
    May 5, 2005
  3. KK
    Replies:
    2
    Views:
    550
    Big Brian
    Oct 14, 2003
  4. MuZZy
    Replies:
    7
    Views:
    1,745
    Mike Hewson
    Jan 7, 2005
  5. Stevie_mac
    Replies:
    7
    Views:
    482
    Richard James
    Apr 12, 2004
Loading...

Share This Page