JMX OutOfMemoryError: Requested array size exceeds VM limit

Discussion in 'Java' started by mikesmith813@gmail.com, Dec 17, 2013.

  1. Guest

    I have a Java server application that consistantly throws an "OutOfMemoryError: Requested array size exceeds VM limit" and crashes about every 70 hours.

    I'm running on Java 1.7 release 45 with a beefy 18 CPU Linux Server with lots of memory. My max heap is set to 50GB and using the new G1GC collector.Profiling with JVisualVM shows that the app is not using more than half of the heap ever and PermSpace looks fine.

    Clearly, something is trying to allocate an array of an incorrect huge sizebut I can't determine what. My code creates objects and sends across JMX but the objects are already allocated in memory and handed to JMX at Exception time. I'm wondering if this is a JMX bug but I cannot find anything online. Any thoughts/ideas are appreciated! Stack trace below:


    Job_Executor6:class com.sun.jmx.remote.opt.util.JobExecutor java.lan.OutOfMemoryError: Requested array size exceeds VM limit
    at java.util.Arrays.copyOf(Unknown Source)
    at java.io.ByteArrayOutputStream.grow(Unknown Source)
    at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
    at java.io.ByteArrayOutputStream.write(Unknown Source)
    at java.io_ObjectOutputStream$BlockDataOutputStream.drain(Unknown Source)
    at java.io_ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(Unknown Source)
    at java.io_ObjectOutputStream.writeObject0(Unknown Source)
    at java.io_ObjectOutputStream.writeObject(Unknown Source)
    at com.sun.jmx.remote.generic.ObjectWrappingImpl.wrap(ObjectWrappingImpl.java:30)
    at java.management.remote.generic.ServerIntermediary$RequestHandler.handleNotifyReqMessage(ServerIntermediary.java:683)
    at java.management.remote.generic.ServerIntermediary$RequestHandler.execute(ServerIntermediary.java:626)
    at java.management.remote.generic.ServerSynchroMessageConnectionImpl$RemoteJob.run(ServerSynchroMessageConnectionImpl.java:266)
    at com.sun.jmx.remote.opt.util.ThreadService$ThreadServiceJob.run(ThreadService.java:208)
    at com.sun.jmx.remote.opt.util.JobExecutor.run(JobExecutor.java:59)
     
    , Dec 17, 2013
    #1
    1. Advertising

  2. Stefan Ram Guest

    writes:
    >Job_Executor6:class com.sun.jmx.remote.opt.util.JobExecutor java.lan.OutOfMemoryError: Requested array size exceeds VM limit


    There is a »g« missing in ».lan.« above. Possibly, the error
    message was edited by someone? Possibly by a human being?

    Hearing that it crashed after about 70 hours, my first idea
    was a growing array. The stack partially confirms this, but
    unexpectedly, it is not a java.util.ArrayList, but a
    java.io.ByteArrayOutputStream.

    >at java.util.Arrays.copyOf(Unknown Source)
    >at java.io.ByteArrayOutputStream.grow(Unknown Source)
    >at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
    >at java.io.ByteArrayOutputStream.write(Unknown Source)


    The next lines look like someone is trying to serialize.

    >at java.io_ObjectOutputStream$BlockDataOutputStream.drain(Unknown Source)
    >at java.io_ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(Unknown Source)
    >at java.io_ObjectOutputStream.writeObject0(Unknown Source)
    >at java.io_ObjectOutputStream.writeObject(Unknown Source)


    Possible that someone is trying to write a VLO (very large
    object) and during serialization, there is not enough memory
    for some array buffering the output. The VLO might grow
    during about 70 hours until it eventually has grown too
    large for serialization. It might contain an array itself
    not directly visible in the stack that has grown that much.
     
    Stefan Ram, Dec 17, 2013
    #2
    1. Advertising

  3. Eric Sosman Guest

    On 12/17/2013 4:32 PM, wrote:
    > I have a Java server application that consistantly throws an "OutOfMemoryError: Requested array size exceeds VM limit" and crashes about every 70 hours.
    >
    > I'm running on Java 1.7 release 45 with a beefy 18 CPU Linux Server with lots of memory. My max heap is set to 50GB and using the new G1GC collector. Profiling with JVisualVM shows that the app is not using more than half of the heap ever and PermSpace looks fine.
    >
    > Clearly, something is trying to allocate an array of an incorrect huge size but I can't determine what. My code creates objects and sends across JMX but the objects are already allocated in memory and handed to JMX at Exception time. I'm wondering if this is a JMX bug but I cannot find anything online. Any thoughts/ideas are appreciated! Stack trace below:
    >
    >
    > Job_Executor6:class com.sun.jmx.remote.opt.util.JobExecutor java.lan.OutOfMemoryError: Requested array size exceeds VM limit
    > at java.util.Arrays.copyOf(Unknown Source)
    > at java.io.ByteArrayOutputStream.grow(Unknown Source)
    > at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
    > at java.io.ByteArrayOutputStream.write(Unknown Source)
    > at java.io_ObjectOutputStream$BlockDataOutputStream.drain(Unknown Source)
    > at java.io_ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(Unknown Source)
    > at java.io_ObjectOutputStream.writeObject0(Unknown Source)
    > at java.io_ObjectOutputStream.writeObject(Unknown Source)
    > at com.sun.jmx.remote.generic.ObjectWrappingImpl.wrap(ObjectWrappingImpl.java:30)
    > at java.management.remote.generic.ServerIntermediary$RequestHandler.handleNotifyReqMessage(ServerIntermediary.java:683)
    > at java.management.remote.generic.ServerIntermediary$RequestHandler.execute(ServerIntermediary.java:626)
    > at java.management.remote.generic.ServerSynchroMessageConnectionImpl$RemoteJob.run(ServerSynchroMessageConnectionImpl.java:266)
    > at com.sun.jmx.remote.opt.util.ThreadService$ThreadServiceJob.run(ThreadService.java:208)
    > at com.sun.jmx.remote.opt.util.JobExecutor.run(JobExecutor.java:59)


    I'm not familiar with JMX, but from the stack trace it's
    clear that somebody is trying to serialize a big object (or a
    big object graph) to a ByteArrayOutputStream. There's a byte[]
    underlying that stream, and (like any other Java array) it can
    have no more than 0x7fffffff elements -- just shy of 2GB. So:

    1) Why are you trying to serialize a 2GB object (or graph)?
    Does one of the objects being serialized have a reference
    to something enormous, like all of Snowden's files?

    2) If you really must serialize something that large, do you
    need to store it all in memory before sending it wherever?
    That is, could you write directly to a socket stream or a
    file instead of storing it up in a ByteArrayOutputStream
    first?

    3) A 2GB network transmission or disk write (however it's
    marshalled) will take macroscopic time: >3.5 minutes at
    10MB/sec. Just sayin' ...

    --
    Eric Sosman
    d
     
    Eric Sosman, Dec 17, 2013
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    5
    Views:
    3,109
  2. Replies:
    14
    Views:
    924
    Mark McIntyre
    Jan 28, 2006
  3. Replies:
    1
    Views:
    1,097
    Victor Bazarov
    Jun 28, 2005
  4. mike
    Replies:
    0
    Views:
    368
  5. Ben
    Replies:
    7
    Views:
    435
    Dave Anderson
    May 17, 2006
Loading...

Share This Page