JMX OutOfMemoryError: Requested array size exceeds VM limit

[mikesmith813]

I have a Java server application that consistantly throws an "OutOfMemoryError: Requested array size exceeds VM limit" and crashes about every 70 hours.

I'm running on Java 1.7 release 45 with a beefy 18 CPU Linux Server with lots of memory. My max heap is set to 50GB and using the new G1GC collector.Profiling with JVisualVM shows that the app is not using more than half of the heap ever and PermSpace looks fine.

Clearly, something is trying to allocate an array of an incorrect huge sizebut I can't determine what. My code creates objects and sends across JMX but the objects are already allocated in memory and handed to JMX at Exception time. I'm wondering if this is a JMX bug but I cannot find anything online. Any thoughts/ideas are appreciated! Stack trace below:


Job_Executor6:class com.sun.jmx.remote.opt.util.JobExecutor java.lan.OutOfMemoryError: Requested array size exceeds VM limit
at java.util.Arrays.copyOf(Unknown Source)
at java.io.ByteArrayOutputStream.grow(Unknown Source)
at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
at java.io.ByteArrayOutputStream.write(Unknown Source)
at java.ibjectOutputStream$BlockDataOutputStream.drain(Unknown Source)
at java.ibjectOutputStream$BlockDataOutputStream.setBlockDataMode(Unknown Source)
at java.ibjectOutputStream.writeObject0(Unknown Source)
at java.ibjectOutputStream.writeObject(Unknown Source)
at com.sun.jmx.remote.generic.ObjectWrappingImpl.wrap(ObjectWrappingImpl.java:30)
at java.management.remote.generic.ServerIntermediary$RequestHandler.handleNotifyReqMessage(ServerIntermediary.java:683)
at java.management.remote.generic.ServerIntermediary$RequestHandler.execute(ServerIntermediary.java:626)
at java.management.remote.generic.ServerSynchroMessageConnectionImpl$RemoteJob.run(ServerSynchroMessageConnectionImpl.java:266)
at com.sun.jmx.remote.opt.util.ThreadService$ThreadServiceJob.run(ThreadService.java:208)
at com.sun.jmx.remote.opt.util.JobExecutor.run(JobExecutor.java:59)

 

[Stefan Ram]

Job_Executor6:class com.sun.jmx.remote.opt.util.JobExecutor java.lan.OutOfMemoryError: Requested array size exceeds VM limit

There is a »g« missing in ».lan.« above. Possibly, the error
message was edited by someone? Possibly by a human being?

Hearing that it crashed after about 70 hours, my first idea
was a growing array. The stack partially confirms this, but
unexpectedly, it is not a java.util.ArrayList, but a
java.io.ByteArrayOutputStream.

at java.util.Arrays.copyOf(Unknown Source)
at java.io.ByteArrayOutputStream.grow(Unknown Source)
at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
at java.io.ByteArrayOutputStream.write(Unknown Source)

The next lines look like someone is trying to serialize.

at java.ibjectOutputStream$BlockDataOutputStream.drain(Unknown Source)
at java.ibjectOutputStream$BlockDataOutputStream.setBlockDataMode(Unknown Source)
at java.ibjectOutputStream.writeObject0(Unknown Source)
at java.ibjectOutputStream.writeObject(Unknown Source)

Possible that someone is trying to write a VLO (very large
object) and during serialization, there is not enough memory
for some array buffering the output. The VLO might grow
during about 70 hours until it eventually has grown too
large for serialization. It might contain an array itself
not directly visible in the stack that has grown that much.

 

[Eric Sosman]

I have a Java server application that consistantly throws an "OutOfMemoryError: Requested array size exceeds VM limit" and crashes about every 70 hours.

I'm running on Java 1.7 release 45 with a beefy 18 CPU Linux Server with lots of memory. My max heap is set to 50GB and using the new G1GC collector. Profiling with JVisualVM shows that the app is not using more than half of the heap ever and PermSpace looks fine.

Clearly, something is trying to allocate an array of an incorrect huge size but I can't determine what. My code creates objects and sends across JMX but the objects are already allocated in memory and handed to JMX at Exception time. I'm wondering if this is a JMX bug but I cannot find anything online. Any thoughts/ideas are appreciated! Stack trace below:


Job_Executor6:class com.sun.jmx.remote.opt.util.JobExecutor java.lan.OutOfMemoryError: Requested array size exceeds VM limit
at java.util.Arrays.copyOf(Unknown Source)
at java.io.ByteArrayOutputStream.grow(Unknown Source)
at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
at java.io.ByteArrayOutputStream.write(Unknown Source)
at java.ibjectOutputStream$BlockDataOutputStream.drain(Unknown Source)
at java.ibjectOutputStream$BlockDataOutputStream.setBlockDataMode(Unknown Source)
at java.ibjectOutputStream.writeObject0(Unknown Source)
at java.ibjectOutputStream.writeObject(Unknown Source)
at com.sun.jmx.remote.generic.ObjectWrappingImpl.wrap(ObjectWrappingImpl.java:30)
at java.management.remote.generic.ServerIntermediary$RequestHandler.handleNotifyReqMessage(ServerIntermediary.java:683)
at java.management.remote.generic.ServerIntermediary$RequestHandler.execute(ServerIntermediary.java:626)
at java.management.remote.generic.ServerSynchroMessageConnectionImpl$RemoteJob.run(ServerSynchroMessageConnectionImpl.java:266)
at com.sun.jmx.remote.opt.util.ThreadService$ThreadServiceJob.run(ThreadService.java:208)
at com.sun.jmx.remote.opt.util.JobExecutor.run(JobExecutor.java:59)

I'm not familiar with JMX, but from the stack trace it's
clear that somebody is trying to serialize a big object (or a
big object graph) to a ByteArrayOutputStream. There's a byte[]
underlying that stream, and (like any other Java array) it can
have no more than 0x7fffffff elements -- just shy of 2GB. So:

1) Why are you trying to serialize a 2GB object (or graph)?
Does one of the objects being serialized have a reference
to something enormous, like all of Snowden's files?

2) If you really must serialize something that large, do you
need to store it all in memory before sending it wherever?
That is, could you write directly to a socket stream or a
file instead of storing it up in a ByteArrayOutputStream
first?

3) A 2GB network transmission or disk write (however it's
marshalled) will take macroscopic time: >3.5 minutes at
10MB/sec. Just sayin' ...