HELP! OutOfMememyError with arrays

[Digital Puer]

Hi, I am running into an OutOfMemoryError and need some help.
My code looks essentially like this:


ByteArrayData ref; // has a field: byte[] _data

while(i = 0 to 100000 files)
{

// reads bytes from file (with FileInputStream) into ref._data
ref = new ByteArrayData(filename);

send_data_over_network(ref);....

// in lieu of free() :
ref._data = null;
ref = null;

// System.gc();

}

I am basically creating new objects inside a tight loop. I am evidently
running out of memory because the memory referenced by 'data' is not
being free'd (to use the C term). Can someone help?

 

[Thomas Hawtin]

// in lieu of free() :
ref._data = null;
ref = null;

// System.gc();

That lot isn't going to help...

I am basically creating new objects inside a tight loop. I am evidently
running out of memory because the memory referenced by 'data' is not
being free'd (to use the C term). Can someone help?

My best guess is that you are not closing resources, or have a
finalisble object that holds onto a large amount of memory. That or one
of the files is too large.

Tom Hawtin

 

[Thomas Hawtin]

That lot isn't going to help...

Actually, I say that but: Suppose the constructor assigning to ref
allocates a lot of memory. The old ref value is unlikely to be
overwritten until after the construction has succeeded. So you could
have two large objects allocated at once.

Tom Hawtin

 

[Digital Puer]

That lot isn't going to help...


My best guess is that you are not closing resources, or have a
finalisble object that holds onto a large amount of memory. That or one
of the files is too large.

Found the problem. I am using ObjectOutputStream to send objects
over sockets. There is evidently a bug that keeps references to
sent objects, so ObjectOutputStream's reset must be called.
See also:

http://java.sun.com/products/jdk/serialization/faq/#OutOfMemoryError

 

[Chris Uppal]

Found the problem. I am using ObjectOutputStream to send objects
over sockets. There is evidently a bug that keeps references to
sent objects, so ObjectOutputStream's reset must be called.

Not a bug, but a necessary part of how serialisation works. The output stream
has to retain a reference to each object that has been written to it, or else
it won't know when/if a second reference to the same object has to be written.
If it didn't do that then object networks would not de-serialise correctly.

-- chris

 

[Thomas Hawtin]

Not a bug, but a necessary part of how serialisation works. The output stream
has to retain a reference to each object that has been written to it, or else
it won't know when/if a second reference to the same object has to be written.
If it didn't do that then object networks would not de-serialise correctly.

Presumably ObjectOutputStream could keep a weak reference.

Tom Hawtin

 

[Timo Stamm]

Found the problem. I am using ObjectOutputStream to send objects
over sockets. There is evidently a bug that keeps references to
sent objects, so ObjectOutputStream's reset must be called.

It's not a bug. This behaviour is intentional:

| The ObjectOutputStream class keeps track of each object it serializes
| and sends only the handle if the object is written into the stream a |
| subsequent time.

 

[Chris Uppal]

Thomas Hawtin wrote:

[me:]

Not a bug, but a necessary part of how serialisation works. The output
stream has to retain a reference to each object that has been written
to it, [...]

Presumably ObjectOutputStream could keep a weak reference.

Hmmm, that's an idea.

I think that it could introduce subtle inconsistancies when writing object that
were themselves only reachable via weak references. E.g.

ObjectOutputStream out = //...
String oops = "oo" + "ps";
out.writeObject(oops.intern());
// ... time passes ...
out.writeObject(oops.intern());

might write two separate objects to the stream, instead of the interned
representation of "oops" twice.

But it's hard to see that as a serious issue, so I suppose it's more likely
that OUS doesn't use a weak collection because Sun either didn't think of it or
didn't think it worthwhile.

-- chris

 

[Thomas Hawtin]

ObjectOutputStream out = //...
String oops = "oo" + "ps";
out.writeObject(oops.intern());

Point of pedantry: oops.intern() there will return the same object as oops.

But it's hard to see that as a serious issue, so I suppose it's more likely
that OUS doesn't use a weak collection because Sun either didn't think of it or
didn't think it worthwhile.

Here's the bug:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4363937

It would slow performance (but I suspect by very little). The main issue
is that apparently the serialisation format has no way to reset for a
single object. An ObjectInputStream would have to keep the deserialised
object strongly referenced, and hence end up with the same memory
problems, only not near the source of the problem.

As with most cases where WeakHashMap or IdentityHashMap is used, it
really wants WeakIdentityHashMap which for some unfathomable reason
still hasn't been provided.

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4500542

Tom Hawtin

 

[Chris Uppal]

Thomas Hawtin wrote:

[me:]

Point of pedantry: oops.intern() there will return the same object as
oops.
Oops!

;-)


http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4363937

Thanks.

-- chris