read, write Objects via Sockets

[tricky]

Hi

I have big problem with memory leak (heap size usage). I send object via
Sockets using object input, output stream.
The problem occurs on the client side when i receive an object

public class My implements Serializable{

/** Creates a new instance of My */
public My() {
}

private String newField =
"ewqeqwewqewdsadsadasdsadsadsadsadadsadsadasdsadsadsadasdasdasdasdasdadsadsasddsadasdsadsa";

public String s(){
return this.newField;
}

}

Server side !!!

ServerSocket s = new ServerSocket(1111);
Socket sv = s.accept();

ObjectOutputStream oos = new ObjectOutputStream(sv.getOutputStream());

while(true){

My m = new My();

oos.writeObject(m);
oos.flush();

m=null;

}
clietn side !!

while(true){


My x = (My)oos.readObject();

System.out.println("pdu size:"+x.s());
x = null;
}


The problem is that memory goes up each time i receive My object. Heap
memory goes up and up and gc() cannot clean memory.

But what is the most funny part . When i'm using insted of My class just
simple byte [] array and send via Socket also using object, input output
stream the problem doesn't occure !. But byte array is also serialized by
default .
Profiler in netbeans shows that it gowas up and down - so the way i would
like it to go
What may be the cause of this problem ? implementing Serializable or write,
read object methods have some problems ?

 

[Gordon Beaton]

I have big problem with memory leak (heap size usage). I send object
via Sockets using object input, output stream. [...]
The problem is that memory goes up each time i receive My object. Heap
memory goes up and up and gc() cannot clean memory.

But what is the most funny part . When i'm using insted of My class
just simple byte [] array and send via Socket also using object,
input output stream the problem doesn't occure !. But byte array is
also serialized by default .

The object stream caches every object you send through it. If you sent
many different objects, the cache grows. That's what you see when you
send your "My" objects.

When you send the same object a second or subsequent times, only a
handle is sent so the receiver can recreate the "same" object. I'll
guess you sent the *same* byte[] each time when you tested that, or
you would have seen the same growth.

You can clear the cache by resetting the stream, but one object sent
both before and after reset will appear at the receiver as two
distinct objects.

/gordon

--

 

[tricky]

I think there is a bug in writeObject or readObject!

So i decided to use ByteArrayOutputStream to change object into byte array
and send via object outputstream using write(byte,offset,len) and it works
fine !
So probably this caching stuff or something else is buggy !

 

[=?ISO-8859-1?Q?Arne_Vajh=F8j?=]

I have big problem with memory leak (heap size usage). I send object via
Sockets using object input, output stream.
The problem occurs on the client side when i receive an object

My x = (My)oos.readObject();

The problem is that memory goes up each time i receive My object. Heap
memory goes up and up and gc() cannot clean memory.

That is a known "feature" of object streams.

You can code like this:

some loop:

oos.writeObject(x);
n++;
if((n % 100) == 0) {
oos.reset();
}

Arne

 

[Lew]

I think there is a bug in writeObject or readObject!

Strong words, considering that you've presented no evidence that you even have
a memory leak, and that the observed behavior is consistent with the design
parameters for the classes in question.

So i [sic] decided to use ByteArrayOutputStream to change object into byte array
and send via object outputstream using write(byte,offset,len) and it works
fine !
So probably this caching stuff or something else is buggy !

Almost certainly not.

A competent programmer blames the following, in order:

1. Their own code.
2. Their own code.
3. Their own design.
4. Their own code.
5. Interactions with third-party libraries due to their own misunderstanding.
6. Their own code.
7. The platform, but only after very, very, very extensive testing and
incontrovertible evidence.

Not only have you not presented incontrovertible evidence of a bug, you have
presented no evidence at all of a bug, nor even of a problem.

 

[Lothar Kimmeringer]

So probably this caching stuff or something else is buggy !

So you are saying that a cache doing its job is regarded to be
buggy? I think I don't want to see your cache-implementations ;-)


Regards, Lothar
--
Lothar Kimmeringer E-Mail: (e-mail address removed)
PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
questions!

 

[Roedy Green]

The problem is that memory goes up each time i receive My object. Heap
memory goes up and up and gc() cannot clean memory.

See http://mindprod.com/jgloss/serialization.html
search for the phase "out of RAM".

The reason is the receiver has to maintain a catalog to all the
objects that have been received so far in the stream in case you send
a object later with a reference to one of them.

You can tell it to have an attack of amnesia with reset.

 

[Guest]

Strong words, considering that you've presented no evidence that you
even have a memory leak, and that the observed behavior is consistent
with the design parameters for the classes in question.

Not only have you not presented incontrovertible evidence of a bug, you
have presented no evidence at all of a bug, nor even of a problem.

It is well known that the object stream classes has a feature where
caching causes out of memory.

It is a feature. But it is not very well documented. And probably
unwanted by most users.

Arne

 

[Knute Johnson]

See http://mindprod.com/jgloss/serialization.html
search for the phase "out of RAM".

The reason is the receiver has to maintain a catalog to all the
objects that have been received so far in the stream in case you send
a object later with a reference to one of them.

You can tell it to have an attack of amnesia with reset.

Roedy:

I looked at your site and now I'm confused. If you send a series of
Objects down the stream and call reset after sending each one, what
exactly happens on the receiving end? Is there some reference kept by
the ObjectInputStream that will prevent the received objects from being
GCed?

Thanks,

 

[Roedy Green]

It is a feature. But it is not very well documented. And probably
unwanted by most users.

The alternative would be to resend every object when there is yet
another reference to it, including string constants, which would
drastically bulk up the stream.

It might have been implemented with a max cache size on the sender
side. It would then delete infrequently used items from its cache if
it were getting near the limit.

 

[Roedy Green]

I looked at your site and now I'm confused. If you send a series of
Objects down the stream and call reset after sending each one, what
exactly happens on the receiving end? Is there some reference kept by
the ObjectInputStream that will prevent the received objects from being
GCed?

Hitting reset effectively closes and reopens the stream. Both sides
start from scratch. Previously received objects would usually be
pointed to by the receiver, e.g. added to a Collection. Any it does
not point to will be GCed. The sender won't expect the receiver to
have kept any objects from the previous batch. So it will resend any
that are referenced again.

 

[=?ISO-8859-1?Q?Arne_Vajh=F8j?=]

The alternative would be to resend every object when there is yet
another reference to it, including string constants, which would
drastically bulk up the stream.

I have never seen an example in real life where this cache feature
was considered useful.

It is not even just the memory issue. It can also give some funny
effects with mutable objects.

Arne

 

[Lew]

It is well known that the object stream classes has a feature where
caching causes out of memory.

It is a feature. But it is not very well documented. And probably
unwanted by most users.

It is common to see memory rise in a Java program without there being a memory
leak. All the OP told us was that they were receiving an increase in memory
usage. I did not see evidence that it was actually a leak. Had they reported
an OOME that'd have been different.

I certainly learned something new about the object streams from this discussion.

 

[Knute Johnson]

Hitting reset effectively closes and reopens the stream. Both sides
start from scratch. Previously received objects would usually be
pointed to by the receiver, e.g. added to a Collection. Any it does
not point to will be GCed. The sender won't expect the receiver to
have kept any objects from the previous batch. So it will resend any
that are referenced again.

So if you send only new objects and reset between them, while slow,
there should be no risk of OOMException right?

 

[Roedy Green]

It is not even just the memory issue. It can also give some funny
effects with mutable objects.

The way it works now, if you change an object, and resend it, it does
not get sent. A reference to the old copy gets sent.

If you used a some-times forgetting cache, that behaviour would be
indeterminate. Sometimes you would get the old object, sometimes the
new. The rule would be, don't modify objects you are in the process of
serialising unless the changes don't matter.

 

[Roedy Green]

So if you send only new objects and reset between them, while slow,
there should be no risk of OOMException right?

Yes. I don't think it would be much slower than not using reset. I
think all reset does in tell both ends to flush their invisible cache
of previously transmitted objects.

 

[Gordon Beaton]

The way it works now, if you change an object, and resend it, it
does not get sent. A reference to the old copy gets sent.

But that isn't an argument against caching per se, which could have
been implemented to detect when a sent object doesn't agree with the
cached one.

The caching itself provides a useful feature, making it possible to
preserve the topology of object references across the serialization.
If you send two distinct objects each with a reference to a common
third object, the recipient shouldn't receive four distinct objects.
Similarly, it makes it possible to correctly transfer structures
containing cyclic references.

/gordon

--

 

[Lothar Kimmeringer]

I have never seen an example in real life where this cache feature
was considered useful.

------ snip
public class MyList implements Serializable
public MyList next;
}

MyList a = new MyList();
MyList b = new MyList();

a.next = b;
b.next = a;
------ snap

Try to serialize that without a cache and you will run into problem.


Regards, Lothar
--
Lothar Kimmeringer E-Mail: (e-mail address removed)
PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
questions!

 

[=?ISO-8859-1?Q?Arne_Vajh=F8j?=]

------ snip
public class MyList implements Serializable
public MyList next;
}

MyList a = new MyList();
MyList b = new MyList();

a.next = b;
b.next = a;
------ snap

Try to serialize that without a cache and you will run into problem.

They keyword was "real life".

In real life I would expect a java.util.LinkedList to be send in
a single writeObject and I will assume that it would work fine.

Arne

 

[Roedy Green]

Yes. I don't think it would be much slower than not using reset. I
think all reset does in tell both ends to flush their invisible cache
of previously transmitted objects.

The other concern, is every time you call reset, all the CLASS
definitions are forgotten too and have to be resent.

You might get a feel for how this works by examining a
ObjectOutputStream with a hex viewer, salted with some unique strings.

Happily GZIP does a great job of compressing the stream.