read, write Objects via Sockets

Discussion in 'Java' started by tricky, Nov 3, 2007.

  1. tricky

    tricky Guest

    Hi

    I have big problem with memory leak (heap size usage). I send object via
    Sockets using object input, output stream.
    The problem occurs on the client side when i receive an object

    public class My implements Serializable{

    /** Creates a new instance of My */
    public My() {
    }

    private String newField =
    "ewqeqwewqewdsadsadasdsadsadsadsadadsadsadasdsadsadsadasdasdasdasdasdadsadsasddsadasdsadsa";

    public String s(){
    return this.newField;
    }

    }

    Server side !!!

    ServerSocket s = new ServerSocket(1111);
    Socket sv = s.accept();

    ObjectOutputStream oos = new ObjectOutputStream(sv.getOutputStream());

    while(true){

    My m = new My();

    oos.writeObject(m);
    oos.flush();

    m=null;

    }
    clietn side !!

    while(true){


    My x = (My)oos.readObject();

    System.out.println("pdu size:"+x.s());
    x = null;
    }


    The problem is that memory goes up each time i receive My object. Heap
    memory goes up and up and gc() cannot clean memory.

    But what is the most funny part . When i'm using insted of My class just
    simple byte [] array and send via Socket also using object, input output
    stream the problem doesn't occure !. But byte array is also serialized by
    default .
    Profiler in netbeans shows that it gowas up and down - so the way i would
    like it to go :D
    What may be the cause of this problem ? implementing Serializable or write,
    read object methods have some problems ?
    tricky, Nov 3, 2007
    #1
    1. Advertising

  2. On Sat, 3 Nov 2007 11:23:16 +0100, tricky wrote:
    > I have big problem with memory leak (heap size usage). I send object
    > via Sockets using object input, output stream.

    [...]
    > The problem is that memory goes up each time i receive My object. Heap
    > memory goes up and up and gc() cannot clean memory.
    >
    > But what is the most funny part . When i'm using insted of My class
    > just simple byte [] array and send via Socket also using object,
    > input output stream the problem doesn't occure !. But byte array is
    > also serialized by default .


    The object stream caches every object you send through it. If you sent
    many different objects, the cache grows. That's what you see when you
    send your "My" objects.

    When you send the same object a second or subsequent times, only a
    handle is sent so the receiver can recreate the "same" object. I'll
    guess you sent the *same* byte[] each time when you tested that, or
    you would have seen the same growth.

    You can clear the cache by resetting the stream, but one object sent
    both before and after reset will appear at the receiver as two
    distinct objects.

    /gordon

    --
    Gordon Beaton, Nov 3, 2007
    #2
    1. Advertising

  3. tricky

    tricky Guest

    I think there is a bug in writeObject or readObject!

    So i decided to use ByteArrayOutputStream to change object into byte array
    and send via object outputstream using write(byte,offset,len) and it works
    fine !
    So probably this caching stuff or something else is buggy !
    tricky, Nov 3, 2007
    #3
  4. tricky wrote:
    > I have big problem with memory leak (heap size usage). I send object via
    > Sockets using object input, output stream.
    > The problem occurs on the client side when i receive an object


    > oos.writeObject(m);


    > My x = (My)oos.readObject();


    > The problem is that memory goes up each time i receive My object. Heap
    > memory goes up and up and gc() cannot clean memory.


    That is a known "feature" of object streams.

    You can code like this:

    some loop:

    oos.writeObject(x);
    n++;
    if((n % 100) == 0) {
    oos.reset();
    }

    Arne
    =?ISO-8859-1?Q?Arne_Vajh=F8j?=, Nov 3, 2007
    #4
  5. tricky

    Lew Guest

    tricky wrote:
    > I think there is a bug in writeObject or readObject!


    Strong words, considering that you've presented no evidence that you even have
    a memory leak, and that the observed behavior is consistent with the design
    parameters for the classes in question.

    > So i [sic] decided to use ByteArrayOutputStream to change object into byte array
    > and send via object outputstream using write(byte,offset,len) and it works
    > fine !
    > So probably this caching stuff or something else is buggy !


    Almost certainly not.

    A competent programmer blames the following, in order:

    1. Their own code.
    2. Their own code.
    3. Their own design.
    4. Their own code.
    5. Interactions with third-party libraries due to their own misunderstanding.
    6. Their own code.
    7. The platform, but only after very, very, very extensive testing and
    incontrovertible evidence.

    Not only have you not presented incontrovertible evidence of a bug, you have
    presented no evidence at all of a bug, nor even of a problem.

    --
    Lew
    Lew, Nov 3, 2007
    #5
  6. tricky wrote:

    > So probably this caching stuff or something else is buggy !


    So you are saying that a cache doing its job is regarded to be
    buggy? I think I don't want to see your cache-implementations ;-)


    Regards, Lothar
    --
    Lothar Kimmeringer E-Mail:
    PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

    Always remember: The answer is forty-two, there can only be wrong
    questions!
    Lothar Kimmeringer, Nov 3, 2007
    #6
  7. tricky

    Roedy Green Guest

    On Sat, 3 Nov 2007 11:23:16 +0100, "tricky" <>
    wrote, quoted or indirectly quoted someone who said :

    >The problem is that memory goes up each time i receive My object. Heap
    >memory goes up and up and gc() cannot clean memory.


    See http://mindprod.com/jgloss/serialization.html
    search for the phase "out of RAM".

    The reason is the receiver has to maintain a catalog to all the
    objects that have been received so far in the stream in case you send
    a object later with a reference to one of them.

    You can tell it to have an attack of amnesia with reset.

    --
    Roedy Green Canadian Mind Products
    The Java Glossary
    http://mindprod.com
    Roedy Green, Nov 3, 2007
    #7
  8. Lew wrote:
    > tricky wrote:
    >> I think there is a bug in writeObject or readObject!

    >
    > Strong words, considering that you've presented no evidence that you
    > even have a memory leak, and that the observed behavior is consistent
    > with the design parameters for the classes in question.


    > Not only have you not presented incontrovertible evidence of a bug, you
    > have presented no evidence at all of a bug, nor even of a problem.


    It is well known that the object stream classes has a feature where
    caching causes out of memory.

    It is a feature. But it is not very well documented. And probably
    unwanted by most users.

    Arne
    =?UTF-8?B?QXJuZSBWYWpow7hq?=, Nov 3, 2007
    #8
  9. Roedy Green wrote:
    > On Sat, 3 Nov 2007 11:23:16 +0100, "tricky" <>
    > wrote, quoted or indirectly quoted someone who said :
    >
    >> The problem is that memory goes up each time i receive My object. Heap
    >> memory goes up and up and gc() cannot clean memory.

    >
    > See http://mindprod.com/jgloss/serialization.html
    > search for the phase "out of RAM".
    >
    > The reason is the receiver has to maintain a catalog to all the
    > objects that have been received so far in the stream in case you send
    > a object later with a reference to one of them.
    >
    > You can tell it to have an attack of amnesia with reset.
    >


    Roedy:

    I looked at your site and now I'm confused. If you send a series of
    Objects down the stream and call reset after sending each one, what
    exactly happens on the receiving end? Is there some reference kept by
    the ObjectInputStream that will prevent the received objects from being
    GCed?

    Thanks,

    --

    Knute Johnson
    email s/nospam/knute/
    Knute Johnson, Nov 3, 2007
    #9
  10. tricky

    Roedy Green Guest

    On Sat, 03 Nov 2007 13:47:32 -0400, Arne Vajhøj <>
    wrote, quoted or indirectly quoted someone who said :

    >It is a feature. But it is not very well documented. And probably
    >unwanted by most users.


    The alternative would be to resend every object when there is yet
    another reference to it, including string constants, which would
    drastically bulk up the stream.

    It might have been implemented with a max cache size on the sender
    side. It would then delete infrequently used items from its cache if
    it were getting near the limit.

    --
    Roedy Green Canadian Mind Products
    The Java Glossary
    http://mindprod.com
    Roedy Green, Nov 3, 2007
    #10
  11. tricky

    Roedy Green Guest

    On Sat, 03 Nov 2007 10:59:25 -0700, Knute Johnson
    <> wrote, quoted or indirectly quoted
    someone who said :

    >I looked at your site and now I'm confused. If you send a series of
    >Objects down the stream and call reset after sending each one, what
    >exactly happens on the receiving end? Is there some reference kept by
    >the ObjectInputStream that will prevent the received objects from being
    >GCed?


    Hitting reset effectively closes and reopens the stream. Both sides
    start from scratch. Previously received objects would usually be
    pointed to by the receiver, e.g. added to a Collection. Any it does
    not point to will be GCed. The sender won't expect the receiver to
    have kept any objects from the previous batch. So it will resend any
    that are referenced again.
    --
    Roedy Green Canadian Mind Products
    The Java Glossary
    http://mindprod.com
    Roedy Green, Nov 3, 2007
    #11
  12. Roedy Green wrote:
    > On Sat, 03 Nov 2007 13:47:32 -0400, Arne Vajhøj <>
    > wrote, quoted or indirectly quoted someone who said :
    >> It is a feature. But it is not very well documented. And probably
    >> unwanted by most users.

    >
    > The alternative would be to resend every object when there is yet
    > another reference to it, including string constants, which would
    > drastically bulk up the stream.


    I have never seen an example in real life where this cache feature
    was considered useful.

    It is not even just the memory issue. It can also give some funny
    effects with mutable objects.

    Arne
    =?ISO-8859-1?Q?Arne_Vajh=F8j?=, Nov 3, 2007
    #12
  13. tricky

    Lew Guest

    Lew wrote:
    >> Not only have you not presented incontrovertible evidence of a bug,
    >> you have presented no evidence at all of a bug, nor even of a problem.


    Arne Vajhøj wrote:
    > It is well known that the object stream classes has a feature where
    > caching causes out of memory.
    >
    > It is a feature. But it is not very well documented. And probably
    > unwanted by most users.


    It is common to see memory rise in a Java program without there being a memory
    leak. All the OP told us was that they were receiving an increase in memory
    usage. I did not see evidence that it was actually a leak. Had they reported
    an OOME that'd have been different.

    I certainly learned something new about the object streams from this discussion.

    --
    Lew
    Lew, Nov 3, 2007
    #13
  14. Roedy Green wrote:
    > On Sat, 03 Nov 2007 10:59:25 -0700, Knute Johnson
    > <> wrote, quoted or indirectly quoted
    > someone who said :
    >
    >> I looked at your site and now I'm confused. If you send a series of
    >> Objects down the stream and call reset after sending each one, what
    >> exactly happens on the receiving end? Is there some reference kept by
    >> the ObjectInputStream that will prevent the received objects from being
    >> GCed?

    >
    > Hitting reset effectively closes and reopens the stream. Both sides
    > start from scratch. Previously received objects would usually be
    > pointed to by the receiver, e.g. added to a Collection. Any it does
    > not point to will be GCed. The sender won't expect the receiver to
    > have kept any objects from the previous batch. So it will resend any
    > that are referenced again.


    So if you send only new objects and reset between them, while slow,
    there should be no risk of OOMException right?

    --

    Knute Johnson
    email s/nospam/knute/
    Knute Johnson, Nov 3, 2007
    #14
  15. tricky

    Roedy Green Guest

    On Sat, 03 Nov 2007 14:32:54 -0400, Arne Vajhøj <>
    wrote, quoted or indirectly quoted someone who said :

    >
    >It is not even just the memory issue. It can also give some funny
    >effects with mutable objects.


    The way it works now, if you change an object, and resend it, it does
    not get sent. A reference to the old copy gets sent.

    If you used a some-times forgetting cache, that behaviour would be
    indeterminate. Sometimes you would get the old object, sometimes the
    new. The rule would be, don't modify objects you are in the process of
    serialising unless the changes don't matter.
    --
    Roedy Green Canadian Mind Products
    The Java Glossary
    http://mindprod.com
    Roedy Green, Nov 3, 2007
    #15
  16. tricky

    Roedy Green Guest

    On Sat, 03 Nov 2007 12:27:49 -0700, Knute Johnson
    <> wrote, quoted or indirectly quoted
    someone who said :

    >So if you send only new objects and reset between them, while slow,
    >there should be no risk of OOMException right?


    Yes. I don't think it would be much slower than not using reset. I
    think all reset does in tell both ends to flush their invisible cache
    of previously transmitted objects.
    --
    Roedy Green Canadian Mind Products
    The Java Glossary
    http://mindprod.com
    Roedy Green, Nov 3, 2007
    #16
  17. On Sat, 03 Nov 2007 19:37:56 GMT, Roedy Green wrote:
    > The way it works now, if you change an object, and resend it, it
    > does not get sent. A reference to the old copy gets sent.


    But that isn't an argument against caching per se, which could have
    been implemented to detect when a sent object doesn't agree with the
    cached one.

    The caching itself provides a useful feature, making it possible to
    preserve the topology of object references across the serialization.
    If you send two distinct objects each with a reference to a common
    third object, the recipient shouldn't receive four distinct objects.
    Similarly, it makes it possible to correctly transfer structures
    containing cyclic references.

    /gordon

    --
    Gordon Beaton, Nov 3, 2007
    #17
  18. Arne Vajhøj wrote:

    > I have never seen an example in real life where this cache feature
    > was considered useful.


    ------ snip
    public class MyList implements Serializable
    public MyList next;
    }

    MyList a = new MyList();
    MyList b = new MyList();

    a.next = b;
    b.next = a;
    ------ snap

    Try to serialize that without a cache and you will run into problem.


    Regards, Lothar
    --
    Lothar Kimmeringer E-Mail:
    PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

    Always remember: The answer is forty-two, there can only be wrong
    questions!
    Lothar Kimmeringer, Nov 3, 2007
    #18
  19. Lothar Kimmeringer wrote:
    > Arne Vajhøj wrote:
    >> I have never seen an example in real life where this cache feature
    >> was considered useful.

    >
    > ------ snip
    > public class MyList implements Serializable
    > public MyList next;
    > }
    >
    > MyList a = new MyList();
    > MyList b = new MyList();
    >
    > a.next = b;
    > b.next = a;
    > ------ snap
    >
    > Try to serialize that without a cache and you will run into problem.


    They keyword was "real life".

    In real life I would expect a java.util.LinkedList to be send in
    a single writeObject and I will assume that it would work fine.

    Arne
    =?ISO-8859-1?Q?Arne_Vajh=F8j?=, Nov 3, 2007
    #19
  20. tricky

    Roedy Green Guest

    On Sat, 03 Nov 2007 19:40:05 GMT, Roedy Green
    <> wrote, quoted or indirectly quoted
    someone who said :

    >>So if you send only new objects and reset between them, while slow,
    >>there should be no risk of OOMException right?

    >
    >Yes. I don't think it would be much slower than not using reset. I
    >think all reset does in tell both ends to flush their invisible cache
    >of previously transmitted objects.


    The other concern, is every time you call reset, all the CLASS
    definitions are forgotten too and have to be resent.

    You might get a feel for how this works by examining a
    ObjectOutputStream with a hex viewer, salted with some unique strings.

    Happily GZIP does a great job of compressing the stream.
    --
    Roedy Green Canadian Mind Products
    The Java Glossary
    http://mindprod.com
    Roedy Green, Nov 3, 2007
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. !George Sgouros
    Replies:
    8
    Views:
    482
    Tor Iver Wilhelmsen
    Dec 23, 2003
  2. 7stud
    Replies:
    11
    Views:
    689
    Dennis Lee Bieber
    Mar 20, 2007
  3. Michael Schumacher

    Write-protecting objects addressed via indirect pointers

    Michael Schumacher, Oct 20, 2009, in forum: C Programming
    Replies:
    13
    Views:
    571
    Flash Gordon
    Oct 20, 2009
  4. sahm
    Replies:
    4
    Views:
    679
  5. Tim Chase
    Replies:
    0
    Views:
    88
    Tim Chase
    Dec 16, 2013
Loading...

Share This Page