Memory Profiler for Heap Analysis

Discussion in 'Java' started by Moritz Wissenbach, Jun 13, 2007.

  1. Hi,

    I have an application (xml-editor) that uses up an unreasonable amount of
    java heap space.
    I have tried profiling with tptp (eclipse) and jmap/jhat, and they do give
    me memory histograms and instance counts, but what I REALLY want is to see
    the total size of a classes instances INCLUDING the referenced objects.

    Is there a profiler that will traverse the heap and find the "recursive"
    size of objects?
    Because knowing that char-Arrays use up most memory is not exactly helpful.

    Plese help, I've searched the web and found nothing; but have been assured
    that something like this exists.

    TIA,

    Moritz
     
    Moritz Wissenbach, Jun 13, 2007
    #1
    1. Advertising

  2. On 13.06.2007 11:51, Moritz Wissenbach wrote:
    > I have an application (xml-editor) that uses up an unreasonable amount
    > of java heap space.
    > I have tried profiling with tptp (eclipse) and jmap/jhat, and they do
    > give me memory histograms and instance counts, but what I REALLY want is
    > to see the total size of a classes instances INCLUDING the referenced
    > objects.
    >
    > Is there a profiler that will traverse the heap and find the "recursive"
    > size of objects?
    > Because knowing that char-Arrays use up most memory is not exactly helpful.
    >
    > Plese help, I've searched the web and found nothing; but have been
    > assured that something like this exists.


    I don't think something like that exists. Assuming something like this
    exists, how does it properly calculate sizes? How many levels does it
    follow when counting? How do you count objects that are referenced by
    multiple other instances etc.

    Having said that I was pretty satisfied with OptimizeIT which also does
    not do recursive analysis but lets you easily find places in code where
    all those objects were allocated and you could walk the object graph. I
    haven't used it in a while so maybe today there are other (free?) tools
    around that server similar purposes.

    Kind regards

    robert
     
    Robert Klemme, Jun 13, 2007
    #2
    1. Advertising

  3. Hi Robert,

    > I don't think something like that exists. Assuming something like this
    > exists, how does it properly calculate sizes? How many levels does it
    > follow when counting?


    Until it hits a cycle?

    > How do you count objects that are referenced by multiple other
    > instances etc.


    Add the objects size to both instaces...?

    Well, I was sceptical too as to wheather this was even possible, but when
    I asked for this feature in the TPTP (eclipse profiler) group, they replied

    > This is a valid and useful use-case for memory profiling. While
    > technically possible (both in theory and in practice), it is not
    > currently
    > in the plan for TPTP.


    So I thought this might be implemented, if not in a free, then in a
    commercial tool.

    Of course it's not a trivial (or probably fast for that matter) task, but
    I guess you could somehow work around the problems.

    I don't want a 100% accurate size count, just an overview over which
    classes use up most memory. I find it kind of hard to see with the
    standard "heap historgram" but maybe I'm missing something.

    Does anyone have any other suggestions? As stated I have an application
    (edior) that uses too much memory (about 21 MB for each MB file size
    loaded) and want to see where all of this goes (DOM, Graphical
    representation, etc..)

    Moritz
     
    Moritz Wissenbach, Jun 13, 2007
    #3
  4. On 13.06.2007 12:54, Moritz Wissenbach wrote:
    > Hi Robert,
    >
    >> I don't think something like that exists. Assuming something like
    >> this exists, how does it properly calculate sizes? How many levels
    >> does it follow when counting?

    >
    > Until it hits a cycle?


    If it does not?

    >> How do you count objects that are referenced by multiple other
    >> instances etc.

    >
    > Add the objects size to both instaces...?


    Remember that n is not restricted to 2, could be much higher.

    > Well, I was sceptical too as to wheather this was even possible, but
    > when I asked for this feature in the TPTP (eclipse profiler) group, they
    > replied
    >
    >> This is a valid and useful use-case for memory profiling. While
    >> technically possible (both in theory and in practice), it is not
    >> currently
    >> in the plan for TPTP.

    >
    > So I thought this might be implemented, if not in a free, then in a
    > commercial tool.
    >
    > Of course it's not a trivial (or probably fast for that matter) task,
    > but I guess you could somehow work around the problems.


    I assume the problems are solvable more on a theoretical level. I don't
    see this problem practically solvable. But maybe I'm missing something
    - the TPTP folks surely have a better understanding of the matter.

    > I don't want a 100% accurate size count, just an overview over which
    > classes use up most memory. I find it kind of hard to see with the
    > standard "heap historgram" but maybe I'm missing something.
    >
    > Does anyone have any other suggestions? As stated I have an application
    > (edior) that uses too much memory (about 21 MB for each MB file size
    > loaded) and want to see where all of this goes (DOM, Graphical
    > representation, etc..)


    DOM is a good candidate. If I were you, I'd create an application
    object model, throw away the DOM parsing and create a SAX parser which
    directly creates the model. Just my 0.02 EUR...

    Kind regards

    robert
     
    Robert Klemme, Jun 13, 2007
    #4
  5. On Wed, 13 Jun 2007 14:09:31 +0200, Robert Klemme
    <> wrote:

    >>> this exists, how does it properly calculate sizes? How many levels
    >>> does it follow when counting?

    >> Until it hits a cycle?

    >
    > If it does not?


    Well, then all the better: it's a proper tree and at some point you'll
    have traversed it completely.

    >>> How do you count objects that are referenced by multiple other
    >>> instances etc.

    >> Add the objects size to both instaces...?

    >
    > Remember that n is not restricted to 2, could be much higher.


    The point you're interested in is the size of a certain object, say A. You
    go ahead and look at what other objects it points. Say it points to object
    B. Since A points to B, B's size needs to be added to A's size. It doesn't
    matter what other objects point to B, for it will always be referenced
    from A, thus, is part of A no matter what.
    On the other hand: I'm looking for things to REMOVE from the program. So,
    if you would remove A in the above scenario, B would still remain in
    memory, so it would be misleading.

    Oh well. If we continue discussing, we could as well implement it. Are you
    in ? ;)


    > DOM is a good candidate. If I were you, I'd create an application
    > object model, throw away the DOM parsing and create a SAX parser which
    > directly creates the model. Just my 0.02 EUR...


    I think the things we want to achieve (online validation for example) do
    require a DOM...

    It's just frustrating. I have the heap dump in front of my eyes, I can see
    all the classes of the DOM, still I can't figure out just how much of the
    memory they cost in total.

    Moritz
     
    Moritz Wissenbach, Jun 13, 2007
    #5
  6. Moritz Wissenbach

    Lew Guest

    Moritz Wissenbach wrote:
    > On Wed, 13 Jun 2007 14:09:31 +0200, Robert Klemme
    > <> wrote:
    >
    >>>> this exists, how does it properly calculate sizes? How many levels
    >>>> does it follow when counting?
    >>> Until it hits a cycle?

    >>
    >> If it does not?

    >
    > Well, then all the better: it's a proper tree and at some point you'll
    > have traversed it completely.
    >
    >>>> How do you count objects that are referenced by multiple other
    >>>> instances etc.
    >>> Add the objects size to both instaces...?

    >>
    >> Remember that n is not restricted to 2, could be much higher.

    >
    > The point you're interested in is the size of a certain object, say A.
    > You go ahead and look at what other objects it points. Say it points to
    > object B. Since A points to B, B's size needs to be added to A's size.
    > It doesn't matter what other objects point to B, for it will always be
    > referenced from A, thus, is part of A no matter what.
    > On the other hand: I'm looking for things to REMOVE from the program.
    > So, if you would remove A in the above scenario, B would still remain in
    > memory, so it would be misleading.


    Strictly speaking, B-the-object's size is not part of A; A only holds a
    reference to B, not the object itself.

    --
    Lew
     
    Lew, Jun 13, 2007
    #6
  7. On 13.06.2007 15:09, Moritz Wissenbach wrote:
    > On Wed, 13 Jun 2007 14:09:31 +0200, Robert Klemme
    > <> wrote:
    >
    >>>> this exists, how does it properly calculate sizes? How many levels
    >>>> does it follow when counting?
    >>> Until it hits a cycle?

    >>
    >> If it does not?

    >
    > Well, then all the better: it's a proper tree and at some point you'll
    > have traversed it completely.


    This still does not address the issue of aliasing properly.

    >>>> How do you count objects that are referenced by multiple other
    >>>> instances etc.
    >>> Add the objects size to both instaces...?

    >>
    >> Remember that n is not restricted to 2, could be much higher.

    >
    > The point you're interested in is the size of a certain object, say A.


    The point I am trying to make is that this concept does not exist with
    regard to /referenced/ objects. The only size you can reasonably well
    detect is the size of the object itself.

    > You go ahead and look at what other objects it points. Say it points to
    > object B. Since A points to B, B's size needs to be added to A's size.
    > It doesn't matter what other objects point to B, for it will always be
    > referenced from A, thus, is part of A no matter what.
    > On the other hand: I'm looking for things to REMOVE from the program.
    > So, if you would remove A in the above scenario, B would still remain in
    > memory, so it would be misleading.
    >
    > Oh well. If we continue discussing, we could as well implement it. Are
    > you in ? ;)


    LOL - no.

    >> DOM is a good candidate. If I were you, I'd create an application
    >> object model, throw away the DOM parsing and create a SAX parser which
    >> directly creates the model. Just my 0.02 EUR...

    >
    > I think the things we want to achieve (online validation for example) do
    > require a DOM...


    If it is a general XML editor then yes, maybe. If you are editing a
    specific XML format, then no.

    > It's just frustrating. I have the heap dump in front of my eyes, I can
    > see all the classes of the DOM, still I can't figure out just how much
    > of the memory they cost in total.


    Did you try the memory profiler from Eclipse's TPTP? How did it go?

    Kind regards

    robert
     
    Robert Klemme, Jun 13, 2007
    #7
  8. On Wed, 13 Jun 2007 15:33:43 +0200, Lew <> wrote:

    >>> Remember that n is not restricted to 2, could be much higher.

    >> The point you're interested in is the size of a certain object, say A.
    >> You go ahead and look at what other objects it points. Say it points to
    >> object B. Since A points to B, B's size needs to be added to A's size.
    >> It doesn't matter what other objects point to B, for it will always be
    >> referenced from A, thus, is part of A no matter what.
    >> On the other hand: I'm looking for things to REMOVE from the program.
    >> So, if you would remove A in the above scenario, B would still remain
    >> in memory, so it would be misleading.

    >
    > Strictly speaking, B-the-object's size is not part of A; A only holds a
    > reference to B, not the object itself.
    >

    Of course. But then the size of A is not representative of memory usage.
     
    Moritz Wissenbach, Jun 13, 2007
    #8
  9. On Wed, 13 Jun 2007 15:45:27 +0200, Robert Klemme
    <> wrote:

    > The point I am trying to make is that this concept does not exist with
    > regard to /referenced/ objects. The only size you can reasonably well
    > detect is the size of the object itself.


    Point taken. That was never the question, but more:
    -given you know what you are doing
    -are able to tune some parameters
    -is there a reasonable approximation

    I thought (and still think) there must be a way

    >>> DOM is a good candidate. If I were you, I'd create an application
    >>> object model, throw away the DOM parsing and create a SAX parser which
    >>> directly creates the model. Just my 0.02 EUR...

    >> I think the things we want to achieve (online validation for example)
    >> do require a DOM...

    >
    > If it is a general XML editor then yes, maybe. If you are editing a
    > specific XML format, then no.


    General XML editor. I have to look into it some more, but 20/1 Memory
    Usage/Document Size seems a little much. Maybe that's what they mean by
    "Java is memory intensive"

    >> It's just frustrating. I have the heap dump in front of my eyes, I can
    >> see all the classes of the DOM, still I can't figure out just how much
    >> of the memory they cost in total.

    >
    > Did you try the memory profiler from Eclipse's TPTP? How did it go?
    >

    Well, maybe I didn't see something, but it basically leaves me with saying
    "Most memory is used by class char[]". Well thanks!

    I can then manually check what references to those char[]s but I don't
    think I'm going to do that a 150 000 times.

    Moritz
     
    Moritz Wissenbach, Jun 13, 2007
    #9
  10. Moritz Wissenbach

    Lew Guest

    Moritz Wissenbach wrote:
    > On Wed, 13 Jun 2007 15:45:27 +0200, Robert Klemme
    > <> wrote:
    >
    >> The point I am trying to make is that this concept does not exist with
    >> regard to /referenced/ objects. The only size you can reasonably well
    >> detect is the size of the object itself.


    Hm, people keep making this point.

    > Point taken. That was never the question, but more:


    > General XML editor. I have to look into it some more, but 20/1 Memory
    > Usage/Document Size seems a little much. Maybe that's what they mean by
    > "Java is memory intensive"


    No, it isn't. "They" are referring to the JVM footprint.

    The overhead you're talking about is DOM overhead, not Java.

    --
    Lew
     
    Lew, Jun 13, 2007
    #10
  11. On Wed, 13 Jun 2007 16:36:53 +0200, Lew <> wrote:

    >> General XML editor. I have to look into it some more, but 20/1 Memory
    >> Usage/Document Size seems a little much. Maybe that's what they mean by
    >> "Java is memory intensive"

    >
    > No, it isn't. "They" are referring to the JVM footprint.
    >
    > The overhead you're talking about is DOM overhead, not Java.


    Well, I meant the general style that comes with Java, for example using
    the collection classes (HashMaps) which will add their share to memory
    usage.
    I guess I'm still surprised that everything in the implementation seems
    ok/necessary and still 1 MB blows up to 20 when loading a file.

    Moritz
     
    Moritz Wissenbach, Jun 13, 2007
    #11
  12. Moritz Wissenbach

    Lew Guest

    Moritz Wissenbach wrote:
    >>> General XML editor. I have to look into it some more, but 20/1
    >>> Memory Usage/Document Size seems a little much. Maybe that's what
    >>> they mean by "Java is memory intensive"


    Lew wrote:
    >> No, it isn't. "They" are referring to the JVM footprint.
    >> The overhead you're talking about is DOM overhead, not Java.


    Moritz Wissenbach wrote:
    > Well, I meant the general style that comes with Java, for example using
    > the collection classes (HashMaps) which will add their share to memory
    > usage.


    It would be different in what other language?

    Moritz Wissenbach wrote:
    > I guess I'm still surprised that everything in the implementation seems
    > ok/necessary and still 1 MB blows up to 20 when loading a file.


    That's the price of all those links between nodes, and all that metadata, and
    whatnot. It's not the language, it's the algorithm. This is still not what
    "they" mean by "Java needs a lot of memory." This is what "they" mean by "DOM
    needs a lot of memory."

    There's no question that the memory cost is high. If that's a problem,
    consider SAX or StAX parsing instead - it's much more PARSimonious. Cover
    what you need in one pass and it's also extremely fast.

    --
    Lew
     
    Lew, Jun 13, 2007
    #12
  13. Moritz Wissenbach

    Juha Laiho Guest

    "Moritz Wissenbach" <> said:
    >I have an application (xml-editor) that uses up an unreasonable amount of
    >java heap space.
    >I have tried profiling with tptp (eclipse) and jmap/jhat, and they do give
    >me memory histograms and instance counts, but what I REALLY want is to see
    >the total size of a classes instances INCLUDING the referenced objects.


    That looks pretty much what Yourkit does (among other things).

    http://www.yourkit.com/

    (disclaimer: I have no affiliation with the company)
    --
    Wolf a.k.a. Juha Laiho Espoo, Finland
    (GC 3.0) GIT d- s+: a C++ ULSH++++$ P++@ L+++ E- W+$@ N++ !K w !O !M V
    PS(+) PE Y+ PGP(+) t- 5 !X R !tv b+ !DI D G e+ h---- r+++ y++++
    "...cancel my subscription to the resurrection!" (Jim Morrison)
     
    Juha Laiho, Jun 18, 2007
    #13
  14. Moritz Wissenbach

    vlad Guest

    > Is there a profiler that will traverse the heap and find the "recursive"
    > size of objects?


    YourKit Java Profiler http://www.yourkit.com/download/ has exactly
    this feature

    BR, Vladimir
     
    vlad, Jun 19, 2007
    #14
  15. On Mon, 18 Jun 2007 20:52:04 +0200, Juha Laiho <> wrote:
    > That looks pretty much what Yourkit does (among other things).
    >
    > http://www.yourkit.com/


    On Tue, 19 Jun 2007 15:00:44 +0200, vlad <>
    wrote:
    > YourKit Java Profiler http://www.yourkit.com/download/ has exactly
    > this feature



    Hey, thanks a lot guys, I'll check that out!
     
    Moritz Wissenbach, Jun 20, 2007
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Michal Slocinski

    Heap dump file size vs heap size

    Michal Slocinski, Mar 25, 2008, in forum: Java
    Replies:
    1
    Views:
    766
    GArlington
    Mar 25, 2008
  2. viki
    Replies:
    6
    Views:
    611
    Erik Wikström
    Jun 28, 2008
  3. ssubbarayan
    Replies:
    5
    Views:
    2,397
    Dave Hansen
    Nov 3, 2009
  4. potatosoftware
    Replies:
    0
    Views:
    671
    potatosoftware
    Nov 4, 2009
  5. Raymond Schanks
    Replies:
    0
    Views:
    574
    Raymond Schanks
    Apr 11, 2010
Loading...

Share This Page