Memory utilization

Discussion in 'Java' started by ghostwhoowalks@gmail.com, Jan 30, 2008.

  1. Guest

    I was trying to profile the following piece of code using NetBeans:

    public static void main(String[] args)
    {
    Map<String, String> map = new HashMap<String, String>();
    for( int i = 0; i < Integer.MAX_VALUE; ++i )
    {
    map.put(Integer.valueOf(i).toString(),
    Integer.valueOf(i).toString());
    }
    }


    I set my max and min heap size to be 1GB. I find that the program dies
    after having created 4M entries (at which point the heap used is at
    1GB) although the memory used by the objects as indicated by the
    Netbeans memory profiler is the following:

    String -- 48MB
    char[] -- 48 MB
    HeapCharBuffer -- 45MB
    Integer -- 19MB
    HashMap$Entry -- 19MB


    This totals to less than 200 MB and the rest of the objects are at 0%
    of the total used memory at most another 200MB. How come the heap
    usage is at 1GB at this point of time? I am extremely confused. Could
    someone please explain what might be going on?

    Thanks
    A
    , Jan 30, 2008
    #1
    1. Advertising

  2. Guest

    On Jan 30, 4:54 am, wrote:
    > I was trying to profile the following piece of code using NetBeans:
    >
    > public static void main(String[] args)
    > {
    > Map<String, String> map = new HashMap<String, String>();
    > for( int i = 0; i < Integer.MAX_VALUE; ++i )
    > {
    > map.put(Integer.valueOf(i).toString(),
    > Integer.valueOf(i).toString());
    > }
    >
    > }
    >
    > I set my max and min heap size to be 1GB. I find that the program dies
    > after having created 4M entries (at which point the heap used is at
    > 1GB) although the memory used by the objects as indicated by the
    > Netbeans memory profiler is the following:
    >
    > String -- 48MB
    > char[] -- 48 MB
    > HeapCharBuffer -- 45MB
    > Integer -- 19MB
    > HashMap$Entry -- 19MB
    >
    > This totals to less than 200 MB and the rest of the objects are at 0%
    > of the total used memory at most another 200MB. How come the heap
    > usage is at 1GB at this point of time? I am extremely confused. Could
    > someone please explain what might be going on?
    >
    > Thanks
    > A


    The profiler itself takes up quite a bit, but doesn't show it to you.
    Also, space is needed for classes and other items of overhead.

    T
    , Jan 30, 2008
    #2
    1. Advertising

  3. GArlington Guest

    On Jan 30, 2:54 am, wrote:
    > I was trying to profile the following piece of code using NetBeans:
    >
    > public static void main(String[] args)
    > {
    > Map<String, String> map = new HashMap<String, String>();
    > for( int i = 0; i < Integer.MAX_VALUE; ++i )
    > {
    > map.put(Integer.valueOf(i).toString(),
    > Integer.valueOf(i).toString());
    > }
    >
    > }
    >
    > I set my max and min heap size to be 1GB. I find that the program dies
    > after having created 4M entries (at which point the heap used is at
    > 1GB) although the memory used by the objects as indicated by the
    > Netbeans memory profiler is the following:
    >
    > String -- 48MB
    > char[] -- 48 MB
    > HeapCharBuffer -- 45MB
    > Integer -- 19MB
    > HashMap$Entry -- 19MB
    >
    > This totals to less than 200 MB and the rest of the objects are at 0%
    > of the total used memory at most another 200MB. How come the heap
    > usage is at 1GB at this point of time? I am extremely confused. Could
    > someone please explain what might be going on?
    >
    > Thanks
    > A


    Something is VERY wrong here:
    "4M entries" in Map<String, String> will mean 8M String(s) - each
    String has an OVERHEAD of 40bytes (AFAIK) - this alone should give you
    320MB heap usage even when ALL String(s) are empty.
    GArlington, Jan 30, 2008
    #3
  4. Lew Guest

    GArlington wrote:
    > each
    > String has an OVERHEAD of 40bytes (AFAIK)


    I'm not sure that's correct. Are you sure it isn't 4 bytes per String? 40
    seems ridiculously high. Where'd you get that number?

    --
    Lew
    Lew, Jan 30, 2008
    #4
  5. Guest

    On Jan 30, 3:20 pm, Lew <> wrote:
    > GArlington wrote:
    > > each
    > > String has an OVERHEAD of 40bytes (AFAIK)

    >
    > I'm not sure that's correct. Are you sure it isn't 4 bytes per String? 40
    > seems ridiculously high. Where'd you get that number?
    >
    > --
    > Lew


    On the machine you are on is it possible to increase your max heap
    space higher? If so problem solved!

    Graeme
    , Jan 30, 2008
    #5
  6. wrote:
    > On Jan 30, 3:20 pm, Lew <> wrote:
    >
    >>GArlington wrote:
    >>
    >>>each
    >>>String has an OVERHEAD of 40bytes (AFAIK)

    >>
    >>I'm not sure that's correct. Are you sure it isn't 4 bytes per String? 40
    >>seems ridiculously high. Where'd you get that number?
    >>
    >>--
    >>Lew

    >
    >
    > On the machine you are on is it possible to increase your max heap
    > space higher? If so problem solved!


    That depends on what the problem really is. The program looks more like
    a test case than an actual application. I believe the problem is
    explaining the relationship between the NetBeans memory profile, the
    program, and the out-of-memory condition. If so, increasing the heap
    space could not possibly solve it, and might make it harder to solve.

    Patricia
    Patricia Shanahan, Jan 30, 2008
    #6
  7. Eric Sosman Guest

    wrote:
    > On Jan 30, 3:20 pm, Lew <> wrote:
    >> GArlington wrote:
    >>> each
    >>> String has an OVERHEAD of 40bytes (AFAIK)

    >> I'm not sure that's correct. Are you sure it isn't 4 bytes per String? 40
    >> seems ridiculously high. Where'd you get that number?


    Might be high, but doesn't seem ridiculous. A String
    instance has its object-ness plus three ints and a char[]
    reference. The char[] will have its own object-ness plus
    a length. If object-ness takes eight bytes and a reference
    takes four (32-bit JVM), we're up to at least 36 bytes. If
    there's an eight-byte alignment requirement, 40 bytes is
    right on the money.

    > On the machine you are on is it possible to increase your max heap
    > space higher? If so problem solved!


    He's trying to create 2G pairs of String objects (72-80
    bytes of overhead per pair) plus 2G HashMap.Entry objects
    (~24 bytes each, under the same set of assumptions), hence
    about 192-208GB -- and that's not counting the "payload" of
    the char values, which will add another 80GB or thereabouts.
    272GB is well beyond what a 32-bit JVM can manage, so he'll
    need to go to a 64-bit version -- and this will expand the
    String overhead to at least 40 bytes, and the HashMap.Entry
    size to at least 36-40, raising the total to about 312-320GB.
    And that's just for this one data structure; the rest of the
    Java environment needs some heap, too. I'd guess that a
    machine with half a terabyte of RAM could just about manage
    it, with luck and a following wind.

    ... but in truth he's not really trying to populate this
    enormous and pointlessly wasteful data structure; he's trying
    to understand how the NetBeans profiler reports the memory
    used by the program, and I'm afraid I can't help much with
    that. After 4M pairs he sees

    > String -- 48MB
    > char[] -- 48 MB
    > HeapCharBuffer -- 45MB
    > Integer -- 19MB
    > HashMap$Entry -- 19MB


    .... which works out to 6 bytes per string, 6 bytes per char[],
    and <5 bytes per HashMap.Entry -- clearly, the profiler reports
    some other notion of "memory used" than we've been talking
    about in the last few messages. All I can suggest is RTFM.

    --
    Eric Sosman, Jan 30, 2008
    #7
  8. Guest

    On Jan 30, 8:48 am, Eric Sosman <> wrote:
    > wrote:
    > > On Jan 30, 3:20 pm, Lew <> wrote:
    > >> GArlington wrote:
    > >>> each
    > >>> String has an OVERHEAD of 40bytes (AFAIK)
    > >> I'm not sure that's correct. Are you sure it isn't 4 bytes per String? 40
    > >> seems ridiculously high. Where'd you get that number?

    >
    > Might be high, but doesn't seem ridiculous. A String
    > instance has its object-ness plus three ints and a char[]
    > reference. The char[] will have its own object-ness plus
    > a length. If object-ness takes eight bytes and a reference
    > takes four (32-bit JVM), we're up to at least 36 bytes. If
    > there's an eight-byte alignment requirement, 40 bytes is
    > right on the money.
    >
    > > On the machine you are on is it possible to increase your max heap
    > > space higher? If so problem solved!

    >
    > He's trying to create 2G pairs of String objects (72-80
    > bytes of overhead per pair) plus 2G HashMap.Entry objects
    > (~24 bytes each, under the same set of assumptions), hence
    > about 192-208GB -- and that's not counting the "payload" of
    > the char values, which will add another 80GB or thereabouts.
    > 272GB is well beyond what a 32-bit JVM can manage, so he'll
    > need to go to a 64-bit version -- and this will expand the
    > String overhead to at least 40 bytes, and the HashMap.Entry
    > size to at least 36-40, raising the total to about 312-320GB.
    > And that's just for this one data structure; the rest of the
    > Java environment needs some heap, too. I'd guess that a
    > machine with half a terabyte of RAM could just about manage
    > it, with luck and a following wind.
    >
    > ... but in truth he's not really trying to populate this
    > enormous and pointlessly wasteful data structure; he's trying
    > to understand how the NetBeans profiler reports thememory
    > used by the program, and I'm afraid I can't help much with
    > that. After 4M pairs he sees
    >
    > > String -- 48MB
    > > char[] -- 48 MB
    > > HeapCharBuffer -- 45MB
    > > Integer -- 19MB
    > > HashMap$Entry -- 19MB

    >
    > ... which works out to 6 bytes per string, 6 bytes per char[],
    > and <5 bytes per HashMap.Entry -- clearly, the profiler reports
    > some other notion of "memoryused" than we've been talking
    > about in the last few messages. All I can suggest is RTFM.
    >
    > --
    >


    Obviously my intention is not to fill 2G of pairs. If you thought so I
    think you are dumber than I am. The point is to understand when the
    JVM craps out how much of memory is taken up by the VM apart from the
    objects used by the application. The profiler reports that the memory
    used by the objects in the heap is atmost 200M. But the heap memory
    used is at 800M. So why is there a 4x increase? Even if you throw in
    200M used by the profiler there is an additional 400M that is
    unaccounted for. I even tried MinHeapRatio and MaxHeapRatio options
    but to no avail. I agree that there will be some overhead with Java
    but 3x or 4x is seriously flawed. I am wondering if someone can point
    me to what might be going wrong.

    A
    , Jan 31, 2008
    #8
  9. Lew Guest

    wrote:
    > The point is to understand when the
    > JVM craps out how much of memory is taken up by the VM apart from the
    > objects used by the application. The profiler reports that the memory
    > used by the objects in the heap is atmost 200M. But the heap memory
    > used is at 800M. So why is there a 4x increase?


    How are you determining that the heap consumption is 800MB ?

    --
    Lew
    Lew, Feb 1, 2008
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jay Kavimandan

    Memory utilization for Windows services

    Jay Kavimandan, Oct 25, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    1,482
    Prasad
    Nov 2, 2004
  2. pt
    Replies:
    1
    Views:
    2,234
    Aaron W. West
    Jun 4, 2004
  3. Replies:
    1
    Views:
    2,656
    Stephen Kellett
    Mar 12, 2006
  4. Anon
    Replies:
    12
    Views:
    553
  5. Replies:
    1
    Views:
    286
    Michael Torrie
    Jul 29, 2007
Loading...

Share This Page