How to get the current allocated total size of an ArrayList?

Discussion in 'Java' started by Robin Wenger, Feb 23, 2011.

  1. Robin Wenger

    Robin Wenger Guest

    Assume I setup an ArrayList like:

    ArrayList<String> mylines = new ArrayList<String>();

    and add lots of lines/strings to it:

    while(...)
    mylines.add(...);

    How can I find out the total size (in Bytes) which is currently allocated by the ArrayList?

    Occasionally I get an OutOfMemory exception. Can I prevent this by preallocating everything in one step
    in advance instead of doing this little by little with every add()?

    Robin
     
    Robin Wenger, Feb 23, 2011
    #1
    1. Advertising

  2. Robin Wenger

    markspace Guest

    On 2/23/2011 7:43 AM, Robin Wenger wrote:

    > How can I find out the total size (in Bytes) which is currently allocated by the ArrayList?



    You can't. I suggest you look at Runtime.freeMemory() instead.
     
    markspace, Feb 23, 2011
    #2
    1. Advertising

  3. Robin Wenger

    Lew Guest

    On 02/23/2011 10:50 AM, markspace wrote:
    > On 2/23/2011 7:43 AM, Robin Wenger wrote:
    >> Assume I setup an ArrayList like:
    >>
    >> ArrayList<String> mylines = new ArrayList<String>();
    >>
    >> and add lots of lines/strings to it:
    >>
    >> while(...)
    >> mylines.add(...);
    >>
    >> How can I find out the total size (in Bytes) which is currently allocated by
    >> the ArrayList?
    >>
    >> Occasionally I get an OutOfMemory exception [sic]. Can I prevent this by
    >> preallocating everything in one step
    >> in advance instead of doing this little by little with every add()?


    No.

    > This post has followups set to cljp, btw, for those reading cljh.


    How annoying.

    The amount of memory involved is hard to estimate without more details.
    Strings don't always live on the heap.

    If you're getting OutOfMemoryError (OOME) [1] it's not necessarily (or even
    likely) the strings that are causing it. If they are, then you should
    consider external storage for the data.

    More likely, and your question implies that this is relevant, you are failing
    to let the garbage collector (GC) do its job. [2] [3] [4]

    While Java cannot have memory leaks in the classic sense, it can have
    so-called "memory leaks", more nearly characterized as memory packratting,
    where your program keeps a reference alive after it's finished with it. This
    is the usual cause of OOME.

    While usually you would avoid the practice of explicit nulling ('foo = null;')
    to release memory, there are a few times when you need it. [2] These include
    spurious references held by collections, which might apply to you.

    If you don't let references die, the sooner the better, the GC cannot do its
    job. Objects that live too long move to the geriatric heap ("tenured
    generation") and are tougher to kill. You want to slay them young - create
    instances to the most local appropriate scope and avoid long-lived references
    to them in collections and elsewhere.

    Strings are a bit of a special case. If your strings are constants, i.e.,
    declared using quoted literals ( "some content" ), they are stored in the
    'String' intern pool and that's that. Strings declared via 'new String()',
    either implicitly or explicitly, usually live on the heap. So your memory
    allocation for that 'List' depends in part on where the strings come from.

    The optimizer can mess with your calculations, too. But most likely figuring
    the size of your 'List' is not going to help. If it's really that big,
    consider buying a cartload of RAM and running 64-bit, or moving to external
    resources for the strings. (DBMS or doc-management system, anyone?)

    Profilers and similar tools, including those inbuilt to Java, help you
    determine if you have a packratting problem, which if I were to wish to make
    money I'd bet you do. Except no one would take the bet.

    Since you haven't shared your code with us, all we can do is speculate. [5]

    [1]
    <http://download.oracle.com/javase/6/docs/api/java/lang/OutOfMemoryError.html>

    [2]
    <http://www.ibm.com/developerworks/java/library/j-jtp01274.html>
    See also:
    [3]
    <http://www.ibm.com/developerworks/java/library/j-jtp03216.html>
    [4]
    <http://www.ibm.com/developerworks/java/library/j-jtp09275.html>

    [5]
    http://sscce.org/

    --
    Lew
    Honi soit qui mal y pense.
     
    Lew, Feb 23, 2011
    #3
  4. Robin Wenger

    Jim Janney Guest

    (Robin Wenger) writes:

    > Assume I setup an ArrayList like:
    >
    > ArrayList<String> mylines = new ArrayList<String>();
    >
    > and add lots of lines/strings to it:
    >
    > while(...)
    > mylines.add(...);
    >
    > How can I find out the total size (in Bytes) which is currently allocated by the ArrayList?
    >
    > Occasionally I get an OutOfMemory exception. Can I prevent this by preallocating everything in one step
    > in advance instead of doing this little by little with every add()?
    >
    > Robin


    Assuming the file doesn't change, you might want to consider a different
    approach. Scan the file once and build a list that stores the offset
    for the beginning of each line. When you need a line, get its offset
    from the list and use a RandomAccessFile to read the line. This uses
    much less memory and still gives you constant-time access to each line.

    --
    Jim Janney
     
    Jim Janney, Feb 23, 2011
    #4
  5. On 02/23/2011 10:43 AM, Robin Wenger wrote:
    > Occasionally I get an OutOfMemory exception. Can I prevent this by preallocating everything in one step
    > in advance instead of doing this little by little with every add()?


    Where is all of your memory going? It could be that it's not your array
    that's eating up the memory.

    Another question is why are you storing every line of a file in memory?
    If your file is really so large that it eats up memory with mere storage
    (so I'm talking on the order of 100s of MB or larger), then you are much
    better off going with a more stream-based approach of access, or perhaps
    instead a compressed storage format of lines (just store the starting
    file offset of each line).

    --
    Beware of bugs in the above code; I have only proved it correct, not
    tried it. -- Donald E. Knuth
     
    Joshua Cranmer, Feb 23, 2011
    #5
  6. Robin Wenger

    Eric Sosman Guest

    On 2/23/2011 10:43 AM, Robin Wenger wrote:
    > Assume I setup an ArrayList like:
    >
    > ArrayList<String> mylines = new ArrayList<String>();
    >
    > and add lots of lines/strings to it:
    >
    > while(...)
    > mylines.add(...);
    >
    > How can I find out the total size (in Bytes) which is currently allocated by the ArrayList?


    There's no way to be sure, but you can get an estimate. First,
    suppose that the ArrayList uses some fixed amount of overhead plus space
    for its capacity to store object references. Each reference is probably
    four or eight bytes, depending on whether you use a 32-bit or 64-bit
    JVM. If the ArrayList is large (so the fixed overhead is negligible),
    all you need to do is multiply the capacity by four or eight, and you'll
    be fairly close.

    There are two problems with this, or possibly more. First, it only
    counts the memory used by the ArrayList itself, and not by the objects
    that it refers to: If you've filled it with four-byte references to
    megabyte objects, you'll be counting the fours and not the megs.

    Second, there's no way to query the capacity of an ArrayList! You
    know it's at least as great as the size(), but it could be larger by an
    unknown amount. You could call trimToSize() to set the capacity equal
    to the size(), but the equality would come unglued as soon as you added
    the next element (or, conceivably, deleted one). If you're populating
    an ArrayList that will thereafter remain unchanged, it's possible to
    find the capacity -- but if items are coming and going, the exercise is
    fairly futile.

    > Occasionally I get an OutOfMemory exception. Can I prevent this by preallocating everything in one step
    > in advance instead of doing this little by little with every add()?


    You might, yes, if you're just a *tiny* bit too large. Remember,
    the ArrayList does not hold those megabyte objects, just references to
    them -- and the references are small. Saving the memory for a few
    references may save kilobytes, but won't save the megabytes that the
    referred-to objects themselves occupy.

    You haven't revealed much about the context of your problem, but
    even so I suspect very strongly that you're barking up the wrong tree.
    It's probably those gazillions of Strings at a couple dozen bytes each
    that use up your memory, not the four or eight bytes per String stored
    in the ArrayList. Stop worrying about the cigarette butts on the
    beach, and turn your attention to the whale carcasses.

    --
    Eric Sosman
    d
     
    Eric Sosman, Feb 24, 2011
    #6
  7. Robin Wenger wrote:
    > Assume I setup an ArrayList like:
    >
    > ArrayList<String> mylines = new ArrayList<String>();
    >
    > and add lots of lines/strings to it:
    >
    > while(...)
    > mylines.add(...);
    >
    > How can I find out the total size (in Bytes) which is currently
    > allocated by the ArrayList?
    >
    > Occasionally I get an OutOfMemory exception. Can I prevent this
    > by preallocating everything in one step in advance instead of
    > doing this little by little with every add()?


    1. There is a -XX: option to have the JVM dump the heap on OOM.
    2. Run the program in your IDE's debug mode.
    3. Use a profiler, e.g. JProfiler.

    --

    "I'm a doctor, not a mechanic." Dr Leonard McCoy <>
    "I'm a mechanic, not a doctor." Volker Borchert <>
     
    Volker Borchert, Feb 26, 2011
    #7
  8. Robin Wenger

    Lew Guest

    On 02/26/2011 03:17 AM, Volker Borchert wrote:
    > Robin Wenger wrote:
    >> Assume I setup an ArrayList like:
    >>
    >> ArrayList<String> mylines = new ArrayList<String>();
    >>
    >> and add lots of lines/strings to it:
    >>
    >> while(...)
    >> mylines.add(...);
    >>
    >> How can I find out the total size (in Bytes) which is currently
    >> allocated by the ArrayList?
    >>
    >> Occasionally I get an OutOfMemory exception. Can I prevent this
    >> by preallocating everything in one step in advance instead of
    >> doing this little by little with every add()?

    >
    > 1. There is a -XX: option to have the JVM dump the heap on OOM.
    > 2. Run the program in your IDE's debug mode.
    > 3. Use a profiler, e.g. JProfiler.


    Well, then, you should mention jmap / jhat.
    http://java.sun.com/developer/technicalArticles/J2SE/monitoring/

    --
    Lew
    Honi soit qui mal y pense.
     
    Lew, Feb 26, 2011
    #8
  9. Robin Wenger

    Daniel Pitts Guest

    On 2/23/2011 7:43 AM, Robin Wenger wrote:
    > Assume I setup an ArrayList like:
    >
    > ArrayList<String> mylines = new ArrayList<String>();
    >
    > and add lots of lines/strings to it:
    >
    > while(...)
    > mylines.add(...);
    >
    > How can I find out the total size (in Bytes) which is currently allocated by the ArrayList?
    >
    > Occasionally I get an OutOfMemory exception. Can I prevent this by preallocating everything in one step
    > in advance instead of doing this little by little with every add()?
    >
    > Robin
    >

    You can prevent this by:
    1) Make sure you're not holding on to objects you no longer need.
    Pack-ratting is the term for holding on to objects too long.
    2) Look into tuning your GC settings. -Xmx512m might be a good start.
    See
    <http://download.oracle.com/javase/1.3/docs/tooldocs/solaris/java.html#options>
    3) Consider streaming processing rather than loading the entire data
    set into memory before processing. This may have additional benefits on
    efficiency (depending on the processing being done). The benefit may be
    outweighed by code complexity, so use judiciously.

    Hope this helps, gl.
    --
    Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>
     
    Daniel Pitts, Feb 26, 2011
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Davide
    Replies:
    10
    Views:
    725
    Oliver Wong
    Jan 10, 2006
  2. Replies:
    5
    Views:
    662
    Matt Wharton
    Dec 9, 2004
  3. Replies:
    10
    Views:
    617
    Keith Thompson
    Jun 10, 2007
  4. red floyd
    Replies:
    1
    Views:
    321
    red floyd
    Oct 10, 2008
  5. mithun
    Replies:
    14
    Views:
    667
    Ian Collins
    Nov 12, 2009
Loading...

Share This Page