How to get the current allocated total size of an ArrayList?

[Robin Wenger]

Assume I setup an ArrayList like:

ArrayList<String> mylines = new ArrayList<String>();

and add lots of lines/strings to it:

while(...)
mylines.add(...);

How can I find out the total size (in Bytes) which is currently allocated by the ArrayList?

Occasionally I get an OutOfMemory exception. Can I prevent this by preallocating everything in one step
in advance instead of doing this little by little with every add()?

Robin

 

[markspace]

How can I find out the total size (in Bytes) which is currently allocated by the ArrayList?

You can't. I suggest you look at Runtime.freeMemory() instead.

 

[Lew]

Assume I setup an ArrayList like:

ArrayList<String> mylines = new ArrayList<String>();

and add lots of lines/strings to it:

while(...)
mylines.add(...);

How can I find out the total size (in Bytes) which is currently allocated by
the ArrayList?

Occasionally I get an OutOfMemory exception [sic]. Can I prevent this by
preallocating everything in one step
in advance instead of doing this little by little with every add()?

No.

This post has followups set to cljp, btw, for those reading cljh.

How annoying.

The amount of memory involved is hard to estimate without more details.
Strings don't always live on the heap.

If you're getting OutOfMemoryError (OOME) [1] it's not necessarily (or even
likely) the strings that are causing it. If they are, then you should
consider external storage for the data.

More likely, and your question implies that this is relevant, you are failing
to let the garbage collector (GC) do its job. [2] [3] [4]

While Java cannot have memory leaks in the classic sense, it can have
so-called "memory leaks", more nearly characterized as memory packratting,
where your program keeps a reference alive after it's finished with it. This
is the usual cause of OOME.

While usually you would avoid the practice of explicit nulling ('foo = null;')
to release memory, there are a few times when you need it. [2] These include
spurious references held by collections, which might apply to you.

If you don't let references die, the sooner the better, the GC cannot do its
job. Objects that live too long move to the geriatric heap ("tenured
generation") and are tougher to kill. You want to slay them young - create
instances to the most local appropriate scope and avoid long-lived references
to them in collections and elsewhere.

Strings are a bit of a special case. If your strings are constants, i.e.,
declared using quoted literals ( "some content" ), they are stored in the
'String' intern pool and that's that. Strings declared via 'new String()',
either implicitly or explicitly, usually live on the heap. So your memory
allocation for that 'List' depends in part on where the strings come from.

The optimizer can mess with your calculations, too. But most likely figuring
the size of your 'List' is not going to help. If it's really that big,
consider buying a cartload of RAM and running 64-bit, or moving to external
resources for the strings. (DBMS or doc-management system, anyone?)

Profilers and similar tools, including those inbuilt to Java, help you
determine if you have a packratting problem, which if I were to wish to make
money I'd bet you do. Except no one would take the bet.

Since you haven't shared your code with us, all we can do is speculate. [5]

[1]
<http://download.oracle.com/javase/6/docs/api/java/lang/OutOfMemoryError.html>

[2]
<http://www.ibm.com/developerworks/java/library/j-jtp01274.html>
See also:
[3]
<http://www.ibm.com/developerworks/java/library/j-jtp03216.html>
[4]
<http://www.ibm.com/developerworks/java/library/j-jtp09275.html>

[5]
http://sscce.org/

 

[Jim Janney]

Assume I setup an ArrayList like:

ArrayList<String> mylines = new ArrayList<String>();

and add lots of lines/strings to it:

while(...)
mylines.add(...);

How can I find out the total size (in Bytes) which is currently allocated by the ArrayList?

Occasionally I get an OutOfMemory exception. Can I prevent this by preallocating everything in one step
in advance instead of doing this little by little with every add()?

Robin

Assuming the file doesn't change, you might want to consider a different
approach. Scan the file once and build a list that stores the offset
for the beginning of each line. When you need a line, get its offset
from the list and use a RandomAccessFile to read the line. This uses
much less memory and still gives you constant-time access to each line.

 

[Joshua Cranmer]

Occasionally I get an OutOfMemory exception. Can I prevent this by preallocating everything in one step
in advance instead of doing this little by little with every add()?

Where is all of your memory going? It could be that it's not your array
that's eating up the memory.

Another question is why are you storing every line of a file in memory?
If your file is really so large that it eats up memory with mere storage
(so I'm talking on the order of 100s of MB or larger), then you are much
better off going with a more stream-based approach of access, or perhaps
instead a compressed storage format of lines (just store the starting
file offset of each line).

 

[Eric Sosman]

Assume I setup an ArrayList like:

ArrayList<String> mylines = new ArrayList<String>();

and add lots of lines/strings to it:

while(...)
mylines.add(...);

How can I find out the total size (in Bytes) which is currently allocated by the ArrayList?

There's no way to be sure, but you can get an estimate. First,
suppose that the ArrayList uses some fixed amount of overhead plus space
for its capacity to store object references. Each reference is probably
four or eight bytes, depending on whether you use a 32-bit or 64-bit
JVM. If the ArrayList is large (so the fixed overhead is negligible),
all you need to do is multiply the capacity by four or eight, and you'll
be fairly close.

There are two problems with this, or possibly more. First, it only
counts the memory used by the ArrayList itself, and not by the objects
that it refers to: If you've filled it with four-byte references to
megabyte objects, you'll be counting the fours and not the megs.

Second, there's no way to query the capacity of an ArrayList! You
know it's at least as great as the size(), but it could be larger by an
unknown amount. You could call trimToSize() to set the capacity equal
to the size(), but the equality would come unglued as soon as you added
the next element (or, conceivably, deleted one). If you're populating
an ArrayList that will thereafter remain unchanged, it's possible to
find the capacity -- but if items are coming and going, the exercise is
fairly futile.

Occasionally I get an OutOfMemory exception. Can I prevent this by preallocating everything in one step
in advance instead of doing this little by little with every add()?

You might, yes, if you're just a *tiny* bit too large. Remember,
the ArrayList does not hold those megabyte objects, just references to
them -- and the references are small. Saving the memory for a few
references may save kilobytes, but won't save the megabytes that the
referred-to objects themselves occupy.

You haven't revealed much about the context of your problem, but
even so I suspect very strongly that you're barking up the wrong tree.
It's probably those gazillions of Strings at a couple dozen bytes each
that use up your memory, not the four or eight bytes per String stored
in the ArrayList. Stop worrying about the cigarette butts on the
beach, and turn your attention to the whale carcasses.

 

[Volker Borchert]

Assume I setup an ArrayList like:

ArrayList<String> mylines = new ArrayList<String>();

and add lots of lines/strings to it:

while(...)
mylines.add(...);

How can I find out the total size (in Bytes) which is currently
allocated by the ArrayList?

Occasionally I get an OutOfMemory exception. Can I prevent this
by preallocating everything in one step in advance instead of
doing this little by little with every add()?

1. There is a -XX: option to have the JVM dump the heap on OOM.
2. Run the program in your IDE's debug mode.
3. Use a profiler, e.g. JProfiler.

 

[Lew]

1. There is a -XX: option to have the JVM dump the heap on OOM.
2. Run the program in your IDE's debug mode.
3. Use a profiler, e.g. JProfiler.

Well, then, you should mention jmap / jhat.
http://java.sun.com/developer/technicalArticles/J2SE/monitoring/

 

[Daniel Pitts]

Assume I setup an ArrayList like:

ArrayList<String> mylines = new ArrayList<String>();

and add lots of lines/strings to it:

while(...)
mylines.add(...);

How can I find out the total size (in Bytes) which is currently allocated by the ArrayList?

Occasionally I get an OutOfMemory exception. Can I prevent this by preallocating everything in one step
in advance instead of doing this little by little with every add()?

Robin

You can prevent this by:
1) Make sure you're not holding on to objects you no longer need.
Pack-ratting is the term for holding on to objects too long.
2) Look into tuning your GC settings. -Xmx512m might be a good start.
See
<http://download.oracle.com/javase/1.3/docs/tooldocs/solaris/java.html#options>
3) Consider streaming processing rather than loading the entire data
set into memory before processing. This may have additional benefits on
efficiency (depending on the processing being done). The benefit may be
outweighed by code complexity, so use judiciously.

Hope this helps, gl.