size limits of arrays/vectors ?

[William Zumwalt]

Hey all,

I'm writing a science application that was previously written in Fortan
and it has arrays that literally had millions of elements. Fortunately
the elements were integers. I'm moving it to Java because of the GUI and
portability is a must. Can Java handle Vectors (or at least primitive
arrays) of this size? The machine certainly has the resources.

 

[Lee Weiner]

I'm writing a science application that was previously written in Fortan
and it has arrays that literally had millions of elements. Fortunately
the elements were integers. I'm moving it to Java because of the GUI and
portability is a must. Can Java handle Vectors (or at least primitive
arrays) of this size? The machine certainly has the resources.

It took me about 90 seconds to test this out, probably less time than it took
you to post the question.

public static void main ( String[] args )
{
Vector vec = new Vector();
for( int i = 0; i < 2500000; i++ )
vec.add( new Integer( i ) );
}

This one worked. When I changed the number to 3,000,000, I got an OutOfMemory
Exception.

Lee Weiner
lee AT leeweiner DOT org

 

[Roedy Green]

I'm writing a science application that was previously written in Fortan
and it has arrays that literally had millions of elements. Fortunately
the elements were integers. I'm moving it to Java because of the GUI and
portability is a must. Can Java handle Vectors (or at least primitive
arrays) of this size? The machine certainly has the resources.

Java ints are 32 bits so arrays and Vectors can hold +2,147,483,647
elements [2^31-1] aka Integer.MAX_VALUE.

You eventually bang into the addressing limit of your hardware.

Sun is offering 64 bit workstations at a discount to help you over
that hump.

 

[Roedy Green]

When I changed the number to 3,000,000, I got an OutOfMemory
Exception.

which could be extended with a java.exe command line option. See
http://mindprod.com/jgloss/javaexe.html

 

[Chris Uppal]

Vector vec = new Vector();
for( int i = 0; i < 2500000; i++ )
vec.add( new Integer( i ) );

Argh!!! Don't even *consider* using Vectors of Integers to hold huge datasets.

Consider the memory reqs (assuming the typical 8-byte overhead for an object
header):

For an array of 2500000 ints:
2,500,000 * 4
= 10 MBytes (plus a little bit for the array header).

For a Vector (or any other collection) of 2500000 Integers:
2,500,000 * (4 + 8) for the Integer objects.
2,500,000 * 4 for the internal array of refs to the Integers
= 50 MBytes (plus a little bit for the Vector itself).

And the difference will be even bigger on a 64-bit JVM.

That's not to mention the massive extra load that 2.5 million extra object will
put on the garbage collector.

Nor does it mention the horrible way that 2.5 million pointers will largely
destroy any cache coherence in memory accesses.

(Of course, the same observations go for Vectors of Floats, Doubles, etc).

Lee, William, I'm sorry if I'm saying things that are already obvious to you.
But I've seen several recent posts by people pulling exactly this kind of trick
(and then wondering why they had problems!), so I don't think the point can be
made strongly enough.

(And it also suggests that there's something wrong with the way that we are
teaching programming, but that's another story).

-- chris