the inefficiency of noncontiguous data - why?

[wittle]

I noticed that breaking up a large array into 2 pieces results in a much
higher memory usage for my program. Example:

A program with:
float[] a = new float[32000000];
uses 138MB of memory on my system.

But a program with:
float[] b = new float[16000000];
float[] c = new float[16000000];
uses 187MB of memory, even though it's the same amount of data.

Why is this? My heap size is 800MB.

 

[Daniel Dyer]

I noticed that breaking up a large array into 2 pieces results in a much
higher memory usage for my program. Example:

A program with:
float[] a = new float[32000000];
uses 138MB of memory on my system.

But a program with:
float[] b = new float[16000000];
float[] c = new float[16000000];
uses 187MB of memory, even though it's the same amount of data.

Why is this? My heap size is 800MB.

How are you measuring memory usage?

Dan.

 

[wittle]

I should clarify that the 800MB is my maximum heap size, not my actual
starting heap size.

 

[wittle]

Using the windows task manager, "Virtual Memory Usage" column.

I noticed that breaking up a large array into 2 pieces results in a
much higher memory usage for my program. Example:

A program with:
float[] a = new float[32000000];
uses 138MB of memory on my system.

But a program with:
float[] b = new float[16000000];
float[] c = new float[16000000];
uses 187MB of memory, even though it's the same amount of data.

Why is this? My heap size is 800MB.

How are you measuring memory usage?

Dan.

--Daniel Dyer
http//www.uncommons.org

 

[Daniel Pitts]

I noticed that breaking up a large array into 2 pieces results in a
much higher memory usage for my program. Example:
A program with:
float[] a = new float[32000000];
uses 138MB of memory on my system.
But a program with:
float[] b = new float[16000000];
float[] c = new float[16000000];
uses 187MB of memory, even though it's the same amount of data.
Why is this? My heap size is 800MB.

How are you measuring memory usage?

--Daniel Dyer
http//www.uncommons.org

[Top posting fixed]
Using the windows task manager, "Virtual Memory Usage" column.

Please don't top post, it confuses the conversation...

Windows task manager does not accurately measure Java memory usage.
The JVM's allocation of memory, and the Java programs allocation of
memory don't always correspond one-to-one. It could be that your test
does a garbage collection in one instance, and not in the other... It
would be interesting to compare

float[] b = new float[16000000];
and compare it to

float[] b = new float[32000000];

and see how the memory differs.

Also, see if running the program several times has different results,
or running it inside an IDE, vs running it outside an IDE.

 

[wittle]

I noticed that breaking up a large array into 2 pieces results in a
much higher memory usage for my program. Example:
A program with:
float[] a = new float[32000000];
uses 138MB of memory on my system.
But a program with:
float[] b = new float[16000000];
float[] c = new float[16000000];
uses 187MB of memory, even though it's the same amount of data.
Why is this? My heap size is 800MB.
How are you measuring memory usage?
Dan.
--Daniel Dyer
http//www.uncommons.org

[Top posting fixed]
Using the windows task manager, "Virtual Memory Usage" column.

Please don't top post, it confuses the conversation...

Windows task manager does not accurately measure Java memory usage.
The JVM's allocation of memory, and the Java programs allocation of
memory don't always correspond one-to-one. It could be that your test
does a garbage collection in one instance, and not in the other... It
would be interesting to compare

float[] b = new float[16000000];
and compare it to

float[] b = new float[32000000];

and see how the memory differs.

Also, see if running the program several times has different results,
or running it inside an IDE, vs running it outside an IDE.

Thanks, I did a little googling to find a more accurate way to measure
my program's memory usage, and found out about the Runtime.maxMemory(),
Runtime.totalMemory(), and Runtime.freeMemory(). I created a small test
program to test the results of allocating memory in one 32000000 length
chunk vs 2 16000000 length chunks and these were the results:

1 32000000 length chunk:
Max memory: 832438272
Total memory: 130035712
Free memory: 1915488

2 16000000 length chunks
Max memory: 832438272
Total memory: 178917376
Free memory: 50663416

So it seems in the latter case the JVM does request and receive more
memory from windows for some reason, but that memory is free for my
program to use, so it is not wasted, so it's fine.

 

[xen]

I'm pretty sure there is no big difference in memory consumption.

The array itself takes up 16 bytes; that's 8 for the object, 4 for the
size, and 4 for alignment.
So your overhead is 16 bytes.
Add to that what the VM needs for memory allocation, which is also
insignificant compared to 64 MB.

 

[Roedy Green]

I noticed that breaking up a large array into 2 pieces results in a much
higher memory usage for my program.

Normally two smaller pieces would use less ram than one big piece
since two small pieces can be fit into the holes between existing
objects, where one big piece might require allocating extra VR to get
contiguous space.

However 1000 small pieces would probably take more space than one big
piece since each piece has tracking overhead, perhaps in the order of
8-32 bytes.

I would not leap to any great generalisations based on that one
example.