How to handle huge array in java?

[Wei]

Hi,

I have a java program which needs to manipulate a big 3D float array
(40x3000x3000). By definition, it'll use 40x3000x3000x4=1.4GB memory,
and my pc has only 512MB physical RAM and I set 2GB page memory.

I tried the following steps:
1. run java with option -Xmx1440m, but it seems doing Garbage
collection when it used up physical ram, and cpu usage drops to <7%,
memory usage also drops to <100MB. It seems hanging there forever.
2. added -Xnoclassgc (trying to disable garbage collection), but no help.
3. I thought about reducing the array size by using short instead of
float, but it's not acurate enough.

While I'm working on new algorithm to reduce the size of the array,
it may not be available soon. Can anyone suggest me how to handle the
big array with my current PC setting?

Thanks a lot,

Wei

 

[Harald Hein]

I have a java program which needs to manipulate a big 3D float
array (40x3000x3000). By definition, it'll use
40x3000x3000x4=1.4GB memory, and my pc has only 512MB physical RAM
and I set 2GB page memory.

Step 1: You consider redesigning your algorithm: Change from an 3D
array which needs to be accessible all at once, to a sequencial list of
values, which you read in one-by-one, manipulate, and write out. Or to
an algorithm which allows you to process the data line-by-line, column-
by-column, small tile-by-small tile, plane-by-plane or whatever clever
subset you can come up with.

If this is not possible:

Step 2: You use a real operating system, which is important because of
the next step.

Step 3: You use java.nio.MappedByteBuffer. If you OS or Java
implementation is shit, forget it. But even if you indeed get working
memory-mapped I/O, consider to use some clever algorithm so you can
localize data access, and don't have to map in the complete 1.4 GB at
once.

 

[Shripathi Kamath]

While I'm working on new algorithm to reduce the size of the array,
it may not be available soon. Can anyone suggest me how to handle the
big array with my current PC setting?

Since you are already working on a software solution, have you considered
buying more RAM? That may be the cheapest stop-gap measure.

 

[Tim Ward]

You use java.nio.MappedByteBuffer.

Only if you're happy to hold the file open for the life of the application,
which you might not be if anything else ever needs to access it. There's a
fundamental design error in MappedByteBuffer such that there's no way to
close the file - the garbage collector *might* do it for you *eventually*,
but then again it might not.

 

[Marco Schmidt]

fup2 comp.lang.java.programmer

Wei:

I have a java program which needs to manipulate a big 3D float array
(40x3000x3000). By definition, it'll use 40x3000x3000x4=1.4GB memory,
and my pc has only 512MB physical RAM and I set 2GB page memory. [...]
While I'm working on new algorithm to reduce the size of the array,
it may not be available soon. Can anyone suggest me how to handle the
big array with my current PC setting?

In case you can't use a sparse matrix as suggested elsewhere, create
some data structure that keeps only part of the array in memory while
having all of it in a file. Read parts into memory, process it and
write it back to the file. RandomAccessFile has methods readFloat /
writeFloat.

Maybe this can be done with Java Advanced Imaging (they have some
disk-based image types, and they have float image types, you could
"abuse" this for your own purposes).

Regards,
Marco

 

[Harald Hein]

Only if you're happy to hold the file open for the life of the
application, which you might not be if anything else ever needs to
access it.

You seem to have misunderstood memory-mapped IO. The file's location on
disk is mapped to virtual memory. Which means only a few entries in the
MMU are set. There is no file to close, because there is no file open.
In fact, you have handed this whole part of the disk over to you OS,
and it becomes part of the process space of the JVM.

Memory-mapped IO circumvents significant parts of the file IO system.
If you map a 1.4 GB file to memory, you don't have 1.4 GB in main
memory, you have just gotten a range of 1.4 GB of virtual memory
reserved for you.

If you start to access this virtual memory range, the OS will page in
(and out!) the parts of the file you accesss from virtual to physical
memory. With a real OS, and decent hardware, this is done via DMA
transfer.

There's a fundamental design error in MappedByteBuffer
such that there's no way to close the file

Not at all. No file is open, so no need to close a file. You have given
the disk space to the OS. Once this has been done, the OS manages the
space as it manages all other virtual memory. And many operating
systems just can't release (thrink) part of a process space while the
process is still running.

- the garbage collector
*might* do it for you *eventually*, but then again it might not.

It doesn't matter, because the memory is not managed by the garbage
collector, it is managed by the virtual memory management system of the
OS. And the OS will page out memory if it has a need to.

 

[Tim Ward]

You seem to have misunderstood memory-mapped IO. The file's location on
disk is mapped to virtual memory. Which means only a few entries in the
MMU are set. There is no file to close, because there is no file open.
In fact, you have handed this whole part of the disk over to you OS,
and it becomes part of the process space of the JVM.

4724038