really lare arrays

[bharath]

I have 5 sequences (A B C D and E) of 100 elements each. But I am
using a recursion

to create E1 the size of E,

D1 whose size is product of D and E1 = 10^4.

C1 whose size is product of C and D1 = 10^6

and so on till I get A1 that has 10^10 elements. Now how do I
implement this? most arrays are not allowed to have more than approx
10^4 elements.

It is quite likely in the process that D1, C1, B! and A1 shrink in
size after accounting for repeating elements and the total of 10^10
might end up being smaller than 10^4.

 

[Jonathan Lee]

and so on till I get A1 that has 10^10 elements. Now how do I
implement this? most arrays are not allowed to have more than approx
10^4 elements.

There's basically two answers I can think of:

1) Whatever you are doing, use a different algorithm. I have never
seen a serious algorithm that started with "Allocate about 10GB of
memory..." Related: use a different data structure. For example, if
you're using matrices, can you use a sparse matrix?

2) If you _really_ need this much memory AT ONCE and supposing your
computer will allocate it for you, you'll have to split up the data
into separate objects that _will_ fit in memory. You'll also need an
object to organize it all. Furthermore, 10^10 presents an addressing
problem. On a 32 bit machine, for instance, you can't actually index
every element of an array that size. And you'll never get it into a
single process' memory for basically the same reason. However, _each_
process can get a couple gigs of memory so maybe you can pipe the data
back and forth between processes...

In other words, use distributed computing techniques.

I think, however, that (1) is far more realistic advice. Maybe tell us
what the actual problem is and we can be more specific with advice.

--Jonathan