Creating large arrays..

[hamishd]

What is the best way to store large arrays of numbers (eg, 4-byte
integers)?

Say I want to store an array of 1billion 4-byte integers.. If my
computer has > 4gB memory, then is this possible?

int Large_Array[1000000000];

This will cause a "stack overflow" when i try and execute. How can I
do this?

 

[Alf P. Steinbach]

* hamishd:

What is the best way to store large arrays of numbers (eg, 4-byte
integers)?

Say I want to store an array of 1billion 4-byte integers.. If my
computer has > 4gB memory, then is this possible?

Depends on the computer and C++ implementation.

int Large_Array[1000000000];

This will cause a "stack overflow" when i try and execute. How can I
do this?

You might try to allocate it dynamically, by using

std::vector<int> Large_Array( 1000000000 );

However, if you're skirting the limits of your computer's memory
capacity, then it's not a good idea even if it seems to work.

You'd be better off using some disk-based structure, processing only
parts of it at a time.


Cheers, & hth.,

- Alf

 

[hamishd]

* hamishd:

What is the best way to store large arrays of numbers (eg, 4-byte
integers)?

Say I want to store an array of 1billion 4-byte integers.. If my
computer has > 4gB memory, then is this possible?

Depends on the computer and C++ implementation.

int Large_Array[1000000000];

This will cause a "stack overflow" when i try and execute. How can I
do this?

You might try to allocate it dynamically, by using

   std::vector<int> Large_Array( 1000000000 );

However, if you're skirting the limits of your computer's memory
capacity, then it's not a good idea even if it seems to work.
thanks.

You'd be better off using some disk-based structure, processing only
parts of it at a time.

What are the quicker ways of doing this?

 

[Alf P. Steinbach]

* hamishd:

* hamishd:

What is the best way to store large arrays of numbers (eg, 4-byte
integers)?
Say I want to store an array of 1billion 4-byte integers.. If my
computer has > 4gB memory, then is this possible?

Depends on the computer and C++ implementation.

int Large_Array[1000000000];
This will cause a "stack overflow" when i try and execute. How can I
do this?

You might try to allocate it dynamically, by using

std::vector<int> Large_Array( 1000000000 );

However, if you're skirting the limits of your computer's memory
capacity, then it's not a good idea even if it seems to work.
thanks.

You'd be better off using some disk-based structure, processing only
parts of it at a time.

What are the quicker ways of doing this?

Depends on what you're doing.

E.g. it might be that what you're doing best fits some kind of tree
structure (note: a B-tree isn't a binary tree ).

Or it might be that what you're really doing is just some kind of
sequential processing, in which case read in suitably large chunks of a
file at a time, process them, and write them back or to some other file.

If it weren't for the size problem a memory mapped file would be ideal,
and IIRC the Boost library provides some fairly portable support for that.

However, the size problem means resorting to old-fashioned reads and
writes, and the clue there for efficiency is to do large enough chunks,
and perhaps also preferentially going down to at least the C file i/o
level instead of C++ iostreams (but it might be good idea to measure!).


Cheers, & hth.,

- Alf

 

[tOmMy]

why not use pointer instead of array?

 

[hamishd]

* hamishd:

* hamishd:
What is the best way to store large arrays of numbers (eg, 4-byte
integers)?
Say I want to store an array of 1billion 4-byte integers.. If my
computer has > 4gB memory, then is this possible?
Depends on the computer and C++ implementation.
int Large_Array[1000000000];
This will cause a "stack overflow" when i try and execute. How can I
do this?
You might try to allocate it dynamically, by using
   std::vector<int> Large_Array( 1000000000 );
However, if you're skirting the limits of your computer's memory
capacity, then it's not a good idea even if it seems to work.

thanks.

You'd be better off using some disk-based structure, processing only
parts of it at a time.

What are the quicker ways of doing this?

Depends on what you're doing.

I'm storing the frequency that a value occurs (value ranging from 1..
a billion).

However.. I suspect the array maybe quite sparce. That is, not all
values will occur at all, so there may be a large number of zeroes.

 

[Daniel Kraft]

What is the best way to store large arrays of numbers (eg, 4-byte
integers)?
Say I want to store an array of 1billion 4-byte integers.. If my
computer has > 4gB memory, then is this possible?
Depends on the computer and C++ implementation.
int Large_Array[1000000000];
This will cause a "stack overflow" when i try and execute. How can I
do this?
You might try to allocate it dynamically, by using
std::vector<int> Large_Array( 1000000000 );
However, if you're skirting the limits of your computer's memory
capacity, then it's not a good idea even if it seems to work.
thanks.
You'd be better off using some disk-based structure, processing only
parts of it at a time.
What are the quicker ways of doing this?

Depends on what you're doing.

I'm storing the frequency that a value occurs (value ranging from 1..
a billion).

However.. I suspect the array maybe quite sparce. That is, not all
values will occur at all, so there may be a large number of zeroes.

You could use either a hash-table or even std::map<int, int> for this
job, I think.

Daniel

 

[Grizlyk]

What is the best way to store large arrays of numbers
...
If my computer has > 4gB memory
...
int Large_Array[1000000000];
This will cause a "stack overflow" when i try and execute.

Do you want to write portable application? C++ knows nothing about
installed memory of your machine. There are some layers between C++
and hardware, you need to refer to the layers to make portable
implementation of the array.

Do you want to write concrete hardware application? You need to refer
to the concrete hardware. For some executable environment more than 32
bit address RAM can be allowed for you and you will not get "stack
overflow" there.

For example, you can process the array with own wrapper (user defined
class) even for very low-memory executable environment or with limited
or poor virtual memory management, implementing the large array
storage as binary file with random access to manage memory explicitly.
The own wrapper also can use own types to represent indexes of the
array (to remove limits of xxx_MAX).

I think, long arrays are known for many years, so must be well-known
wrappers for the work.

Maksim A. Polyanin
http://grizlyk1.narod.ru/cpp_new