Very big arrays

[victorgrnt]

Hello,

I'm trying to make a dll with some stored values, looking to store
several million doubles in a static array like this:

double delta[x_size][y_size][z_size];
void delta_init() {
delta[0][0][0] = 8.65;
delta[0][0][1] = 1.35;
delta[0][0][2] = 2.11;
delta[0][0][3] = 3.29;
delta[0][0][4] = 5.147;

etc ...

I'm trying to compile the code under Visual C++ Pro 2005, and the code
just does not compile - it runs over 10 hours and just hands there. I
was able to compile smaller arrays though.

How can I make the code compile? What is a better way to do this?
Thank you for your advice.

 

[peter koch]

(e-mail address removed) skrev:

Hello,

I'm trying to make a dll with some stored values, looking to store
several million doubles in a static array like this:

double delta[x_size][y_size][z_size];
void delta_init() {
delta[0][0][0] = 8.65;
delta[0][0][1] = 1.35;
delta[0][0][2] = 2.11;
delta[0][0][3] = 3.29;
delta[0][0][4] = 5.147;

etc ...

I'm trying to compile the code under Visual C++ Pro 2005, and the code
just does not compile - it runs over 10 hours and just hands there. I
was able to compile smaller arrays though.

It must be a program with millions of lines, then?

How can I make the code compile? What is a better way to do this?
Thank you for your advice.

Technically it haven't failed yet, but I don't want to bet that the
compile will eventually succeed. But your question is platform
specific, so it is off-topic here. Go visit some group where you would
be on topic - perhaps microsoft.public.vc.language?

/Peter

 

[Salt_Peter]

Hello,

I'm trying to make a dll with some stored values, looking to store
several million doubles in a static array like this:

double delta[x_size][y_size][z_size];
void delta_init() {
delta[0][0][0] = 8.65;
delta[0][0][1] = 1.35;
delta[0][0][2] = 2.11;
delta[0][0][3] = 3.29;
delta[0][0][4] = 5.147;

etc ...

I'm trying to compile the code under Visual C++ Pro 2005, and the code
just does not compile - it runs over 10 hours and just hands there. I
was able to compile smaller arrays though.

How can I make the code compile? What is a better way to do this?
Thank you for your advice.

lol, 10 hours? Why would you store values in a dll? Wouldn't that eat
away at memory available to your program - dll must be loaded in mem?
Have you considered storing these in a file or several files? Or
perhaps develop an algorithm to initialize the values? Do you realize
that an array requires contiguous memory? Why don't you store related
values together in a struct/class and store instances of the
struct/class instead?

If i had to store points, for example, with an x, y and a z:

#include <iostream>
#include <ostream>
#include <vector>
#include <deque>

template< typename T >
struct Point
{
// universal ctor
Point(const T& x_ = T(),
const T& y_ = T(),
const T& z_ = T())
: x(x_), y(y_), z(z_) { }
// copy ctor
Point(const Point& copy)
{
x = copy.x;
y = copy.y;
z = copy.z;
}
private:
T x, y, z;
};

int main() {
// one million Points as doubles, all set to 0.0
std::deque< Point< double > > dcontainer(1000000); // done

std::cout << "double container's size = ";
std::cout << dcontainer.size() * sizeof(Point< double >);
std::cout << std::endl;

// one million Points as floats, all set to 0.0
std::deque< Point< float > > fcontainer(1000000);

std::cout << "float container's size = ";
std::cout << fcontainer.size() * sizeof(Point< float >);
std::cout << std::endl;

return 0;
}

/* output on this platform:

double container's size = 24000000 // 24 MB
float container's size = 12000000 // 12 MB

*/

 

[Jacek Dziedzic]

Hello,

I'm trying to make a dll with some stored values, looking to store
several million doubles in a static array like this:

Assuming sizeof(double)==8, several million doubles will
likely occupy several tens of MB. You really want a dll that
big?

double delta[x_size][y_size][z_size];
void delta_init() {
delta[0][0][0] = 8.65;
delta[0][0][1] = 1.35;
delta[0][0][2] = 2.11;
delta[0][0][3] = 3.29;
delta[0][0][4] = 5.147;

etc ...

I'm trying to compile the code under Visual C++ Pro 2005, and the code
just does not compile - it runs over 10 hours and just hands there. I
was able to compile smaller arrays though.

Take a look at the HDD light, perhaps the compiler ran out of
memory and started thrashing?

How can I make the code compile? What is a better way to do this?
Thank you for your advice.

What exactly disallows you to read this data from a file?

HTH,
- J.

 

[victorgrnt]

Thank you Peter & Peter.

I have a function that is slow to compute, so I pre-compute the values
and store them in a dll to retrieve in real-time. Obviously computing
the values in real-time would make the dll too slow. Few megs of RAM
that I'll need to use for it are hardly an issue on most desktops. Am I
missing something?

 

[wkaras]

Hello,

I'm trying to make a dll with some stored values, looking to store
several million doubles in a static array like this:

double delta[x_size][y_size][z_size];
void delta_init() {
delta[0][0][0] = 8.65;
delta[0][0][1] = 1.35;
delta[0][0][2] = 2.11;
delta[0][0][3] = 3.29;
delta[0][0][4] = 5.147;

etc ...

I'm trying to compile the code under Visual C++ Pro 2005, and the code
just does not compile - it runs over 10 hours and just hands there. I
was able to compile smaller arrays though.

How can I make the code compile? What is a better way to do this?
Thank you for your advice.

Why don't you use static initialization?

double delta[x_size][y_size][z_size] =
{
{
{
8.65,
1.35,
2.11,
.
.
.

 

[Markus Svilans]

lol, 10 hours? Why would you store values in a dll? Wouldn't that eat
away at memory available to your program - dll must be loaded in mem?
Have you considered storing these in a file or several files? Or

Especially if you store your numbers in a binary file, with fixed
record length, you may not even have to load the entire file into
memory (assuming disk access is fast enough for your purposes). You
could just seek to the values in the file and read them in, as you need
them. This way you could get away with using far less memory than a few
megabytes, or a few tens of megabytes.

Regards,
Markus.

 

[peter koch]

(e-mail address removed) skrev:

Thank you Peter & Peter.

I have a function that is slow to compute, so I pre-compute the values
and store them in a dll to retrieve in real-time. Obviously computing
the values in real-time would make the dll too slow. Few megs of RAM
that I'll need to use for it are hardly an issue on most desktops. Am I
missing something?

Yes. If the purpose is simply to keep some precomputed values, it would
be much better to write those values to a file and read them back when
you need them (using a std::vector for flexibility).

/Peter

 

[Henrique Bastos]

peter koch escreveu:

(e-mail address removed) skrev:
Yes. If the purpose is simply to keep some precomputed values, it would
be much better to write those values to a file and read them back when
you need them (using a std::vector for flexibility).

/Peter

I think you should store the precomputed values into a file and map
that file to memory. This is an OS feature, and by doing this, you
won´t have to worry about lots of the memory management specific
details.

Regards,
Henrique

 

[peter koch]

Henrique Bastos skrev:

peter koch escreveu:


I think you should store the precomputed values into a file and map
that file to memory. This is an OS feature, and by doing this, you
won´t have to worry about lots of the memory management specific
details.

Apart from non-portability and (probably) some faster load-time, I do
not see your method having an advantage.

/Peter

 

[ouyangsonghua]

Hi,
I have the similar problem. It looks only occur on VC 2005
I have to compile a an 47K size array
UINT8 UIRES_FONTLIB_DATA_000[2774*16+5] =
{
0x00,0x00,0xad,0x61,/* file size of fontfile */
0x00,0x01,0x00,0x00,0x00,0x0b,0x00...}

I compile OK on all old VC build(6.0, or 7.0). but recently, I upgrade
my VC to VC2005 Express, then it failed to compile for this array.
takes too long to compile, I have to ctrl+break it.
if I cut the array shorter, then It compile OK too.

who can tell what's the problem? a bug of VC2005?



Thanks,

 

[peter koch]

(e-mail address removed) skrev:

Hi,
I have the similar problem. It looks only occur on VC 2005
I have to compile a an 47K size array
UINT8 UIRES_FONTLIB_DATA_000[2774*16+5] =
{
0x00,0x00,0xad,0x61,/* file size of fontfile */
0x00,0x01,0x00,0x00,0x00,0x0b,0x00...}

I compile OK on all old VC build(6.0, or 7.0). but recently, I upgrade
my VC to VC2005 Express, then it failed to compile for this array.
takes too long to compile, I have to ctrl+break it.
if I cut the array shorter, then It compile OK too.

who can tell what's the problem? a bug of VC2005?

I believe it is. I recommend you take it to a microsoft group
(microsoft.public.vc.language if my memory serves me). There is a
service pack for the VC2005 Express - did you apply it? It just might
solve your problem.

/Peter