Array Size - Upper Limit

[D. Susman]

Hi,

I know that this issue is indeed strictly operating system dependent
but I am just curious:

I have a five dimensional array, whose size sums to 68 MB (almost).
This array is contained by class X. When I simply attempt to create an
instance of that class, I get segmentation fault.

I am working on Solaris (which has 8 KB pages ). What may the case be?
There seems to be an upper limit for the size of an array.

Thanks.

 

[Victor Bazarov]

I know that this issue is indeed strictly operating system dependent
but I am just curious:

I have a five dimensional array, whose size sums to 68 MB (almost).
This array is contained by class X. When I simply attempt to create an
instance of that class, I get segmentation fault.

What does "simply attempt to create an instance" mean? An automatic
object? Have you tried creating it dynamically?

I am working on Solaris (which has 8 KB pages ). What may the case be?

If that's an automatic object you're trying to create, your stack is
not big enough to contain it. Try increasing the stack size (usually
it is a linker option, RTFM to learn which one it's on your system).

Also, consider that putting a 68 MB stress on your stack may not be
a good idea.

There seems to be an upper limit for the size of an array.

.... but I am not certain you've reached it yet. There are other,
more severe limitations on your program, and they can be at play
here.

V

 

[Jim Langston]

Hi,

I know that this issue is indeed strictly operating system dependent
but I am just curious:

I have a five dimensional array, whose size sums to 68 MB (almost).
This array is contained by class X. When I simply attempt to create an
instance of that class, I get segmentation fault.

A stack seems to always be smaller than a free store. There are usually
ways when compiling a program to increase the stack size (see your compiler
documentation) but 68MB is probably pushing it. As Victor said, you should
probably move the allocation of the memory to the freestore, which generally
has a much larger limit. Use new and some sort of smart pointer (I
personally would use a naked pointer, but that is generally frouwned upon).

I am working on Solaris (which has 8 KB pages ). What may the case be?
There seems to be an upper limit for the size of an array.

Check your compiler documentation for increasing the size of the stack.
That documentation should state the default size of the stack and how large
it can be increased to with some compiler switch.

 

[James Kanze]

What does "simply attempt to create an instance" mean? An automatic
object? Have you tried creating it dynamically?

Or statically? Or throwing it as an exception?

If that's an automatic object you're trying to create, your stack is
not big enough to contain it. Try increasing the stack size (usually
it is a linker option, RTFM to learn which one it's on your system).

Usually, it's a simple command which you execute before invoking
the program. Under Solaris, "ulimit -s unlimited" should work.
(With ulimit -s set to unlimited, you can get as much stack as
you could get with dynamic allocation.)

Also, consider that putting a 68 MB stress on your stack may
not be a good idea.

Doing it once shouldn't cause any problems on a modern machine
(32 bits or more). I wouldn't suggest doing it in a lot of
different functions, which might end up calling each other, but
if there's only one instance for the life of the program,
declaring it in main is probably all right. (With a data
structure of this size, he's not going to be porting to a 16 bit
machine anyway.)

... but I am not certain you've reached it yet.

The formal upper limit is numeric_limits<size_t>::max(). Under
Solaris, either 4E09 or 1.8E19, depending on compiler options.
In practice, of course, the program will have to allocate memory
for it at some time, and that's likely to fail long before you
reach the formal limit (especially in the second case).

There are other, more severe limitations on your program, and
they can be at play here.

Yes. Note that even if the system accepts the allocation, if
you start accessing it randomly, cache misses will slow the
program down considerably. For even larger allocations, where
there isn't enough physical memory, page faults can slow the
system down to the point of becoming unusable. (When I test our
application here under Purify---which increases the memory
footprint significantly---, I have to terminate every other
application on the system. Otherwise, X gets paged out, and the
disk starts thrashing every time I move the cursor.)

 

[James Kanze]

A stack seems to always be smaller than a free store.

On the systems I use regularly (Solaris on Sparc and Linux on a
PC), the stack and the free store compete for the same memory,
with the first to ask for it getting it. (Once allocated for
one, the memory can never be used for the other.)

By default, an artificial limit is imposed by the OS on the
stack size, so that a program with endless recursion will
terminate in my lifetime. The user can normally specify this
limit (8 Meg by default on my Sparc), however, or specify that
he doesn't want it at all.

Of course, if the stack becomes so big that you start paging on
every function call, it's going to slow things down a bit.
(Function calls don't actually require any memory accesses on a
Sparc, but it would be a rare function that didn't actually make
some accesses somewhere, at least if the code isn't optimized.)

There are usually ways when compiling a program to increase
the stack size (see your compiler documentation) but 68MB is
probably pushing it.

One, you don't have to recompile, just modify the runtime
environment, and two, on a modern machine (at least on a PC or a
Sparc), there's absolutely no problem with 68MB on the stack if
you've got enough memory. (On one of the Linux machines here,
with a fair amount of memory, I was able to recurse 30 times
with a data structure of 68 Meg on the stack. On my Sparc, with
a lot less real memory, I interrupted the process after 16
recursions, because the disk thrashing was interfering with
other more important processes.)

As Victor said, you should probably move the allocation of the
memory to the freestore, which generally has a much larger
limit. Use new and some sort of smart pointer (I personally
would use a naked pointer, but that is generally frouwned
upon).

It depends on where and what. If he's only allocating a single
instance, either a static variable or an automatic variable in
main would do the job just as well. (I'd go for the static
instance, since this would mean that the program could run in
the default environment, without the user having to invoke
ulimit first.) If he needs several, at least under Solaris, it
really doesn't matter whether he uses dynamic allocation or auto
variables, as far as memory goes. I'd go with dynamic
allocation, however, because it allows error checking---you're
informed if the allocation fails, rather than just core dumping.

Check your compiler documentation for increasing the size of
the stack. That documentation should state the default size
of the stack and how large it can be increased to with some
compiler switch.

"man ulimit" should be sufficient.