Array Size - Upper Limit

Discussion in 'C++' started by D. Susman, Dec 18, 2007.

  1. D. Susman

    D. Susman Guest

    Hi,

    I know that this issue is indeed strictly operating system dependent
    but I am just curious:

    I have a five dimensional array, whose size sums to 68 MB (almost).
    This array is contained by class X. When I simply attempt to create an
    instance of that class, I get segmentation fault.

    I am working on Solaris (which has 8 KB pages ). What may the case be?
    There seems to be an upper limit for the size of an array.

    Thanks.
    D. Susman, Dec 18, 2007
    #1
    1. Advertising

  2. D. Susman wrote:
    > I know that this issue is indeed strictly operating system dependent
    > but I am just curious:
    >
    > I have a five dimensional array, whose size sums to 68 MB (almost).
    > This array is contained by class X. When I simply attempt to create an
    > instance of that class, I get segmentation fault.


    What does "simply attempt to create an instance" mean? An automatic
    object? Have you tried creating it dynamically?

    > I am working on Solaris (which has 8 KB pages ). What may the case be?


    If that's an automatic object you're trying to create, your stack is
    not big enough to contain it. Try increasing the stack size (usually
    it is a linker option, RTFM to learn which one it's on your system).

    Also, consider that putting a 68 MB stress on your stack may not be
    a good idea.

    > There seems to be an upper limit for the size of an array.


    .... but I am not certain you've reached it yet. There are other,
    more severe limitations on your program, and they can be at play
    here.

    V
    --
    Please remove capital 'A's when replying by e-mail
    I do not respond to top-posted replies, please don't ask
    Victor Bazarov, Dec 18, 2007
    #2
    1. Advertising

  3. D. Susman

    Jim Langston Guest

    D. Susman wrote:
    > Hi,
    >
    > I know that this issue is indeed strictly operating system dependent
    > but I am just curious:
    >
    > I have a five dimensional array, whose size sums to 68 MB (almost).
    > This array is contained by class X. When I simply attempt to create an
    > instance of that class, I get segmentation fault.


    A stack seems to always be smaller than a free store. There are usually
    ways when compiling a program to increase the stack size (see your compiler
    documentation) but 68MB is probably pushing it. As Victor said, you should
    probably move the allocation of the memory to the freestore, which generally
    has a much larger limit. Use new and some sort of smart pointer (I
    personally would use a naked pointer, but that is generally frouwned upon).

    > I am working on Solaris (which has 8 KB pages ). What may the case be?
    > There seems to be an upper limit for the size of an array.


    Check your compiler documentation for increasing the size of the stack.
    That documentation should state the default size of the stack and how large
    it can be increased to with some compiler switch.

    --
    Jim Langston
    Jim Langston, Dec 18, 2007
    #3
  4. D. Susman

    James Kanze Guest

    On Dec 18, 5:05 pm, "Victor Bazarov" <> wrote:
    > D. Susman wrote:
    > > I know that this issue is indeed strictly operating system dependent
    > > but I am just curious:


    > > I have a five dimensional array, whose size sums to 68 MB (almost).
    > > This array is contained by class X. When I simply attempt to create an
    > > instance of that class, I get segmentation fault.


    > What does "simply attempt to create an instance" mean? An automatic
    > object? Have you tried creating it dynamically?


    Or statically? Or throwing it as an exception:)?

    > > I am working on Solaris (which has 8 KB pages ). What may the case be?


    > If that's an automatic object you're trying to create, your stack is
    > not big enough to contain it. Try increasing the stack size (usually
    > it is a linker option, RTFM to learn which one it's on your system).


    Usually, it's a simple command which you execute before invoking
    the program. Under Solaris, "ulimit -s unlimited" should work.
    (With ulimit -s set to unlimited, you can get as much stack as
    you could get with dynamic allocation.)

    > Also, consider that putting a 68 MB stress on your stack may
    > not be a good idea.


    Doing it once shouldn't cause any problems on a modern machine
    (32 bits or more). I wouldn't suggest doing it in a lot of
    different functions, which might end up calling each other, but
    if there's only one instance for the life of the program,
    declaring it in main is probably all right. (With a data
    structure of this size, he's not going to be porting to a 16 bit
    machine anyway.)

    > > There seems to be an upper limit for the size of an array.


    > ... but I am not certain you've reached it yet.


    The formal upper limit is numeric_limits<size_t>::max(). Under
    Solaris, either 4E09 or 1.8E19, depending on compiler options.
    In practice, of course, the program will have to allocate memory
    for it at some time, and that's likely to fail long before you
    reach the formal limit (especially in the second case).

    > There are other, more severe limitations on your program, and
    > they can be at play here.


    Yes. Note that even if the system accepts the allocation, if
    you start accessing it randomly, cache misses will slow the
    program down considerably. For even larger allocations, where
    there isn't enough physical memory, page faults can slow the
    system down to the point of becoming unusable. (When I test our
    application here under Purify---which increases the memory
    footprint significantly---, I have to terminate every other
    application on the system. Otherwise, X gets paged out, and the
    disk starts thrashing every time I move the cursor.)

    --
    James Kanze (GABI Software) email:
    Conseils en informatique orientée objet/
    Beratung in objektorientierter Datenverarbeitung
    9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
    James Kanze, Dec 19, 2007
    #4
  5. D. Susman

    James Kanze Guest

    On Dec 18, 6:29 pm, "Jim Langston" <> wrote:
    > D. Susman wrote:
    > > I know that this issue is indeed strictly operating system dependent
    > > but I am just curious:


    > > I have a five dimensional array, whose size sums to 68 MB (almost).
    > > This array is contained by class X. When I simply attempt to create an
    > > instance of that class, I get segmentation fault.


    > A stack seems to always be smaller than a free store.


    On the systems I use regularly (Solaris on Sparc and Linux on a
    PC), the stack and the free store compete for the same memory,
    with the first to ask for it getting it. (Once allocated for
    one, the memory can never be used for the other.)

    By default, an artificial limit is imposed by the OS on the
    stack size, so that a program with endless recursion will
    terminate in my lifetime. The user can normally specify this
    limit (8 Meg by default on my Sparc), however, or specify that
    he doesn't want it at all.

    Of course, if the stack becomes so big that you start paging on
    every function call, it's going to slow things down a bit:).
    (Function calls don't actually require any memory accesses on a
    Sparc, but it would be a rare function that didn't actually make
    some accesses somewhere, at least if the code isn't optimized.)

    > There are usually ways when compiling a program to increase
    > the stack size (see your compiler documentation) but 68MB is
    > probably pushing it.


    One, you don't have to recompile, just modify the runtime
    environment, and two, on a modern machine (at least on a PC or a
    Sparc), there's absolutely no problem with 68MB on the stack if
    you've got enough memory. (On one of the Linux machines here,
    with a fair amount of memory, I was able to recurse 30 times
    with a data structure of 68 Meg on the stack. On my Sparc, with
    a lot less real memory, I interrupted the process after 16
    recursions, because the disk thrashing was interfering with
    other more important processes.)

    > As Victor said, you should probably move the allocation of the
    > memory to the freestore, which generally has a much larger
    > limit. Use new and some sort of smart pointer (I personally
    > would use a naked pointer, but that is generally frouwned
    > upon).


    It depends on where and what. If he's only allocating a single
    instance, either a static variable or an automatic variable in
    main would do the job just as well. (I'd go for the static
    instance, since this would mean that the program could run in
    the default environment, without the user having to invoke
    ulimit first.) If he needs several, at least under Solaris, it
    really doesn't matter whether he uses dynamic allocation or auto
    variables, as far as memory goes. I'd go with dynamic
    allocation, however, because it allows error checking---you're
    informed if the allocation fails, rather than just core dumping.

    > > I am working on Solaris (which has 8 KB pages ). What may
    > > the case be? There seems to be an upper limit for the size
    > > of an array.


    > Check your compiler documentation for increasing the size of
    > the stack. That documentation should state the default size
    > of the stack and how large it can be increased to with some
    > compiler switch.


    "man ulimit" should be sufficient.

    --
    James Kanze (GABI Software) email:
    Conseils en informatique orientée objet/
    Beratung in objektorientierter Datenverarbeitung
    9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
    James Kanze, Dec 19, 2007
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Gaurav Sharma

    upper limit - no of fstreams

    Gaurav Sharma, Nov 2, 2004, in forum: C++
    Replies:
    1
    Views:
    293
    hari4063
    Nov 2, 2004
  2. =?iso-8859-1?q?Leandro?=

    upper memory limit

    =?iso-8859-1?q?Leandro?=, Aug 6, 2004, in forum: Python
    Replies:
    4
    Views:
    551
    Tim Roberts
    Aug 10, 2004
  3. Replies:
    1
    Views:
    1,079
    Victor Bazarov
    Jun 28, 2005
  4. xor

    Time upper limit

    xor, Apr 10, 2004, in forum: Ruby
    Replies:
    7
    Views:
    145
    Sam Roberts
    Apr 13, 2004
  5. BlackHelicopter
    Replies:
    0
    Views:
    515
    BlackHelicopter
    Jan 31, 2013
Loading...

Share This Page