total array size limitations?

Discussion in 'C Programming' started by glennklockwood, Mar 13, 2009.

  1. I'm in the process of porting some FORTRAN code to C and am running
    into a problem where the program crashes if I set the size of arrays
    too large. In the below code, my program runs fine if I set MAXN to
    50000 or less, but at 60000 or more, the program immediately crashes
    and GDB indicates the error is at the printf line.

    What's strange is that I've tried this by separating my 2-dimensional
    arrays into a bunch of one-dimensional arrays (eg, xc[60000][6] into
    xc0[60000], xc1[60000], xc2[60000], etc) and the same sort of crashing
    stopped if I commented out a few of those one-dimensional
    declarations. Because of this, I know that the problem isn't with one
    giant array, but the total size of the arrays I'm declaring. I assume
    this all has to do with some sort of memory limit and a lot of spotty
    advice posted over the internet in similar cases have suggested it has
    something to do with a 64K stack size, but this doesn't make much
    sense to me.

    Can anyone shine some light on what exactly is going wrong and how I
    can avoid it? By my calculation the size of these declared variables
    is a little over 9MB which isn't a huge amount, and the FORTRAN
    version of this code works for MAXN in excess of 100000.

    If it matters, I am compiling on linux amd64, and this problem has
    happened with both GCC 4.1 and Sun's C compiler (SS12).

    Thanks much.

    glenn

    -- Begin code --

    #define MAXN 60000
    #define MAXLT 10

    int main()
    {
    double ener[100][MAXLT], ihite[100][MAXLT];
    double xc[MAXN][6], yc[MAXN][6], zc[MAXN][6];
    double aij[MAXLT][MAXLT], bij[MAXLT][MAXLT], amass[MAXLT],
    twopi[MAXLT][MAXLT], eta[MAXLT][MAXLT], ze[MAXLT][MAXLT],
    rhop[MAXLT][MAXLT], za[MAXLT];
    int ltype[MAXN], ibx[MAXN], iby[MAXN], ibz[MAXN];
    char k9[16];

    double etot, xl, yl, zl, zreft, deltim, temp;
    int nmol, istill, nsur, jread, icalc, nbin;

    int i,j;
    int dblesize, intsize;
    FORTRAN_OFFSET_TYPE offset;
    FORTRAN_OFFSET_TYPE recsize;
    FILE *fp;

    int inum[MAXLT], iff;

    recsize = 0;
    dblesize = sizeof(xl);
    intsize = sizeof(i);

    printf( " What file?\n" );

    .... (it crashes before the printf)
     
    glennklockwood, Mar 13, 2009
    #1
    1. Advertising

  2. In article <>,
    glennklockwood <> wrote:
    >I'm in the process of porting some FORTRAN code to C and am running
    >into a problem where the program crashes if I set the size of arrays
    >too large. In the below code, my program runs fine if I set MAXN to
    >50000 or less, but at 60000 or more, the program immediately crashes
    >and GDB indicates the error is at the printf line.


    Any question that starts out like this, has only one answer:

    The (total) size of automatic objects in C is strictly limited (*).
    Use globals and/or dynamic allocation (malloc and friends) if at all
    possible.

    (*) And don't even think of using the 's word' in this group.
     
    Kenny McCormack, Mar 13, 2009
    #2
    1. Advertising

  3. glennklockwood

    Bartc Guest

    "glennklockwood" <> wrote in message
    news:...
    > I'm in the process of porting some FORTRAN code to C and am running
    > into a problem where the program crashes if I set the size of arrays
    > too large. In the below code, my program runs fine if I set MAXN to
    > 50000 or less, but at 60000 or more, the program immediately crashes
    > and GDB indicates the error is at the printf line.


    > double ener[100][MAXLT], ihite[100][MAXLT];
    > double xc[MAXN][6], yc[MAXN][6], zc[MAXN][6];
    > double aij[MAXLT][MAXLT], bij[MAXLT][MAXLT], amass[MAXLT],
    > twopi[MAXLT][MAXLT], eta[MAXLT][MAXLT], ze[MAXLT][MAXLT],
    > rhop[MAXLT][MAXLT], za[MAXLT];
    > int ltype[MAXN], ibx[MAXN], iby[MAXN], ibz[MAXN];


    Try putting static in front of these big arrays. Or moving the whole lot to
    just before the main function, if the names will not clash.

    --
    Bartc
     
    Bartc, Mar 13, 2009
    #3
  4. On Mar 13, 5:21 pm, (Kenny McCormack)
    wrote:
    > The (total) size of automatic objects in C is strictly limited (*).
    > Use globals and/or dynamic allocation (malloc and friends) if at all
    > possible.
    >
    > (*) And don't even think of using the 's word' in this group.


    Making everything global certainly did the trick; thank you for the
    very quick reply. I suppose I should have checked the books a little
    more thoroughly.
     
    glennklockwood, Mar 13, 2009
    #4
  5. glennklockwood <> writes:

    > I'm in the process of porting some FORTRAN code to C and am running
    > into a problem where the program crashes if I set the size of arrays
    > too large. In the below code, my program runs fine if I set MAXN to
    > 50000 or less, but at 60000 or more, the program immediately crashes
    > and GDB indicates the error is at the printf line.
    >
    > What's strange is that I've tried this by separating my 2-dimensional
    > arrays into a bunch of one-dimensional arrays (eg, xc[60000][6] into
    > xc0[60000], xc1[60000], xc2[60000], etc) and the same sort of crashing
    > stopped if I commented out a few of those one-dimensional
    > declarations. Because of this, I know that the problem isn't with one
    > giant array, but the total size of the arrays I'm declaring. I assume
    > this all has to do with some sort of memory limit and a lot of spotty
    > advice posted over the internet in similar cases have suggested it has
    > something to do with a 64K stack size, but this doesn't make much
    > sense to me.


    On a typical system, a C program has a fixed-size region of memory set
    aside for a stack. This is a very simple memory allocation scheme;
    there is a single pointer (normally in a CPU register) marking the
    "top". When a function gets called, it grabs some space
    from the top of the stack, moving the pointer to indicate the new top,
    and uses this space to store its local variables. When it returns, it
    restores the pointer to its original value, thus allowing the space to
    be reused later.

    In your code, since the arrays you declare are local to the function
    main(), they are allocated in this manner. But it appears that the
    total size of your arrays is bigger than the total size of the stack
    (typically a few megabytes). So trying to access those arrays runs off
    the end of the stack and causes a crash.

    The simplest way to fix it, in your case, would be to declare those
    arrays with the `static' keyword. This causes them to be placed in a
    different area of memory, which doesn't need to be reused like the
    stack. It also means that multiple calls to the function will all
    access the same copy of the arrays, rather than each one getting its own
    copy, but that is irrelevant here since I presume you don't intend to
    have main() recursively call itself. There is also the possibly
    convenient side effect that it will cause all the arrays to be
    initialized with zeros.

    On Unix, you can query and set the default stack size in kbytes with
    "ulimit -s" (from a Bourne shell) or "limit stacksize" (from a C shell)
    or the rlimit() function (from within the program), so that would be
    another possibility. A third option would be to allocate those arrays
    dynamically using malloc(), but that is a little awkward (though
    certainly possible) to do with multidimensional arrays.

    > Can anyone shine some light on what exactly is going wrong and how I
    > can avoid it? By my calculation the size of these declared variables
    > is a little over 9MB which isn't a huge amount, and the FORTRAN
    > version of this code works for MAXN in excess of 100000.


    I make it more like 36 MB. The C `double' type occupies 8 bytes on
    x86/amd64.

    > If it matters, I am compiling on linux amd64, and this problem has
    > happened with both GCC 4.1 and Sun's C compiler (SS12).


    FWIW, on the Linux amd64 box I have access to, the stack is 10 MB by
    default. Yours could be configured differently, of course.

    > Thanks much.
    >
    > glenn
    >
    > -- Begin code --
    >
    > #define MAXN 60000
    > #define MAXLT 10
    >
    > int main()


    I'll just mention that modern C standards call for you to use "void" as
    the argument list of a function that doesn't take any arguments. So
    this should read

    int main(void)

    (Technically speaking, there seems to be some dispute as to whether
    "int main()" is actually illegal or merely obsolescent.)
     
    Nate Eldredge, Mar 13, 2009
    #5
  6. glennklockwood

    CBFalconer Guest

    glennklockwood wrote:
    >
    > I'm in the process of porting some FORTRAN code to C and am running
    > into a problem where the program crashes if I set the size of arrays
    > too large. In the below code, my program runs fine if I set MAXN to
    > 50000 or less, but at 60000 or more, the program immediately crashes
    > and GDB indicates the error is at the printf line.


    Automatic storage is generally assigned on a stack (if the system
    uses a stack). Stack size is normally limited. I suggest using
    pointers and mallocing the associated storage.

    --
    [mail]: Chuck F (cbfalconer at maineline dot net)
    [page]: <http://cbfalconer.home.att.net>
    Try the download section.
     
    CBFalconer, Mar 14, 2009
    #6
  7. On 14 Mar 2009 at 3:23, Richard Harter wrote:
    ><> wrote:
    >>I suggest using pointers and mallocing the associated storage.

    >
    > And, of course, one should check out hashlib and nmalloc.


    *snigger*

    You forgot ggets(), the swiss army knife of C programming that solves
    every problem!
     
    Antoninus Twink, Mar 14, 2009
    #7
  8. glennklockwood

    CBFalconer Guest

    Malcolm McLean wrote:
    > "Eric Sosman" <> wrote:
    >
    >> Fortran variables have what C would call "static" storage (or at
    >> any rate they did years ago when I used FORTRAN). The arrays in
    >> your C program use "automatic" storage, which is rather different.
    >> On many systems, the amount of space available for "automatic"
    >> variables is considerably less than is provided for other classes
    >> of data.

    >
    > Fortran 77 is still alive and kicking. I have to use it myself,
    > sometimes.
    >
    > All arguments are passed by reference, and all variables are static.
    > So code can be very optimal - no hard to detect pointer aliasing
    > problems.


    And no routines are re-entrant, nor is recurrance allowed or
    feasible. It is impossible to isolate variables for use in a
    single function.

    --
    [mail]: Chuck F (cbfalconer at maineline dot net)
    [page]: <http://cbfalconer.home.att.net>
    Try the download section.
     
    CBFalconer, Mar 15, 2009
    #8
  9. glennklockwood

    CBFalconer Guest

    Richard Harter wrote:
    > CBFalconer <> wrote:
    >> glennklockwood wrote:
    >>
    >>> I'm in the process of porting some FORTRAN code to C and am
    >>> running into a problem where the program crashes if I set the
    >>> size of arrays too large. In the below code, my program runs
    >>> fine if I set MAXN to 50000 or less, but at 60000 or more, the
    >>> program immediately crashes and GDB indicates the error is at
    >>> the printf line.

    >>
    >> Automatic storage is generally assigned on a stack (if the
    >> system uses a stack). Stack size is normally limited. I
    >> suggest using pointers and mallocing the associated storage.

    >
    > And, of course, one should check out hashlib and nmalloc.


    Why? I failed to see any application for them in the OPs request.
    Please teach us your magnificent method of detecting requirements
    in verbiage such as the above.

    --
    [mail]: Chuck F (cbfalconer at maineline dot net)
    [page]: <http://cbfalconer.home.att.net>
    Try the download section.
     
    CBFalconer, Mar 15, 2009
    #9
  10. glennklockwood

    CBFalconer Guest

    Richard Harter wrote:
    > CBFalconer <> wrote:
    >> Richard Harter wrote:
    >>

    .... snip ...
    >>
    >>> And, of course, one should check out hashlib and nmalloc.

    >>
    >> Why? I failed to see any application for them in the OPs request.
    >> Please teach us your magnificent method of detecting requirements
    >> in verbiage such as the above.

    >
    > Surely you must be able to find something. You have so often been
    > able to find reason for recommending them in the most unpromising
    > of circumstances.


    I am not responsible for your inability to detect suitable
    applications. I, on the other hand, having designed those modules,
    am aware of their abilities and can notice useful applications.

    --
    [mail]: Chuck F (cbfalconer at maineline dot net)
    [page]: <http://cbfalconer.home.att.net>
    Try the download section.
     
    CBFalconer, Mar 16, 2009
    #10
  11. glennklockwood

    Guest

    On 15 Mar, 00:11, CBFalconer <> wrote:
    > Richard Harter wrote:
    > > CBFalconer <> wrote:


    <snip>

    <snip some memory allocation stuff>

    > > And, of course, one should check out hashlib and nmalloc.

    >
    > Why?  I failed to see any application for them in the OPs request.
    > Please teach us your magnificent method of detecting requirements
    > in verbiage such as the above.


    they are pulling your leg. These are not serious posts.
     
    , Mar 16, 2009
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Robert Strickland

    File Upload and File Size Limitations

    Robert Strickland, Oct 29, 2004, in forum: ASP .Net
    Replies:
    6
    Views:
    4,668
    Lau Lei Cheong
    Nov 1, 2004
  2. Neil Zanella
    Replies:
    4
    Views:
    487
    Gianni Mariani
    Oct 9, 2003
  3. GroupReader

    Are there any page size limitations?

    GroupReader, Jul 3, 2007, in forum: ASP .Net
    Replies:
    5
    Views:
    776
    GroupReader
    Jul 3, 2007
  4. Paolo Zaffino
    Replies:
    7
    Views:
    7,372
    Prasad, Ramit
    Oct 19, 2011
  5. Francois Grieu
    Replies:
    44
    Views:
    1,060
Loading...

Share This Page