malloc of large string works, but same sized array fails

[mathog]

This is an odd one. Normally it doesn't matter if a string is allocated
at run time or if it is set up as an array. In this case it does
though. This causes a segfault at run time:

#define MYMAXSTRING 20000000

int main(int argc, char *argv[]){
char bigstring[MYMAXSTRING];

whereas this works with no problems

#define MYMAXSTRING 20000000

int main(int argc, char *argv[]){
char *bigstring=NULL;


bigstring=malloc(MYMAXSTRING);
if(!bigstring){
(void) fprintf(stderr,"Could not allocate initial memory\n");
exit(EXIT_FAILURE);
}


When run in gdb it shows the segfault on the main() line. There is
plenty of free memory on the system.

It was compiled like:

gcc -Wall -ansi -g -pedantic -o prog prog.c

with gcc 4.2.3 (yes, that is pretty old).

Why the difference?

Thanks,

David Mathog

 

[Kaz Kylheku]

This is an odd one. Normally it doesn't matter if a string is allocated
at run time or if it is set up as an array.

All allocation happens at "run time".

Even static arrays are allocated at run-time, at least on on a system where
programs are not singly instantiated on some EEPROM chip, but loaded into
memory.

Something has to find space for the program and its static data; that happens
at run time, though it is before your main function is called.

In this case it does though. This causes a segfault at run time:

#define MYMAXSTRING 20000000

int main(int argc, char *argv[]){
char bigstring[MYMAXSTRING];

This array is in autoamtic storage. Automatic storage is allocated at run time;
often from a region of space called a "stack".

Implementations often impose more stringent limits on automatic storage (local
variables) than they do on dynamic storage (malloc).

It was compiled like:

gcc -Wall -ansi -g -pedantic -o prog prog.c

Are you on some kind of Unix? What is your system's limit on stack size?

 

[mathog]

Are you on some kind of Unix? What is your system's limit on stack size?

That seems to be the issue:

ulimit -a | grep stack

stack size (kbytes, -s) 8192

Thanks,

David Mathog

 

[glen herrmannsfeldt]

All allocation happens at "run time".

Even static arrays are allocated at run-time, at least on on
a system where programs are not singly instantiated on some
EEPROM chip, but loaded into memory.

On many systems, I would say that static data is allocated
at link time. It is often given its run-time address (possibly
with a load time offset) at link time.

Something has to find space for the program and its static data;
that happens at run time, though it is before your main
function is called.

OK, "find space for", but the address is already assigned within
that space. Note that it also has to "find space for" the stack,
but you can still "allocate" on the stack.

In this case it does though. This causes a segfault at run time:
#define MYMAXSTRING 20000000
int main(int argc, char *argv[]){
char bigstring[MYMAXSTRING];

This array is in autoamtic storage. Automatic storage is allocated
at run time; often from a region of space called a "stack".

Implementations often impose more stringent limits on automatic
storage (local variables) than they do on dynamic storage (malloc).

This problem occurs more often in Fortran programs, which more
naturally allocate on the stack.

Yesterday, in a different thread on a different news group, I was
remembering an Alpha/OSF1 system that restricted static data
to 64K, even though stack and malloc() could allocate GB.
(Alpha is a 64 bit processor, but somehow uses 16 bits for static.)

Are you on some kind of Unix? What is your system's limit
on stack size?

And, within the system limit, the shell can have its own limit.


-- glen

 

[BartC]

That seems to be the issue:

ulimit -a | grep stack

stack size (kbytes, -s) 8192

Is that a hard upper limit? Often the stack size is the initial size, which
grows as needed. 8KB would be a ludicrous maximum stack size on a machine
with presumably several GB of RAM. On my Windows machine, that 20MB
allocation is no problem.

Try ulimit -s unlimited

 

[Keith Thompson]

Is that a hard upper limit? Often the stack size is the initial size, which
grows as needed. 8KB would be a ludicrous maximum stack size on a machine
with presumably several GB of RAM. On my Windows machine, that 20MB
allocation is no problem.

The "kbytes" indicates the units in which the number 8192 is expressed.
The limit is 8 MB, not 8 KB.

Try ulimit -s unlimited

The problem with that is that anyone running your program will have to
do the same thing to make it work. It's probably better to modify the
program so it works with the default settings.

 

[glen herrmannsfeldt]

(snip)

I remember the MS-DOS and OS/2 linker from MS which had a default
something like 8K or so for way too long. At least with OS/2 where
the stack was virtual.

The "kbytes" indicates the units in which the number 8192 is expressed.
The limit is 8 MB, not 8 KB.

The problem with that is that anyone running your program will have to
do the same thing to make it work. It's probably better to modify the
program so it works with the default settings.

I suppose, but at some point it is a mistake.

Historically, C programs tended to allocate large regions with
malloc() and smaller ones with auto variables. One reason for that
is that in K&R C and C89, auto arrays dimensions are compile time
constants, and large ones tend to be run time variables.
(The OP didn't explain where the large number came from.)

Since C programs didn't tend to allocate large variables on
the stack, the defaults were set small. The need for larger stack
was rare enough not to bother most people.

Fortran programs have tended to do more allocation on the stack,
including larger arrays. Compilers will even generate large
temporary arrays on the stack, sometimes surprising users.

But at some point, it is a chicken and egg problem. People follow
your suggestion and don't allocate large regions on the stack,
shell writers use that as a reason to set a small default.

-- glen

 

[Ike Naar]

char *bigstring=NULL;

bigstring=malloc(MYMAXSTRING);

Just wondering, why not simply

char *bigstring = malloc(MYMAXSTRING);

?

 

[Malcolm McLean]

Just wondering, why not simply
char *bigstring = malloc(MYMAXSTRING);
?

or better still
char *bigstring = malloc(MYMAXSTRING + 1);

 

[Keith Thompson]

or better still
char *bigstring = malloc(MYMAXSTRING + 1);

Is MYMAXSTRING the maximum *size* of the string or the maximum *length*
of the string? How do you know?