restrictions with malloc???

[=?ISO-8859-1?Q?Martin_J=F8rgensen?=]

Hi,

I have a (bigger) program with about 15-30 malloc's in it (too big to
post it here)... The last thing I tried today was to add yet another
malloc **two_dimensional_data. But I found out that malloc always
returned null at this moment and the program exited (even though if I
malloc'ed only 20 bytes or something)... Then I googled for this problem
and found something about a memory pool??? Is that standard C?

I didn't understand it, but it said something about that it was possible
to choose the size of the memory pool to be like 4 kb (or whatever)...?
I copy/pasted the code, but it didn't work...

Then I came to think of this "heap"-thing which I never really
understood. Is there any size limit on this "heap"? And can I have more
than 20-30 malloc's in my program without any trouble?

My solution was to allocate this two_dimensional_data-array statically
(declared it as "double two_dimensional_data[15][15]"), but I prefer the
dynamically method if possible...

And suppose at some moment I need to allocate 200 MB RAM. AFAIR this
isn't possible even though the system has that amount of memory free?
What could one do in such a case?

Therefore the question: Is there any restrictions with using malloc that
is relevant to know of?


Best regards / Med venlig hilsen
Martin Jørgensen

 

[Keith Thompson]

I have a (bigger) program with about 15-30 malloc's in it (too big to
post it here)... The last thing I tried today was to add yet another
malloc **two_dimensional_data. But I found out that malloc always
returned null at this moment and the program exited (even though if I
malloc'ed only 20 bytes or something)... Then I googled for this
problem and found something about a memory pool??? Is that standard C?

I didn't understand it, but it said something about that it was
possible to choose the size of the memory pool to be like 4 kb (or
whatever)...? I copy/pasted the code, but it didn't work...

Then I came to think of this "heap"-thing which I never really
understood. Is there any size limit on this "heap"? And can I have
more than 20-30 malloc's in my program without any trouble?

When you say "20-30 malloc's", do you mean 20-30 malloc() calls in
your program, or do you mean that malloc() is called 20-30 times as
your program is executing?

Either way, the number of malloc calls is unlikely to be relevant.

A call to malloc() attempts to allocate memory for your program to
use. The manner in which it does this is implementation-specific, but
typically there's some big chunk of (real or virtual) memory
somewhere, possibly expandable as necessary, and malloc() gives you
sub-chunks of that chunk. This big chunk is commonly referred to as
the "heap", but knowing the name for it doesn't really do you much
good. The point is that it's finite, and you can eventually exhaust
it. The actual size is not defined by the language; there may not
even be a way to find out.

What's relevant is the total amount of memory you've allocated.

Your system may provide some way to control memory limits (see "limit"
or "ulimit" if you're on a Unix-like system).

If you add to each malloc() call a printf statement that logs how much
memory you're allocating, it may be helpful in tracking down the
problem -- assuming you're actually running out of memory. Or
something like an interactive debugger might provide some facility for
tracking this information. Or you might be able to use something
called a "memory profiler". (None of these tools are specified by the
language, so the details are off-topic here.)

It's also possible that malloc() is failing because you've done
something that invokes undefined behavior. For example, if you write
outside the bounds of a chunk of allocated memory, you might corrupt
the internal data structures that malloc() uses to keep track of
what's been allocated. This kind of problem can be very difficult to
track down.

The comp.lang.c FAQ, <http://www.c-faq.com/>. has a section on memory
allocation that you might find useful.

 

[=?ISO-8859-1?Q?Martin_J=F8rgensen?=]

When you say "20-30 malloc's", do you mean 20-30 malloc() calls in
your program, or do you mean that malloc() is called 20-30 times as
your program is executing?

Isn't that the same thing?

I mean... I have a lot of variables defined as **two_D_arrays:

main()
{
two_D_array1 = malloc( number of bytes);
printf("bla.bla");
// load from files
two_D_array2 = malloc( number of bytes);
two_D_array3 = malloc( number of bytes);

two_D_array1[0] = malloc( (number of bytes)*(number of bytes) );
two_D_array1[1] = malloc( (number of bytes)*(number of bytes) );
etc...

for(i=1; i<number; i++)
{
two_D_array1 = two_D_array1[i-1] + number_of_cols;
two_D_array2 = two_D_array2[i-1] + number_of_cols;
// and a lot more...
}

// bla.bla. input from file/output to screen
// more mallocs here...
}

Malloc is probably called about 20-30 times, so there are about 20-30
lines with the word malloc() in it...

Either way, the number of malloc calls is unlikely to be relevant.

I thought so....

A call to malloc() attempts to allocate memory for your program to
use. The manner in which it does this is implementation-specific, but
typically there's some big chunk of (real or virtual) memory
somewhere, possibly expandable as necessary, and malloc() gives you
sub-chunks of that chunk. This big chunk is commonly referred to as
the "heap", but knowing the name for it doesn't really do you much
good. The point is that it's finite, and you can eventually exhaust
it. The actual size is not defined by the language; there may not
even be a way to find out.

But I only allocated about 7-8 kilo-bytes.... I must have done something
wrong somewhere.... Perhaps there's an index wrong somewhere and I might
have allocated too little memory for a variable that overwrites
something, don't you think?

AFAIR I also debugged a for-loop like:

for(i=0; i<number; i++) // and suddenly "i" changed from 2 to 6...

What's relevant is the total amount of memory you've allocated.

About 8 kilo-bytes is no problem, is it?

Your system may provide some way to control memory limits (see "limit"
or "ulimit" if you're on a Unix-like system).

Hmmm.. I'm on windows xp using visual studio 2005. How do I see if there
is any limit?

If you add to each malloc() call a printf statement that logs how much
memory you're allocating, it may be helpful in tracking down the
problem -- assuming you're actually running out of memory. Or
something like an interactive debugger might provide some facility for
tracking this information. Or you might be able to use something
called a "memory profiler". (None of these tools are specified by the
language, so the details are off-topic here.)

What an interactive debugger? Do I have an interactive debugger with
visual studio 2005? Since I can make breakpoints and step into the
program, is that what you mean by "interactive"?

I've also read a bit about those profiler tools (valgrant etc) but never
tried them... I guess I'll start with printf("...")-lines again and I
hope that's sufficient...

It's also possible that malloc() is failing because you've done
something that invokes undefined behavior. For example, if you write
outside the bounds of a chunk of allocated memory, you might corrupt
the internal data structures that malloc() uses to keep track of
what's been allocated. This kind of problem can be very difficult to
track down.

I think that's the real problem...

The comp.lang.c FAQ, <http://www.c-faq.com/>. has a section on memory
allocation that you might find useful.

So, there's no problem in doing like hundreds of malloc()-calls if
they're only requesting perhaps 50 bytes of memory each time, is there?

Also I never figured out about this "memory pool-thing"... Does anyone
know anything about this:

http://www.die.net/doc/linux/man/man3/mpool.3.html

That's not standard C is it or??? Is there something like that on
windows-machines that I need to know of?

It says: "The function mpool_open initializes a memory pool."...


Best regards / Med venlig hilsen
Martin Jørgensen

 

[Bill Pursell]

I mean... I have a lot of variables defined as **two_D_arrays:

main()
{
two_D_array1 = malloc( number of bytes);
printf("bla.bla");
// load from files
two_D_array2 = malloc( number of bytes);
two_D_array3 = malloc( number of bytes);

two_D_array1[0] = malloc( (number of bytes)*(number of bytes) );
two_D_array1[1] = malloc( (number of bytes)*(number of bytes) );
etc...

This is almost certainly not what you want. If you want a rowsXcols
array of ints you could do:

int **two_D_array;
two_D_array = malloc(rows * sizeof(int *));
for (int i=0; i< 15; i++)
two_D_array = malloc(cols * sizeof(int));
And be able to reference values as two_D_array[x][y],

Or you can do
int *array = malloc (rowsXcols * sizeof(int));
and lose the ability to reference easily with array[x][y] notation.
(you need to do array[rows*i + j]

 

[Pedro Graca]

I guess I'll start with printf("...")-lines again and I
hope that's sufficient...

Try this at the top of your source file, after the necessary #include's:

void * dbg_malloc(size_t siz, int pos) {
void * tmp = malloc(siz);
fprintf(stderr, "DEBUG: Allocated %lu bytes at line %d.\n",
(unsigned long)siz, pos);
return tmp;
}
/* all malloc's following this line will be replaced with dbg_malloc */
#define malloc(x) dbg_malloc(x, __LINE__)

And you may want to

#undef malloc
/* no more replacing of malloc's */

at the end of the file for completeness

 

[pemo]

Hmmm.. I'm on windows xp using visual studio 2005.

I believe that VS is capable of ascertaining whether or not you've corrupted
memory allocated using malloc etc - try performing a 'Debug' build.

What an interactive debugger? Do I have an interactive debugger with
visual studio 2005? Since I can make breakpoints and step into the
program, is that what you mean by "interactive"?

Yes, if you're able to set breakpoints etc you're using an /interactive
debugger/.

I've also read a bit about those profiler tools (valgrant etc) but
never tried them... I guess I'll start with printf("...")-lines again
and I hope that's sufficient...


I think that's the real problem...

Try as I might, I can't get Visual Stdio to issue what I believe is its
'damage' message - but perhaps I've not got the right compiler switches
turned on ... however, if you can track down how to activate this VS
feature, you may well find it useful --- and, if you do discover the
/howto/, please let me know how to do it!

 

[Pedro Graca]

void * dbg_malloc(size_t siz, int pos) {
void * tmp = malloc(siz);
fprintf(stderr, "DEBUG: Allocated %lu bytes at line %d.\n",
(unsigned long)siz, pos);
return tmp;
}

Or even ...

void * dbg_malloc(size_t siz, int pos) {
fprintf(stderr, "DEBUG: Trying to allocate %lu bytes at line %d.\n",
(unsigned long)siz, pos);
return malloc(siz);
}

 

[Keith Thompson]

Keith Thompson wrote: [...]

When you say "20-30 malloc's", do you mean 20-30 malloc() calls in
your program, or do you mean that malloc() is called 20-30 times as
your program is executing?

Isn't that the same thing?

No. For example:

for (i = 0; i < 100; i ++) {
a = malloc(1024);
b = malloc(2048);
}

There are 2 malloc() calls in the program, but malloc() is called 200
times when the program executes.

[...]

But I only allocated about 7-8 kilo-bytes

Are you sure about that? You know there's a bug somewhere; are you
sure it doesn't involve allocating more than you intended?

[...]

Hmmm.. I'm on windows xp using visual studio 2005. How do I see if
there is any limit?

I have no idea. Check your system's documentation; failing that, try
a newsgroup where it's topical. (We discuss the language here, not
the details of any particular implementation.) But if you're really
not allocating more than about 8 kbytes, that's probably not the
problem.

What an interactive debugger? Do I have an interactive debugger with
visual studio 2005? Since I can make breakpoints and step into the
program, is that what you mean by "interactive"?
Yes.

[...]

Also I never figured out about this "memory pool-thing"... Does anyone
know anything about this:

http://www.die.net/doc/linux/man/man3/mpool.3.html

That's not standard C is it or??? Is there something like that on
windows-machines that I need to know of?

No, it's not standard C.

 

[=?ISO-8859-1?Q?Martin_J=F8rgensen?=]

I mean... I have a lot of variables defined as **two_D_arrays:

main()
{
two_D_array1 = malloc( number of bytes);
printf("bla.bla");
// load from files
two_D_array2 = malloc( number of bytes);
two_D_array3 = malloc( number of bytes);

two_D_array1[0] = malloc( (number of bytes)*(number of bytes) );
two_D_array1[1] = malloc( (number of bytes)*(number of bytes) );
etc...

This is almost certainly not what you want. If you want a rowsXcols
array of ints you could do:

int **two_D_array;
two_D_array = malloc(rows * sizeof(int *));
for (int i=0; i< 15; i++)
two_D_array = malloc(cols * sizeof(int));
And be able to reference values as two_D_array[x][y],

Okay my mistake... It was just pseudo-code because I was in a hurry...
In this case I have the same number of rows as columns so I guess this
is the same thing as yours code except I allocate everything in the
2D-array at once, while your code allocates one row every time:


for(i=1; i<number; i++)
{
two_D_array1 = two_D_array1[i-1] + number_of_cols;
two_D_array2 = two_D_array2[i-1] + number_of_cols;
}

Or you can do
int *array = malloc (rowsXcols * sizeof(int));
and lose the ability to reference easily with array[x][y] notation.
(you need to do array[rows*i + j]

Yep, I know... But it easy with array[row][column]-notation.


Best regards / Med venlig hilsen
Martin Jørgensen

 

[=?ISO-8859-1?Q?Martin_J=F8rgensen?=]

Try this at the top of your source file, after the necessary #include's:

void * dbg_malloc(size_t siz, int pos) {
void * tmp = malloc(siz);
fprintf(stderr, "DEBUG: Allocated %lu bytes at line %d.\n",
(unsigned long)siz, pos);
return tmp;
}
/* all malloc's following this line will be replaced with dbg_malloc */
#define malloc(x) dbg_malloc(x, __LINE__)

And you may want to

#undef malloc
/* no more replacing of malloc's */

at the end of the file for completeness

Thanks good idea... I already know which line is the problem, but this
method is better. And I don't see any problem with the line, so it's
really strange. It's probably some corrupt memory somewhere...


Best regards / Med venlig hilsen
Martin Jørgensen

 

[Keith Thompson]

Thanks good idea... I already know which line is the problem, but this
method is better. And I don't see any problem with the line, so it's
really strange. It's probably some corrupt memory somewhere...

Something that's fairly likely to corrupt the memory managed by
malloc() is an incorrect call to malloc() or free(). For example, if
you call free() with a pointer that wasn't returned by malloc(), or
free() the same pointer twice, things can go wrong.

If you replace each call to malloc() and each call to free() (and
calloc() and realloc() as well) with a call to a wrapper that logs the
arguments and result, analysis of the log might lead you to the
problem.

For example, expanding on Martin's code:

void *dbg_malloc(size_t siz, int pos)
{
void *tmp = malloc(siz);
fprintf(stderr, "DEBUG: %d: malloc(%ld) --> %p\n",
pos, (unsigned long)siz, tmp);
return tmp;
}

void *dbg_free(void *ptr, int pos)
{
fprintf(stderr, "DEBUG: %d: free(%p)\n", pos, ptr);
free(ptr);
}

If the malloc()s and free() don't match up, there's your problem.

Another approach is to temporarily comment out any calls to free(),
and see if the problem goes away.

 

[Pedro Graca]

Try this at the top of your source file, after the necessary
#include's: [snip dbg_malloc() definition]
#define malloc(x) dbg_malloc(x, __LINE__)

Thanks good idea...

Something that's fairly likely to corrupt the memory managed by
malloc() is an incorrect call to malloc() or free(). For example, if
you call free() with a pointer that wasn't returned by malloc(), or
free() the same pointer twice, things can go wrong.

If you replace each call to malloc() and each call to free() (and
calloc() and realloc() as well) with a call to a wrapper that logs the
arguments and result, analysis of the log might lead you to the
problem.

For example, expanding on Martin's code:

That was my code

Some time ago I wrote wrappers for malloc() and friends.
They're available at
<http://hexkid.blogspot.com/2006/02/memory-management-revisited.html>,
look for dynmem.h and dynmem.c at the bottom of the post.

You will probably need some tweaking to use the "#define malloc" hack or
rename all pmg_* functions to dbg_* (much better naming).

 

[Keith Thompson]

That was my code

[snip]

Whoops, sorry about that (I miscounted the '>'s).

 

[=?ISO-8859-1?Q?Martin_J=F8rgensen?=]

Another approach is to temporarily comment out any calls to free(),
and see if the problem goes away.

Hi, I've been too lazy to actually do something about the problem since
I figured out from everybody's comments that there isn't any "easy way"
and there shouldn't be any problems with about 30 mallocs that totally
occupies (only) about 8 kb...

So today I guess I'll temporarily comment out a lot of the code and
hopefully I can track down the problem after some hours........ Sigh....


And actually I don't use free() anywhere because after I allocate these
arrays I need them for the rest of my program (well perhaps except for
the last 2-3 lines).


Best regards / Med venlig hilsen
Martin Jørgensen

 

[=?ISO-8859-1?Q?Martin_J=F8rgensen?=]

Martin Jørgensen wrote: -snip-

Try as I might, I can't get Visual Stdio to issue what I believe is its
'damage' message - but perhaps I've not got the right compiler switches
turned on ... however, if you can track down how to activate this VS
feature, you may well find it useful --- and, if you do discover the
/howto/, please let me know how to do it!

Sure... I asked this question in another newsgroup...


Best regards / Med venlig hilsen
Martin Jørgensen

 

[=?ISO-8859-1?Q?Martin_J=F8rgensen?=]

Martin Jørgensen wrote: -snip-

Try as I might, I can't get Visual Stdio to issue what I believe is its
'damage' message - but perhaps I've not got the right compiler switches
turned on ... however, if you can track down how to activate this VS
feature, you may well find it useful --- and, if you do discover the
/howto/, please let me know how to do it!

WOW! Thanks *so* much for leading me on the right track... You can look
here:

http://msdn2.microsoft.com/en-us/library/e73x0s4b(VS.80).aspx

And I found the error using that function, so I'll definately use that
from now on.... I just placed a lot of _ASSERTE(_CrtCheckMemory() )
calls inside my code untill I found out where the program stopped and
*VOILA*!

I'm not sure I would ever have found this error if it wasn't for this
function

I simply overwrote something by using a wrong index counter that
determines when to stop writing data, in my for-loop....


Best regards / Med venlig hilsen
Martin Jørgensen