how to figure out the size of buffer from malloc

[pembed2012]

I am wondering how does 'free' know how many bytes to free? For example:

char* s = (char*)malloc(10);
free(s); // how many bytes to free?

How does free know how many bytes to free? I have heard that malloc
also allocate additional memory to hold some header information. Where
is this header information stored? If we have access to the header, we
can find that out?

Thanks!

 

[James Kuyper]

I am wondering how does 'free' know how many bytes to free? For example:

char* s = (char*)malloc(10);
free(s); // how many bytes to free?

How does free know how many bytes to free? I have heard that malloc
also allocate additional memory to hold some header information. Where
is this header information stored? If we have access to the header, we
can find that out?

Howe that information is stored is an implementation detail, likely to
be different on different implementations. As a result, there's no
portable way you can obtain that information.

One common technique is to have malloc() allocate a little bit more than
the requested amount of memory, and to store the size of the allocation
in that extra piece of memory. This extra memory is usually stored at a
fixed negative offset from the pointer that you get from malloc().

Another technique is round each allocation request up to a power of 2,
and to maintain a separate pool of memory for each power of 2. By
looking at the address, free() can tell which pool it came from, and
therefore, how much memory to free.

There's several other, more sophisticated techniques that are in actual
use, but I hope that those two are sufficient for you to understand why
there's no portable way to get this information.

 

[John Gordon]

I am wondering how does 'free' know how many bytes to free? For example:

char* s = (char*)malloc(10);
free(s); // how many bytes to free?

A common method is for malloc to allocate a few extra bytes, store the
size in the extra space at the beginning, and return a pointer to the
remaining area.

In your sample code above, malloc might actually allocate fourteen bytes,
store the size information in the first four bytes and return a pointer
to the remaining ten bytes.

Then when free is called, it knows it can back up four bytes from the
given address to find the size of the area to be freed.

How does free know how many bytes to free? I have heard that malloc
also allocate additional memory to hold some header information. Where
is this header information stored? If we have access to the header, we
can find that out?

It's all compiler- and library-specific. The language itself doesn't
specify how it is done.

 

[Lew Pitcher]

I am wondering how does 'free' know how many bytes to free?

As others have said, malloc() and free() are part of the language
definition, and /how/ they work is considered to be "magic" as far as we
mortal userspace programmers go.

OTOH, you can get a feel for what sort of code /might/ go into malloc() and
free() by checking out K&R "The C Programming Language". The original K&R
(and, I believe, the K&R rewritten for C89) has a chapter called "The Unix
System Interface", in which an example "Storage Allocator" is detailed.

HTH

 

[Keith Thompson]

I am wondering how does 'free' know how many bytes to free? For example:

char* s = (char*)malloc(10);

The cast is unnecessary and can hide errors.

char *s = malloc(10);

Note that this can fail; you should always check whether malloc()
returned a null pointer.

free(s); // how many bytes to free?

How does free know how many bytes to free? I have heard that malloc
also allocate additional memory to hold some header information. Where
is this header information stored? If we have access to the header, we
can find that out?

The comp.lang.c FAQ is at <http://c-faq.com/>. You've just asked
question 7.26.

 

[pembed2012]

A common method is for malloc to allocate a few extra bytes, store the
size in the extra space at the beginning, and return a pointer to the
remaining area.

In your sample code above, malloc might actually allocate fourteen
bytes, store the size information in the first four bytes and return a
pointer to the remaining ten bytes.

Then when free is called, it knows it can back up four bytes from the
given address to find the size of the area to be freed.


It's all compiler- and library-specific. The language itself doesn't
specify how it is done.

Thanks, I've tried this but it does NOT work

This code prints: 0, it should print: 10

int get_malloc_size(void *vp)
{
char *cp = (char*)vp;
static int __done = -1;
static int __offset = -1;
if(__done != 0) {
char *p;
int n,i;
while(1) {
n=random()%10000;
p=(char*)malloc(n);
for(i=1;i<100;i++) {
if(*(int*)(p-i)==n) {
__offset=i;
break;
}
}
free(p);
if(__offset != -1)
break;
}
__done=0;
}
return *(int*)(cp-__offset);
}

main()
{
char *p=(char*)malloc(10);
printf("%d\n", get_malloc_size((void*)p));
free(p);
}

 

[John Gordon]

Thanks, I've tried this but it does NOT work

I suggested one possible way that malloc might store the size of the
allocated area. There are lots of different ways this can be done. Your
malloc implementation must use a different mechanism.

I thought I was clear when I said:

If you want to know how it's done on your system, you'll have to look
at the system documentation.

 

[Keith Thompson]

In <[email protected]> pembed2012 <[email protected]>
writes: [...]

How does free know how many bytes to free? I have heard that malloc
also allocate additional memory to hold some header information. Where
is this header information stored? If we have access to the header, we
can find that out?

It's all compiler- and library-specific. The language itself doesn't
specify how it is done.

Thanks, I've tried this but it does NOT work

This code prints: 0, it should print: 10

int get_malloc_size(void *vp)
{
char *cp = (char*)vp;
static int __done = -1;
static int __offset = -1;

Identifiers starting with two underscores are reserved to the
implementation. You should *never* define such identifiers in
your own code. The rules are a little more complex for identifiers
starting with a single underscore but you should still avoid them.

[snip]

return *(int*)(cp-__offset);
}

main()

Should be "int main(void)".

{
char *p=(char*)malloc(10);
printf("%d\n", get_malloc_size((void*)p));
free(p);
}

Why would you expect it to work? There is *no* portable way to
determine the size of a malloc()ed chunk of memory.

Actually, there is: just remember the argument you passed to malloc():

size_t requested_size = 10;
char *p = malloc(requested_size);
if (p != NULL) {
/* You know you've allocated requested_size bytes. */
}

Note that the implementation could allocate more memory than you
request; it won't necessarily store the requested size anywhere.
But that shouldn't matter to your program.

If you think it does matter, explain what you're trying to accomplish
and we can probably help you find another way.

 

[Shao Miller]

Thanks, I've tried this but it does NOT work

You can make your own allocator and freer and use them instead of
'malloc' and 'free'. In fact, you could even do:

#include <standard_stuff1.h>
#include <standard_stuff2.h>

#include <system_stuff.h>

#include "my_stuff.h"

#define malloc my_malloc
#define free my_free

Then you could use 'malloc' and 'free' but the result would be calls to
your functions instead of the implementation's.

Here's an example alternative to using 'malloc' that I haven't looked at
for a while:

/**
* Delete all lines containing
* 'ALL_IN_ONE' if you split this example
* into separate files.
*/
#define ALL_IN_ONE 1


/****
* pfxalloc.h
*
* It took several minutes to find
* a free *alloc. (No pun intended.)
*/
#ifndef PFXALLOC_H_

/* For 'offsetof' and 'size_t' */
#include <stddef.h>

/*** Macros */
#define PFXALLOC_H_

#ifndef alignof

/* Derived from Mr. Chris M. Thomasson */
#define alignof(type) \
(offsetof(struct {char c; type t;}, t))

#ifdef __alignof_is_defined
#undef __alignof_is_defined
#endif /* __alignof_is_defined */

#define __alignof_is_defined 1
#endif /* alignof */

/*** Functions */

/**
* Allocate memory for an object and an
* associated "prefix" object.
*
* @v pfx_size The "prefix" size.
* @v obj_alignment The object's alignment
* requirement.
* @v obj_size The object's size.
* @ret void * Points to the object,
* or is NULL for failure.
*/
extern void * pfxalloc(size_t, size_t, size_t);

/**
* Get an object's associated "prefix" object.
*
* @v obj The object to fetch the
* associated "prefix" for.
* @ret void * Points to the "prefix",
* or is NULL for failure.
*/
extern void * pfxget(void *);

#endif /* PFXALLOC_H_ */


/**** pfxalloc.c */

/* For 'malloc' */
#include <stdlib.h>
#if !ALL_IN_ONE
/* For 'size_t', 'ptrdiff_t' and 'NULL' */
#include <stddef.h>
/* For 'alignof' */
#include "pfxalloc.h"
#endif /* !ALL_IN_ONE */

/*** Constants */
enum cv {
cv_ptrdiff_alignment =
alignof(ptrdiff_t),
cv_zero = 0
};

/*** Functions */

/**
* The layout looks like:
* +-----+-------------+--------+--------+
* | pfx | opt_padding | offset | |
* +- -+- -+- -+ object |
* | header | |
* +----------------------------+--------+
*/
void * pfxalloc(
size_t pfx_size,
size_t obj_alignment,
size_t obj_size
) {
size_t offset_at;
ptrdiff_t * offset;
size_t header_size;
size_t total_size;
unsigned char * mem;
unsigned char * obj;

/* Sanity-check alignment value */
if (obj_alignment & (obj_alignment - 1))
return NULL;

/* Calculate the offset position */
offset_at = pfx_size;
offset_at += sizeof *offset;
/* Check for wrap-around */
if (offset_at < pfx_size)
return NULL;
--offset_at;
offset_at /= sizeof *offset;

/* Calculate the header size */
header_size = (offset_at + 1) * sizeof *offset;
/* Check for wrap-around */
if (header_size < pfx_size)
return NULL;

/* Calculate padding */
if (obj_alignment > cv_ptrdiff_alignment) {
size_t new_hdr_size = header_size;
new_hdr_size += obj_alignment;
/* Check for wrap-around */
if (new_hdr_size < header_size)
return NULL;
--new_hdr_size;
new_hdr_size /= obj_alignment;
new_hdr_size *= obj_alignment;
header_size = new_hdr_size;
}

/* Allocate storage */
total_size = header_size + obj_size;
/* Check for wrap-around */
if (total_size < pfx_size || total_size < obj_size)
return NULL;
mem = malloc(total_size);
if (!mem)
return mem;

/* Point to the object */
obj = mem + header_size;

/* Note the offset to the prefix */
offset = (ptrdiff_t *) obj - 1;
*offset = obj - mem;

return obj;
}

/* Fetch an asociated prefix for an object */
void * pfxget(void * obj) {
ptrdiff_t * offset;
unsigned char * prefix;

if (!obj)
return obj;

offset = obj;
--offset;
prefix = obj;
prefix -= *offset;

return prefix;
}

(It could probably be improved.)

 

[Mark Adler]

If you're not too worried about portability, you can use
malloc_usable_size(), malloc_size(), or _msize(), depending on the
system. The value returned is the actual size of the block, which may
be a bit larger than what was requested.

Mark

 

[pembed2012]

Thanks, I've tried this but it does NOT work

This code prints: 0, it should print: 10

int get_malloc_size(void *vp)
{
char *cp = (char*)vp;
static int __done = -1;
static int __offset = -1;

Identifiers starting with two underscores are reserved to the
implementation. You should *never* define such identifiers in your own
code. The rules are a little more complex for identifiers starting with
a single underscore but you should still avoid them.

[snip]

return *(int*)(cp-__offset);
}

main()

Should be "int main(void)".

{
char *p=(char*)malloc(10);
printf("%d\n", get_malloc_size((void*)p)); free(p);
}

Why would you expect it to work? There is *no* portable way to
determine the size of a malloc()ed chunk of memory.

Actually, there is: just remember the argument you passed to malloc():

size_t requested_size = 10;
char *p = malloc(requested_size);
if (p != NULL) {
/* You know you've allocated requested_size bytes. */
}

Note that the implementation could allocate more memory than you
request; it won't necessarily store the requested size anywhere. But
that shouldn't matter to your program.

If you think it does matter, explain what you're trying to accomplish
and we can probably help you find another way.

Sorry it wasn't clear, I need to take a pointer from malloc and find out
the size of the allocation. To allow for dynamic pointers as well as
stack arrays. Like this:

myfunc(void* buf){
int bufsiz=-1;
if(sizeof(buf)==sizeof(void*)) { // pointer
bufsiz=get_malloc_size(buf)'
}
if(bufsize==-1){ // array
bufsiz=sizeof(buf);
}
// .... process buffer ...
}

Fully portable is best, but a solution working on 90% of compilers would
be OK if I understood right for you say it can't be done 100%

 

[Ike Naar]

myfunc(void* buf){
int bufsiz=-1;
if(sizeof(buf)==sizeof(void*)) { // pointer

Since the type of buf is void*, the condition is always true.

bufsiz=get_malloc_size(buf)'
}
if(bufsize==-1){ // array
bufsiz=sizeof(buf);
}
// .... process buffer ...
}

Fully portable is best, but a solution working on 90% of compilers would
be OK if I understood right for you say it can't be done 100%

One method that works 100% is to remember the size of the buffer
in a variable, and pass that information to myfunc along with the
pointer to the buffer.

#include <stdlib.h>

void myfunc(void *buf, size_t bufsize);

void using_pointer(void)
{
size_t size = 684;
char *pointer = malloc(size);
if (pointer != NULL)
{
myfunc(pointer, size);
}
}

void using_array(void)
{
char array[684];
myfunc(array, sizeof array);
}

 

[James Kuyper]

On 02/17/2012 02:45 PM, pembed2012 wrote:
....

Sorry it wasn't clear, I need to take a pointer from malloc and find out
the size of the allocation. To allow for dynamic pointers as well as
stack arrays. Like this:

myfunc(void* buf){
int bufsiz=-1;

It's absolutely guaranteed, at this point, that sizeof(buf) ==
sizeof(void*). As a result, 2 of your next three lines are pointless,
which suggests that you were not aware of that fact.

if(sizeof(buf)==sizeof(void*)) { // pointer
bufsiz=get_malloc_size(buf)'
}

Why did you think it was possible for the condition of that if() to not
be met?

 

[pembed2012]

On 02/17/2012 02:45 PM, pembed2012 wrote: ...

It's absolutely guaranteed, at this point, that sizeof(buf) ==
sizeof(void*). As a result, 2 of your next three lines are pointless,
which suggests that you were not aware of that fact.


Why did you think it was possible for the condition of that if() to not
be met?

Often this function will be called with a dynamic pointer from malloc.
However, it is also possible to call it with a stack array:

char ary[10];
char* ptr=malloc(10);
myfunc((void*)ary); // should work
myfunc((void*)ptr); // should also work

It started off with the buffers being NULL-free strings. However now I
need to work with strings with embedded NULL characters. This means I
can't find the length of the buffer with strlen. And it would be too much
change to add a new argument to pass the length of the buffer. So the
only good solution is for myfunc to work out by it's self 1) if the
buffer is a dynamic pointer or stack array and 2) the size of the buffer.

 

[James Kuyper]

On 02/17/2012 02:45 PM, pembed2012 wrote: ...

It's absolutely guaranteed, at this point, that sizeof(buf) ==
sizeof(void*). As a result, 2 of your next three lines are pointless,
which suggests that you were not aware of that fact.


Why did you think it was possible for the condition of that if() to not
be met?

Often this function will be called with a dynamic pointer from malloc.
However, it is also possible to call it with a stack array:

char ary[10];
char* ptr=malloc(10);
myfunc((void*)ary); // should work
myfunc((void*)ptr); // should also work

Assuming that the function's declaration matches the definition you've
given above, the casts are unnecessary in your code above; they specify
the same conversion that will happen anyway even if they weren't present.

Experienced programmers treat every occurrence of a cast as a potential
time bomb that needs to be examined carefully to make sure it's not an
actual bomb. While your code has mistakes, that's not one of them, so
you're better off removing those unnecessary casts.

Again, why do you think sizeof(buf)==sizeof(void*) will be false for one
of those two cases (presumably the first one)?

Incidentally, it's the wrong test even if it could work the way you were
expecting it to do. Consider:

char arr[sizeof(void*)];

It started off with the buffers being NULL-free strings. However now I
need to work with strings with embedded NULL characters. This means I
can't find the length of the buffer with strlen. And it would be too much
change to add a new argument to pass the length of the buffer. So the
only good solution is for myfunc to work out by it's self 1) if the
buffer is a dynamic pointer or stack array and 2) the size of the buffer.

Unfortunately, it's not possible for your function to figure out either
of those things in the way that you want it to. Your best solution is to
make the changes needed to add a new argument containing the length of
the buffer.

 

[Shao Miller]

Sorry it wasn't clear, I need to take a pointer from malloc and find out
the size of the allocation. To allow for dynamic pointers as well as
stack arrays. Like this:

myfunc(void* buf){
int bufsiz=-1;

/* The following condition is always true */

if(sizeof(buf)==sizeof(void*)) { // pointer

/* Oops. Semi-colon needed instead of ' */

bufsiz=get_malloc_size(buf)'
}

/* Oops. 'bufsiz', not 'bufsize' */

if(bufsize==-1){ // array /* This will never happen */
bufsiz=sizeof(buf);
}
// .... process buffer ...
}

Fully portable is best, but a solution working on 90% of compilers would
be OK if I understood right for you say it can't be done 100%

I don't think it matters what kind of pointer-to-object type you
originally pass to 'myfunc'. 'buf' is a 'void *' and 'sizeof buf'
should be equal to 'sizeof (void *)'.

Did you understand my example allocator that I posted? It ought to be
at least 90% portable.

 

[Shao Miller]

On 02/17/2012 02:45 PM, pembed2012 wrote: ...

It's absolutely guaranteed, at this point, that sizeof(buf) ==
sizeof(void*). As a result, 2 of your next three lines are pointless,
which suggests that you were not aware of that fact.


Why did you think it was possible for the condition of that if() to not
be met?

Often this function will be called with a dynamic pointer from malloc.
However, it is also possible to call it with a stack array:

char ary[10];
char* ptr=malloc(10);
myfunc((void*)ary); // should work
myfunc((void*)ptr); // should also work

It started off with the buffers being NULL-free strings. However now I
need to work with strings with embedded NULL characters. This means I
can't find the length of the buffer with strlen. And it would be too much
change to add a new argument to pass the length of the buffer. So the
only good solution is for myfunc to work out by it's self 1) if the
buffer is a dynamic pointer or stack array and 2) the size of the buffer.

I don't think a string has 'NULL' in it. I think a string can have a
null terminator. I think that 'NULL' is an invalid pointer value.

I think you are mistaken about what you are doing and expecting.

It looks like 'ary' has type 'char[10]'. When 'ary' is used in your
examples, it yields a 'char *' value. All pointer-to-object types can
be implicitly converted to 'void *', so your cast is redundant.

It looks like the type of 'ptr' is 'char *'. All pointer-to-object
types can be implicitly converted to 'void *', so your cast is redundant.

In C, if you have a pointer into some object (single object, array
object, "stack," allocated, static-duration; no difference), I do not
believe that you can expect that a pointer value carries size
information nor that even if it does, that you can access that information.

 

[Keith Thompson]

Often this function will be called with a dynamic pointer from malloc.
However, it is also possible to call it with a stack array:

char ary[10];
char* ptr=malloc(10);
myfunc((void*)ary); // should work
myfunc((void*)ptr); // should also work

Not possible. It's not just not 100% portable, it's not possible in any
implementation I know of. You'll need to find a way to keep track of
the size yourself.

It started off with the buffers being NULL-free strings. However now I
need to work with strings with embedded NULL characters.

NULL is (a macro that expands to) a null *pointer* constant. You're
talking about null *characters*. I know that some character set
standards use the term "NULL", but you shouldn't use that term to refer
to null characters in the context of C.

A "string" in C is, by definition, "a contiguous sequence of characters
terminated by and including the first null character", so there's no
such thing as a "null-free string". A character array that doesn't
contain any null characters is not a string; it's simply a character
array.

This means I
can't find the length of the buffer with strlen.

strlen doesn't give you the size of a buffer; it searches for the
first null character. You need to decide whether you want the
length of a string contained in a buffer, or the size of the buffer
itself, and you can't reasonably decide that based on its content.
If the buffer holds arbitrary data, it might still contain a null
character that's not meant to be a string terminator; what should
your function return in that case, and how can it decide?

And it would be too much
change to add a new argument to pass the length of the buffer. So the
only good solution is for myfunc to work out by it's self 1) if the
buffer is a dynamic pointer or stack array and 2) the size of the buffer.

Then I'm afraid you've got "too much change" ahead of you -- unless
perhaps you can reformulate the problem somehow. (We don't know *why*
you need this functionality.)

 

[Shao Miller]

Then I'm afraid you've got "too much change" ahead of you -- unless
perhaps you can reformulate the problem somehow. (We don't know *why*
you need this functionality.)

Or the OP could wrap the *alloc functions and 'free' so that the size is
tracked. They could even #define over the *alloc functions and 'free'.
Then they could also implement their 'alloc_size' function or whatever
it was called.

Of course, doing that doesn't help with the misunderstanding that the
size of an _array_ is somehow available from a pointer _value_.

 

[Ian Pilcher]

It started off with the buffers being NULL-free strings. However now I
need to work with strings with embedded NULL characters. This means I
can't find the length of the buffer with strlen. And it would be too much
change to add a new argument to pass the length of the buffer. So the
only good solution is for myfunc to work out by it's self 1) if the
buffer is a dynamic pointer or stack array and 2) the size of the buffer.

Prepare to be disappointed.