clc_wprealloc

[Malcolm]

I am going to propose this function for the clc library

void *clc_wprealloc(void *buff, size_t size)
{
void *answer;

answer = realloc(buff, size);
if(!answer)
free(buff);
return answer;
}

wp stands for waterproof. What's the point of this, you might ask. The point
is that, 9 times out of 10, if system won't let you resize the array you are
working on there is no option but to free it, often terminate the program,
or else abort the function and return to a higher level.

However, since realloc() won't free the buff it fails to resize, you need an
awkward dance with a temporary to avoid a memory leak - which probably can't
happen in reality anyway because you are asking for a few bytes to hold
characters on a system with 4GB of swap space, but you still want the
program to be correct.

clc_wprealloc() has to be part of a widely-used library like clc will
become. Adding it into user code would create dependencies and confuse
people as to why standard realloc() wasn't being called.

 

[Alan Balmer]

I am going to propose this function for the clc library

void *clc_wprealloc(void *buff, size_t size)
{
void *answer;

answer = realloc(buff, size);
if(!answer)
free(buff);
return answer;
}

wp stands for waterproof.

?? Good functional name ??

clc_realloc_or_perish() would seem better.

What's the point of this, you might ask. The point
is that, 9 times out of 10, if system won't let you resize the array you are
working on there is no option but to free it, often terminate the program,
or else abort the function and return to a higher level.

YMMV, but I usually find I want to do something other than free the
buffer and throw away the data I already have.

 

[Richard Heathfield]

I am going to propose this function for the clc library

void *clc_wprealloc(void *buff, size_t size)
{
void *answer;

answer = realloc(buff, size);
if(!answer)
free(buff);
return answer;
}

wp stands for waterproof. What's the point of this, you might ask. The
point is that, 9 times out of 10, if system won't let you resize the array
you are working on there is no option but to free it, often terminate the
program, or else abort the function and return to a higher level.

What to do when memory fails (a short summary culled from <cough> well, call
it my contribution to "the literature"):

1) Terminate the program. Fine for student code.

2) Break down the memory requirement into smaller chunks.

3) Use less memory.

4) Use a fixed length buffer.

5) Pre-allocate an emergency reserve.

6) Use virtual disk space (perhaps on another machine over a network).

Lots of options really. You don't necessarily have to give up just because
one allocation failed.

<snip>

 

[Rouben Rostamian]

I am going to propose this function for the clc library

void *clc_wprealloc(void *buff, size_t size)
{
void *answer;

answer = realloc(buff, size);
if(!answer)
free(buff);
return answer;
}

wp stands for waterproof. What's the point of this, you might ask. The point
is that, 9 times out of 10, if system won't let you resize the array you are
working on there is no option but to free it, often terminate the program,
or else abort the function and return to a higher level.

That's good, but there is one caveat: after realloc() fails and free()
is called, the value of errno may be undefined. Therefore the caller
of clc_wprealloc() should not attempt to print a failure message with
perror() as he would have done after a direct call to realloc().

 

[Blah]

What to do when memory fails (a short summary culled from <cough> well, call
it my contribution to "the literature"):

1) Terminate the program. Fine for student code.

2) Break down the memory requirement into smaller chunks.

3) Use less memory.

4) Use a fixed length buffer.

5) Pre-allocate an emergency reserve.

6) Use virtual disk space (perhaps on another machine over a network).

Lots of options really. You don't necessarily have to give up just because
one allocation failed.

Do you happen to have any good examples, or know a site with good
examples of working around running out of memory? This is one (of many)
area of coding that I could use improvement on.

 

[Hallvard B Furuseth]

That's good, but there is one caveat: after realloc() fails and free()
is called, the value of errno may be undefined.

malloc & co do not need to set errno anyway, though some (most?)
implementations do. But clc_wprealloc() could always save errno before
free() and restore it afterwards.

 

[Dan Pop]

malloc & co do not need to set errno anyway, though some (most?)
implementations do.

But, since most of them only set it to ENOMEM, there is little point in
checking errno after a failed malloc and friends call.

But clc_wprealloc() could always save errno before
free() and restore it afterwards.

An exercise in pedantry free() has no business to touch errno and,
even if it does, losing ENOMEM is no big deal.

Even better, set errno to a CLC_Ecode in this case.

Dan

 

[Richard Heathfield]

What to do when memory fails [...]:
1) Terminate the program. Fine for student code.
2) Break down the memory requirement into smaller chunks.
3) Use less memory.
4) Use a fixed length buffer.
5) Pre-allocate an emergency reserve.
6) Use virtual disk space (perhaps on another machine over a network).
Lots of options really. You don't necessarily have to give up just
because one allocation failed.

Do you happen to have any good examples, or know a site with good
examples of working around running out of memory? This is one (of many)
area of coding that I could use improvement on.

Example of breaking down the requirement: you want sixteen million buckets
for a hash table. The allocator says no, so you ask for sixteen lots of one
million buckets instead, and do the first nibble of the lookup in code.
Slightly slower, but better than nothing, eh?

Example of using less memory/fixed length buffer: you are copying N bytes,
starting at S, from a file, and writing them to a new file. Ideally, you'd
like a buffer as big as the file. If you can get it, great. If not, you can
manage with, say, S bytes - or N bytes - or, if it comes right down to it,
a single character (the ultimately minimalist fixed-length buffer).

Thinking up examples for emergency reserves and virtual space is left as an
exercise, because I'd rather be writing a cool game than doing Usenet right
now.

 

[=?ISO-8859-1?Q?Bj=F8rn_Augestad?=]

I am going to propose this function for the clc library

void *clc_wprealloc(void *buff, size_t size)
{
void *answer;

answer = realloc(buff, size);
if(!answer)
free(buff);
return answer;
}

wp stands for waterproof. What's the point of this, you might ask. The point
is that, 9 times out of 10, if system won't let you resize the array you are
working on there is no option but to free it, often terminate the program,
or else abort the function and return to a higher level.

However, since realloc() won't free the buff it fails to resize, you need an
awkward dance with a temporary to avoid a memory leak - which probably can't
happen in reality anyway because you are asking for a few bytes to hold
characters on a system with 4GB of swap space, but you still want the
program to be correct.

clc_wprealloc() has to be part of a widely-used library like clc will
become. Adding it into user code would create dependencies and confuse
people as to why standard realloc() wasn't being called.

Hi, Malcolm, and thank you for writing.

Others have mentioned that there are many ways of recovering from an out
of memory situation. Most of these strategies does not involve freeing
the allocated memory. Since clc_wprealloc() always frees the original
memory if the call to realloc() fails, it cannot be used as a general
realloc() replacement/enhancement.

Another problem with clc_wprealloc() is that it may invoke a double
free(). If size is 0, then it is implementation defined if realloc()
returns NULL or not. (This can of course easily be fixed.

Maybe clc_wprealloc() should be renamed to something that better
describes what its intended use is and reduces the association with
realloc? clc_resize(), clc_extend(), better ideas ?

Or how about something like this little snippet, which will solve parts
of the problem with realloc()?

int clc_realloc(void** p, size_t size)
{
void* np = realloc(*p, size);
if(np != NULL)
*p = np;

if(size == 0)
return 1;
else
return np != NULL;
}

The unfortunate thing is that such a solution requires casts to void**
to be able to call it without warnings, which is quite annoying in my
view. (BTW, why is such a cast needed?)

 

[Hallvard B Furuseth]

Another problem with clc_wprealloc() is that it may invoke a double
free(). If size is 0, then it is implementation defined if realloc()
returns NULL or not. (This can of course easily be fixed.

No. ANSI 4.10.3.4 ends says 'If size is zero and ptr is not a null
pointer, the object it points to is freed.' But if you don't trust
every realloc() to handle size=0 correctly, just handle it by hand.

int clc_realloc(void** p, size_t size)
{
void* np = realloc(*p, size);
if(np != NULL)
*p = np;

Absolutely not. See below.

The unfortunate thing is that such a solution requires casts to void**
to be able to call it without warnings, which is quite annoying in my
view. (BTW, why is such a cast needed?)

Because `int *ip' and `void *vp' may have different representations.
So if you pass &ip to a function, it must be accessed as an int**, not
a void **. Otherwise you'd write the representation of a void pointer
into an int pointer.

 

[Richard Heathfield]

I am going to propose this function for the clc library

void *clc_wprealloc(void *buff, size_t size)
{
void *answer;

answer = realloc(buff, size);
if(!answer)
free(buff);
return answer;
}

wp stands for waterproof. What's the point of this, you might ask. The
point is that, 9 times out of 10, if system won't let you resize the
array you are working on there is no option but to free it, often
terminate the program, or else abort the function and return to a higher
level.

However, since realloc() won't free the buff it fails to resize, you need
an awkward dance with a temporary to avoid a memory leak - which probably
can't happen in reality anyway because you are asking for a few bytes to
hold characters on a system with 4GB of swap space, but you still want
the program to be correct.

clc_wprealloc() has to be part of a widely-used library like clc will
become. Adding it into user code would create dependencies and confuse
people as to why standard realloc() wasn't being called.

Hi, Malcolm, and thank you for writing.

Others have mentioned that there are many ways of recovering from an out
of memory situation. Most of these strategies does not involve freeing
the allocated memory. Since clc_wprealloc() always frees the original
memory if the call to realloc() fails, it cannot be used as a general
realloc() replacement/enhancement.

Another problem with clc_wprealloc() is that it may invoke a double
free(). If size is 0, then it is implementation defined if realloc()
returns NULL or not. (This can of course easily be fixed.

Maybe clc_wprealloc() should be renamed to something that better
describes what its intended use is and reduces the association with
realloc? clc_resize(), clc_extend(), better ideas ?

Or how about something like this little snippet, which will solve parts
of the problem with realloc()?

It doesn't.

int clc_realloc(void** p, size_t size)
{
void* np = realloc(*p, size);
if(np != NULL)
*p = np;

if(size == 0)
return 1;
else
return np != NULL;
}

The unfortunate thing is that such a solution requires casts to void**
to be able to call it without warnings, which is quite annoying in my
view. (BTW, why is such a cast needed?)

From "the literature":

"The next task is to take care of tracking all the calls to
the realloc wrapper. This function, like the other wrapper
functions, used to take void ** as its first parameter.
This was a really good idea. Instead of slavishly following
the realloc interface, I could improve upon it. After all,
it was very clear to me that the ANSI committee, or Dennis
Ritchie, or whoever it was, had not considered the design
of the realloc function as carefully as they should have
done. So, how to improve it?

I could pass the address of the old pointer, rather than
its value, and update it within the wrapper function if
and only if the internal realloc call succeeded. The
function would return not a pointer but an integer
representing success or failure, in accordance with my
normal practice at that time. Therefore, the

ptr = realloc(ptr, newsize)

bug would be squashed forever! I was really rather pleased
with this idea, and I lost little time in coding it. It
worked rather well, too.

You're right. As I discovered when recompiling and retesting
the code for this book, it's not a portable technique. While
it is true that you can assign any object pointer to a void *
pointer without loss of information, the same does not
necessarily apply to a void ** pointer! It worked fine in
the compiler I was using at the time, but that's the best that
can be said about it. In order to force it to work in ANSI C,
I'd have to cast, in the call, something like this:

DidItWork = ReallocMemory(&(void *)p, newsize);

Sometimes casting is necessary (as here), but it's always ugly.
If your function's design requires that the user must always or
nearly always cast his arguments to that function, then you have
to think seriously about whether you have the right design.

Having thought seriously about whether I had the right design,
I concluded that I didn't. I had no desire to write &(void *)
anywhere in my source code, ever. So I considered some other
possibilities. One idea was to wrap the pointer in a structure.
The trouble with that idea, though, was that I'd have to wrap
up the pointer, pass the structure, dig out the pointer within
the wrapper, do the allocation, update the pointer, and then
let the calling function dig out the pointer again, so that it
could actually be used. That sounded like a lot of hard work,
especially for the user-programmer -- something I try to avoid
if I can; I like the library to do the work whenever possible.
I was determined, though, to come up with the best possible
design, rather than just meekly copy the ANSI method.

Finally, I established a foolproof technique. Well, almost
foolproof. It would mean trusting the calling function a bit,
though. If I simply took a normal void *, I would avoid the
casting problem. Of course, the fact that C always passes
parameters by value would mean that I couldn't (effectively)
update the pointer, so I'd have to return a pointer from the
function instead of a status code. That would be all right,
though, because I could use NULL to indicate failure.
Admittedly, that would place the onus on the user-programmer
to use a spare pointer in case of failure, but that was a
small price to pay for the cleaner technique.

I took a step back to admire my handiwork and discovered
that I had just re-implemented ANSI C's realloc design! There's
a time to fight and a time to accept the inevitable, so I gave
in; it seems that whoever designed realloc did know what they
were doing, after all. There's a lesson there somewhere for
overzealous and arrogant wheel inventors."

 

[Dan Pop]

No. ANSI 4.10.3.4 ends says 'If size is zero and ptr is not a null
pointer, the object it points to is freed.'

C99 has changed that.

Dan

 

[Richard Heathfield]

Richard Heathfield wrote:

From "the literature":

LOL.

That was a great post, is it a quote from C Unleashed?
Yes.

If so, is the
rest of the book written in the same style?

Well, that's a secret, which can only be revealed to very special people.

 

[Randy Howard]

I took a step back to admire my handiwork and discovered
that I had just re-implemented ANSI C's realloc design! There's
a time to fight and a time to accept the inevitable, so I gave
in; it seems that whoever designed realloc did know what they
were doing, after all. There's a lesson there somewhere for
overzealous and arrogant wheel inventors."

*chuckle*

Flashbacks from the book this is quoted from popped up when this
thread first appeared. I was thinking "Why do we need a realloc
wrapper anyway? I've seen this before," I just didn't remember
where I had seen it. It seems there is a trend creeping into the
clc library effort to reinvent every single wheel in the
standard library. Such "copies", open source or not, are not
usually something one would need to add to an existing C
development environment for any real reason. Adding useful
functionality not available elsewhere (portably) was as I
remember the purpose offered up when libclc was originally
proposed.

A lot of the string functions added so far fall into the category
of functions similar (or identical) to those available as
extensions on some, but not all platforms. As such, putting them
into libclc allows programmers to use them on any platform capable
of building the library (rather limited currently due to the
distribution being based on tar files and autoconf, but hopefully
that will change over time).

On the other hand, adding clc_ versions of existing, standard
functions doesn't seem to buy much, if anything. As an example, it
would be fairly difficult to defend the use of clc_printf() in a
code review over printf() if the calling semantics and feature set
was identical in both, just for the sake of using libclc. There
should be some tangible, measurable benefit of putting it in the
library, and that doesn't mean "Well, on the DS9K it happens to be
3 clock ticks faster than the implementation in the standard
library that comes with the stock compiler".

Ideally, the clc_* functions are so compelling on their own, that
when a developer encounters a project using them, he immediately
wants to download a copy of the library for his own usage because
the API's richness warrants it based on functionality and/or ease
of use alone. If *similar* to a standard function, it should be
suitably different to warrant rewriting existing code to use the
clc variant, or it probably shouldn't be in there in the first
place.

--
Randy Howard
(remove the obvious bits from my address to reply.)
"Most of the drivers nowadays are a bit like Eddie Irvine, who if
he was half as fast as he thought he was, would be moderate."
-- Sir Stirling Moss

 

[Ian Woods]

What, people who bought a copy? That *is* a select group.

It still seems to be popular at the local public libary, as all three
copies are checked out. Wait, three copies? I thought they bought four
when I requested it! Maybe one went walkabout.



Brian Rodenborn

Wow! Obviously someone thought it was so good they'd better steal it. Now
how's that for an endorsement

Ian Woods

 

[Ben Pfaff]

Example of using less memory/fixed length buffer: you are copying N bytes,
starting at S, from a file, and writing them to a new file.

Sounds like hell on caches.

Actually, this situation allows me to use a favorite construct:

char buf[BUFSIZ];
FILE *file = ...;

while (fwrite(buf, 1, fread(buf, 1, sizeof buf, file), file) == sizeof buf)
continue;

 

[Richard Heathfield]

You can't take the address of the result of a cast operator - it's not
an lvalue.

Right. I obviously didn't have my head screwed on properly that day. Not
only is it butt-ugly, but it's also illegal. All the more reason not to use
it.

<snip>

 

[Malcolm]

Others have mentioned that there are many ways of recovering from an out
of memory situation. Most of these strategies does not involve freeing
the allocated memory. Since clc_wprealloc() always frees the original
memory if the call to realloc() fails, it cannot be used as a general
realloc() replacement/enhancement.

It's not meant to replace all calls to realloc(). Sometimes the data is
precious, sometimes, as Richard Heathfield said, you want to implement a
fallback out of memory strategy (though I don't think such situations are as
common as he implies).
Generally if you can't get the memory you want, the thing to do is to
terminate the functin with an error code. Say I am inputting an
English-language sentence as part of an adventure game.
We haven't decided whether to limit the user's space or not, so this is a
case for a realloc() to allow an arbitrary sized buffer. A real sentence,
i.e. nor an attack by hackers, is likely to be under 1000 characters long.
On a big machine, this means that for all practical purposes it is
impossible to run out of memory. It certainly isn't worth rewriting our
complex parse_sentence(char *input) function as as series of
partial_parse(char *part_of_sentence) functions just to cope with out of
memory.
However we still need to keep the check of out of memory, partly because of
the hacker risk, partly because there might be a bug, partly just to have a
formally correct program.
What I've described isn't an unusual situation. The unusual situation is to
be capable of continuing the program when you don't get the memory you ask
for. In such a situation, stdlib realloc() is available. For everything
else, clc_wprealloc() eliminates a temporary variable from code, making it
cleaner.

 

[Randy Howard]

What I've described isn't an unusual situation. The unusual situation is to
be capable of continuing the program when you don't get the memory you ask
for. In such a situation, stdlib realloc() is available. For everything
else, clc_wprealloc() eliminates a temporary variable from code, making it
cleaner.

So put code into libclc to eliminate a temp variable? Puhleeze.