xmalloc

[CBFalconer]

(e-mail address removed) (Roland Pibinger) writes:

[...]

The discussion basically boils down to one question: Is OOM an
error that reasonably can and should be handled by the
application or is it a fatal error? The 'fatal error' advocates
have already shown how they tackle the problem. Now it's time
for the other camp to demonstrate how OOM can be consistently
_handled_ throughout the program ('return NULL;' is not enough).

Who says it can be handled *consistently*?

That is part of the issue. malloc() return NULL. The if condition
following the malloc() is like a yippy dog chasing a car. Now it's
got an error, what is it going to do with it?

The answer is simple. The model for calling malloc is:

if (!(p = malloc(N * sizeof *p))) callerrorrecoveryetc();
else { /* all is well */
....
}

 

[Malcolm McLean]

And the janitor might unplug the computer to plug in the
floor polisher, or the fire sprinklers might open up and flood
the motherboard, or a meteorite might smash the computer to
tiny bits, or the Borg might teleport it away for assimilation.
Does the existence of Failure Mode A dissuade you from taking
any precautions against Failure Mode B? "I don't wear seat
belts in cars, because avian flu might kill me anyhow." Pfui!

Yes. That's actually a fundamental observation.

Let's say we've got a failure rate of 10% per annum. By eliminating cause B,
we can reduce that to 9.9%. Unfortunately the cost of the product goes up
10% as result. Almost certainly it is not worth it.
Now imagine the failure rate is 1% per annnum. By eliminating cause B, we
can reduce that to 0.9%. Again, cost rises by 10%.
In the final scenario, failure rate is 0.1% per annum. By eliminating cause
B we can reduc that to 0.01% per annum, again for the same price.

In the middle scenario eliminating feature B is quite attractive. In the
last one, almost certainly you will want to do it, because the you can claim
a nearly failure-free product. The only exception would be if customers care
about price and not about quality at all - a T-shirt with a slogan on it,
for instance.

The value of eliminating an error goes up as it accounts for a greater
proportion of overall errors.

 

[Roland Pibinger]

Already mentioned a few times in this thread:

buff = malloc(image_size);
if (buff == NULL) {
fprintf (stderr, "Image too large (%lu) to paste\n",
(unsigned long)image_size);
return;
}
/* read image into buff, insert in current document */

Here's another I think has been referred to:

must_have_mem = malloc(how_much);
if (must_have_mem == NULL) {
fprintf (stderr, "Out of memory; shutting down\n");
save_snapshot(snapshot_file);
exit (EXIT_FAILURE);
}
/* store "must have" data in allocated memory */

The above approachs do not handle OOM ('handle' in the meaning of
'remedy'). They pass the problem to the caller or exit the program.
The second approach would soon be refactored into a malloc_or_exit()
function.

And here's still another (I don't remember whether it's
cropped up in this thread yet):

void *getmem(size_t bytes) {
void *new = malloc(bytes);
if (new == NULL && bytes > 0) {
fprintf (stderr, "Failed to allocate %lu bytes\n",
(unsigned long)bytes);
free (emergency_stash);
emergency_stash = NULL;
new = malloc(bytes);
if (new == NULL) {
fprintf (stderr, "You were warned ...!\n");
exit (EXIT_FAILURE);
}
fprintf (stderr, "Running on fumes: save your work "
"and exit soon!\n");
}
return NULL;
}

This actually is an attempt to cope with OOM. But, as demonstrated,
the possibilities are very limited. You cannot do much when the
primary source of your program is exhausted.

Even if out-of-memory is a "fatal error," it does not follow
that the program should have no opportunity to "die with dignity."
Have you made a will, Roland?

I consider to use

void *xmalloc (size_t bytes) {
return getmem (bytes);
}

which, IMHO, elegently combines the best of all approaches posted so
far in this thread ;-)

In general I prefer: 'Repair what you can — but when you must fail,
fail noisily and as soon as possible'
(http://www.catb.org/~esr/writings/taoup/html/ch01s06.html#id2878538)

 

[Keith Thompson]

That is part of the issue.
malloc() return NULL. The if condition following the malloc() is like
a yippy dog chasing a car. Now it's got an error, what is it going to
do with it?

The "yippy dog chasing a car" analogy applies equally well when
malloc() returns a non-null result. Now you've got a valid pointer to
allocated memory; what are you going to do with it?

The answer is: do whatever makes sense in the context of the program.

Determining the answers to such questions is called "programming";
it's a non-trivial activity.

 

[Eric Sosman]

The above approachs do not handle OOM ('handle' in the meaning of
'remedy'). They pass the problem to the caller or exit the program.
The second approach would soon be refactored into a malloc_or_exit()
function.

Where did "remedy" creep into the picture? Both of the
above sketches acknowledge that the program is unable to
carry out some operation in the absence of sufficient memory,
but both "handle" the lack by taking a substitute action.
The first might be found in something like an interactive
editor, where the user gets a chance to say "Forty gig? Oh,
right, that's the original super-high-resolution file, not
the low-res image I actually intended to use." The second
does something that seems to have escaped your notice, to
wit, it calls save_snapshot() to preserve the valuable state
before pulling the plug. Neither strategy could be implemented
if malloc() just decided to blow up the program on its own whim.

This actually is an attempt to cope with OOM. But, as demonstrated,
the possibilities are very limited. You cannot do much when the
primary source of your program is exhausted.

How much you can do depends on how much you have made
provision for. That in turn depends on your analysis of how
much memory the program requires for "last gasp" operations.
At a PPOE this feature of the program received regular Q/A:
a whole bunch of state was loaded into memory, and then the
test harness induced artificial malloc failures; if the
program's emergency stash wasn't enough to sustain the last
gasp, a bug got filed.

I consider to use

void *xmalloc (size_t bytes) {
return getmem (bytes);
}

which, IMHO, elegently combines the best of all approaches posted so
far in this thread ;-)

In general I prefer: 'Repair what you can — but when you must fail,
fail noisily and as soon as possible'
(http://www.catb.org/~esr/writings/taoup/html/ch01s06.html#id2878538)

"When you must fail" is not blanket advice to give up
the ghost at the first sign of adversity. You might also
note that all three of the sketches I provided made noise.

 

[Nick Keighley]

In which case would call regular malloc()
However you are doubling the cost of the software with that strategy, all
because a 2GB machine might refuse to give a few hundred bytes of memory. If
it is running a life-support machine, fair enough, but most software
doesn't.

On the other hand you could argue that xmalloc() is a bad idea, because no
library that uses it could ever find its way into a life-support machine.
On the other hand, you could argue that the life support machine is more
likely to blow a fuse than to exit in xmalloc(), so you need a backup
anyway.

I assume "life-support machine" is a meta syntactic variable for
safety/mission critical software.

Just how important does a program have to be before its given
LSM status and doesn't crash without doing *some* error handling?

- actual life support machine
- anti-lock brakes
- communications equipment used by emergency services
- communications equipment used to report aircraft movements
- your bank account
- your word-processor
- your central heating system
- the game you've been playing for a week
- unix filter
- post to clc

 

[Christopher Benson-Manica]

It could be enhanced by providing a (cough) global failure function, say
xmalloc_failure, that gets called when malloc returns NULL:

void* (*xmalloc_failure)(size_t) = &default_xmalloc_failure;

Why make it a global function?

static void* (*xmalloc_failure)( size_t );

void register_xmalloc_failfunc( void* (*failfunc)(size_t) ) {
xmalloc_failure = failfunc;
}

There's no need to expose xmalloc_failure or default_xmalloc_failure
to client code, and I would personally prefer not to.

 

[Christopher Benson-Manica]

void *xmalloc(int sz)
{

assert(sz >= 0);
if(sz == 0)
sz = 1;

All right, I give up. Why should xmalloc() quietly prohibit zero-byte
allocations? If a client asks for zero bytes, why not just *give* it
zero bytes? For that matter, given the above code, what is wrong with

if( sz <= 0 ) {
sz=1;
}

? You seem to be assuming that xmalloc()'s clients aren't very bright
anyway.

 

[Harald van =?UTF-8?B?RMSzaw==?=]

All right, I give up. Why should xmalloc() quietly prohibit zero-byte
allocations? If a client asks for zero bytes, why not just *give* it
zero bytes? For that matter, given the above code, what is wrong with

if( sz <= 0 ) {
sz=1;
}

? You seem to be assuming that xmalloc()'s clients aren't very bright
anyway.

malloc(0) can make sense, but is allowed by the standard to either return
NULL or non-NULL on success. Because of this, xmalloc(0) calls malloc(1) so
that a successfull allocation will always return non-NULL.

 

[Christopher Benson-Manica]

malloc(0) can make sense, but is allowed by the standard to either return
NULL or non-NULL on success. Because of this, xmalloc(0) calls malloc(1) so
that a successfull allocation will always return non-NULL.

I take about half of this point. While the code as written did indeed
need to distinguish between failure to allocate non-zero bytes, and
successful allocation of zero bytes (a fact which I missed), it's not
clear to me that xmalloc()'s clients gain anything by essentially
having all calls to malloc(0) replaced by malloc(1). The pointer
returned by malloc(0), if not NULL, can't be used to access an object
anyway, so from xmalloc()'s clients' perspectives, it doesn't much
matter whether xmalloc(0) returns NULL or not:

void *xmalloc( size_t sz ) {
if( sz == 0 ) {
return NULL;
}
/* ... */
}

May as well save the call to malloc(0), if you ask me.

 

[Harald van =?UTF-8?B?RMSzaw==?=]

I take about half of this point. While the code as written did indeed
need to distinguish between failure to allocate non-zero bytes, and
successful allocation of zero bytes (a fact which I missed), it's not
clear to me that xmalloc()'s clients gain anything by essentially
having all calls to malloc(0) replaced by malloc(1). The pointer
returned by malloc(0), if not NULL, can't be used to access an object
anyway, so from xmalloc()'s clients' perspectives, it doesn't much
matter whether xmalloc(0) returns NULL or not:

void *xmalloc( size_t sz ) {
if( sz == 0 ) {
return NULL;
}
/* ... */
}

May as well save the call to malloc(0), if you ask me.

It depends on the project. Sometimes it matters to have different calls to
malloc(0) return values that do not compare equal, sometimes it doesn't.

 

[banansol]

What malloc() problem? And this "solution" suffers from the same problem
as any other "solution" of the same kind - it's no earthly use in a
library,
and libraries are where malloc calls belong.

Does that mean that malloc calls not are to be used in regular
programs? If yes, what should one use instead? Get some library
or write one by oneself?

Also what does the x mean in xmalloc? Iv'e seen it numerous of
times in sources.

Thanks!

 

[CBFalconer]

All right, I give up. Why should xmalloc() quietly prohibit
zero-byte allocations? If a client asks for zero bytes, why not
just *give* it zero bytes? For that matter, given the above code,

Because the C-standard allows a malloc call for zero bytes to
return a NULL pointer, and you can't tell that from a failure. For
this (and other) reasons my nmalloc increments a zero allocation
size to 1, and then allocates accordingly. Same for realloc.

 

[Richard Heathfield]

(e-mail address removed) said:

Does that mean that malloc calls not are to be used in regular
programs?

Ah, I see that my comment may have been misleading. No, I don't wish to
suggest that you should *never* use malloc except in libraries. It's
just that it has been my experience that, once a program has been
through the mill and been refactored and rejigged and generally beaten
into shape, the malloc stuff tends to end up in a separate module that
is eminently suitable for being reused in other programs.

In other words, I was just describing what *usually* happens, not what
*must* happen.

Also what does the x mean in xmalloc? Iv'e seen it numerous of
times in sources.

No idea. And I don't really care either, since I never use it.

 

[Christopher Benson-Manica]

It depends on the project. Sometimes it matters to have different calls to
malloc(0) return values that do not compare equal, sometimes it doesn't.

And if it matters, and if the implementation chooses to always return
NULL for malloc(0), what then?

 

[Harald van =?UTF-8?B?RMSzaw==?=]

And if it matters, and if the implementation chooses to always return
NULL for malloc(0), what then?

Then the xmalloc helper function changes the request for 0 bytes by one for
1 byte, and all is fine.

 

[Christopher Benson-Manica]

Christopher Benson-Manica wrote:

Then the xmalloc helper function changes the request for 0 bytes by one for
1 byte, and all is fine.

My point is that malloc() does not behave like that, and so at that
point xmalloc() becomes more than a drop-in replacement for malloc().
At that point the function should at the least be renamed so that it
is clear that it cannot be replaced by malloc() without verifying that
a program does not rely on this IMHO subtle behavior of xmalloc().

 

[Harald van =?UTF-8?B?RMSzaw==?=]

My point is that malloc() does not behave like that, and so at that
point xmalloc() becomes more than a drop-in replacement for malloc().
At that point the function should at the least be renamed so that it
is clear that it cannot be replaced by malloc() without verifying that
a program does not rely on this IMHO subtle behavior of xmalloc().

Well, yes, just like you can't blindly replace xmalloc calls with malloc
calls without adding NULL checks on every call.

 

[Keith Thompson]

My point is that malloc() does not behave like that, and so at that
point xmalloc() becomes more than a drop-in replacement for malloc().
At that point the function should at the least be renamed so that it
is clear that it cannot be replaced by malloc() without verifying that
a program does not rely on this IMHO subtle behavior of xmalloc().

The behavior of xmalloc is still *consistent* with the allowed
behavior of malloc; no portable code can assume that malloc(0) can
return a non-null result.

The fact that xmalloc differs from malloc is made clear enough by the
fact that it has a different name.

 

[Eric Sosman]

Keith Thompson wrote On 06/26/07 00:05,:

The behavior of xmalloc is still *consistent* with the allowed
behavior of malloc; no portable code can assume that malloc(0) can
return a non-null result.

malloc() is permitted to call exit()? When did
that happen?