When to check the return value of malloc

[Flash Gordon]

Malcolm McLean wrote, On 22/01/08 11:59:

Try to follow the logic.
Memory allocation failure always causes a program to behave in some way
that is sub-optimal for the user. Otherwise the program wouldn't request
the memory at all. So failures, like the Federal Government running out
of money, are rare in terms of cycles. Flash can just about tolerate his
machine running out of memory once a month. I'd say that if the machine
runs out of memory every day, pretty soon the system will be replaced.

Actually it would not because that would be too expensive (the notebook
cost over 1000UKP. Now that RAM prices for notebooks have dropped it
*might* be upgraded.

The question is, where will the failure occur? In a big allocation, or a
little allocation?

Depends. On the days when I hit allocation failures I have normally
already started all the memory hogs and can see that memory is exhausted
using a monitor *before* I hit the errors. In fact, I run a resource
monitor continuously for precisely this reason.

With some simplifying assumptions,

Without fully documenting assumptions your calculation is worth nothing.

the answer is very
easy to calculate, and so you can work out the size of allocation that
is orders of magnitude less likely to fail than a hardware failure, that
is to says the error-handling code is vanishingly unlikely to be executed.

So when my resource monitor says that 100% of memory is in use I am
still more likely to have the HW fail than have a small memory
allocation fail? I think not.

If you remove those simplifying assumptions the question does become a
bit more difficult.

You need to because they are false. Basing your decisions on false
assumptions is always a bad idea, especially when someone has pointed
out that they are false.

However the costs of error-handling code are not
negligible.

Depends on whether you are good at design and programming.

Adding code that will never be executed is classical example
of over-engineering.

Ah, but it has already been pointed out that this is not code that will
never be executed.

The answer to the OP's question is "the point at
which a hardware failure becomes statistically more likely".

Which makes it easy. Since you cannot calculate either probability with
any great assurance you have to assume you need the checks.

 

[Flash Gordon]

CBFalconer wrote, On 21/01/08 23:19:

Malcolm McLean wrote:
... snip ...

Why don't you simply make sz a size_t, after which you don't need
the assert (which is a horrible thing to let loose anyhow). If an
int can hold larger values than a size_t you will also avoid asking
for impossibly large allocations, which at present will cause an
infinite loop.

Actually, if int is larger than size_t and overly large value will be
reduced in the usual way for conversion to unsigned leading to a far
*smaller* amount being allocated successfully than the amount that was
requested. A bug I had not thought of until now and another good reason
for not using int.

 

[Kelsey Bjarnason]

[snips]

/*
failsafe malloc drop-in

I'm not sure what "drop in" means; I suspect something like "drop-in
replacement", in which case your comments lie: it is *not* a drop-in
replacement for malloc, as it doesn't even use the right type for the
size parameter.

It's one thing to write bad code, it's another entirely to try to fool
the unwary coder into doing things that are going to break because you're
lying to him.

Params: sz - amount of memory to allocate Returns: allocated block
(never returns NULL even on block 0)

WTF is "block 0"?

void *xmalloc(int sz)
{
void *answer = 0;

assert(sz >= 0);

Last I checked, assert only exists in debug mode as a rule. So in
production, you merrily ignore sizes less than zero and just pass them on
to malloc? How bizarre.

 

[user923005]

Try to follow the logic.
Memory allocation failure always causes a program to behave in some way that
is sub-optimal for the user. Otherwise the program wouldn't request the
memory at all. So failures, like the Federal Government running out of
money, are rare in terms of cycles.

Hopefully, every sort of failure is rare.

Flash can just about tolerate his
machine running out of memory once a month. I'd say that if the machine runs
out of memory every day, pretty soon the system will be replaced.

Or he will stick in a new stick of RAM.

The question is, where will the failure occur? In a big allocation, or a
little allocation? With some simplifying assumptions, the answer is very
easy to calculate, and so you can work out the size of allocation that is
orders of magnitude less likely to fail than a hardware failure, that is to
says the error-handling code is vanishingly unlikely to be executed.

Several little allocations = 1 big allocation.

If you remove those simplifying assumptions the question does become a bit
more difficult. However the costs of error-handling code are not negligible.

if (foo == NULL)
{
printf("Memory allocation failure at file=%s, function=%s line=(%d)
\n", __FILE__, __FUNCTION__, (int) __LINE__);
exit(EXIT_FAILURE);
}

Is certainly better than traipsing along merrily and assuming all is
well. If something more sophisticated is wanted, then it can be
provided.

Adding code that will never be executed is classical example of
over-engineering. The answer to the OP's question is "the  point at which a
hardware failure becomes statistically more likely".

Hardware failures are detected and diagnosed. Disk drives have Reed-
Solomon (or better) error managers and business systems will typically
have RAID protection. If there is a failure, most of the time
execution is not even interrupted. On my machines, I will get a pop-
up telling me to replace a drive or rebuild part of the array.

Failure to examine function returns when it is simple to perform is
inexcusable.

IMO-YMMV.

 

[CBFalconer]

.... snip ...

If you think the odds are quite small that other applications which
are not intentionally trying to allocate all of RAM can make your
malloc() fail are quite small, then I can assure you that you are
mistaken. It is not at all uncommon for people to run lots of
applications at the same time. Sometimes the applications are long-
running applications. Sometimes these applications have resource
leaks. Memory allocation failure for a large or small memory
request is not a rare occurence. I think it is very careless to
assume that any memory allocation will succeed without testing it.

You don't need any other application. Here is a copy of an earlier
message of mine:

---- unquoted copy ---

Oh? Try the following:

#include <stdio.h>
#include <stdlib.h>

#define SZ 40

int main(void) {
unsigned long count;
void *ptr;

count = 0;
while (ptr = malloc(SZ)) count++;
printf("Failed after %lu tries\n", count);
return 0;
}

Incidentally, on my machine (a 500 Mhz Pentium running W98 FE with
512 meg of memory) this failed after roughly 10 million tries due
to the swap file filling, in about 43 seconds. The interesting
effect is that the swap file had expanded from roughly 10 megs to
about 170 megs. After the program terminated it took several
minutes for the swap file to self (or something) empty. It only
recovered to 15 meg, however.

I have not tested it under any Linux.
---- end quote --
Try the download section.

 

[CBFalconer]

CBFalconer wrote, On 21/01/08 23:19:
.... snip ...


Actually, if int is larger than size_t and overly large value will
be reduced in the usual way for conversion to unsigned leading to a
far *smaller* amount being allocated successfully than the amount
that was requested. A bug I had not thought of until now and another
good reason for not using int.

Don't worry about it. size_t is guaranteed to be able to express
any size usable in the machine.

 

[Joe Wright]

You need to be sure that you will access the memory in a way that will
cause a crash. For example, if you allocate a 10,000 element array,
and the first element you access is the 5,000th, then you may not get
an error even if malloc() returned 0.

-- Richard

Given 'char *cp = malloc(10000);' returns 0, how would you go about
addressing 'the 5,000th' element?

 

[Keith Thompson]

Given 'char *cp = malloc(10000);' returns 0, how would you go about
addressing 'the 5,000th' element?

cp[4999], of course.

Naturally this invokes undefined behavior. The point is that, on some
systems, dereferencing cp itself (where cp==NULL) is likely to trap,
whereas referring to cp[4999] might not because 4999 might be a valid
address.

 

[Flash Gordon]

CBFalconer wrote, On 22/01/08 23:08:

Don't worry about it. size_t is guaranteed to be able to express
any size usable in the machine.

I, of course, would use size_t, I was pointing out that the problem in
Malcolm's code of using int if int is larger than size_t is actually
different to what you claimed.

 

[Joseph Huber]

Incidentally, on my machine (a 500 Mhz Pentium running W98 FE with
512 meg of memory) this failed after roughly 10 million tries due
to the swap file filling, in about 43 seconds. The interesting
effect is that the swap file had expanded from roughly 10 megs to
about 170 megs. After the program terminated it took several
minutes for the swap file to self (or something) empty. It only
recovered to 15 meg, however.

I have not tested it under any Linux.

I would even go further:
A decent server/multiuser/multitasking system will not allow a
(nonprivileged) single user or process to allocate all available
physical (or virtual page/swap file) memory, because this would be a DOS
(Denial Of Service) to other users/processes.
Such a system will put limits or quotas on the memory-/pagefile-usage,
adjusted to the expected application profiles.
So the silly "I have GB of RAM - malloc will almost never fail" attitude
may be understandable for people never using anything than personal
systems, but intolerable for anybody else.

 

[Malcolm McLean]

Last I checked, assert only exists in debug mode as a rule. So in
production, you merrily ignore sizes less than zero and just pass them on
to malloc? How bizarre.

Yes.
I didn't decide the assert() strategy, but it is the one chosen by the
standard library and is reasonable for some applications, like video games,
where it is better to continue execution rather than admit programmer error.

As lot of compilers keep the asserts even in release mode. however.

 

[santosh]

Yes.
I didn't decide the assert() strategy, but it is the one chosen by the
standard library and is reasonable for some applications, like video
games, where it is better to continue execution rather than admit
programmer error.

As lot of compilers keep the asserts even in release mode. however.

Even under switches instructing strictest possible conformance? Any
examples?

 

[Kelsey Bjarnason]

Yes.
I didn't decide the assert() strategy, but it is the one chosen by the
standard library and is reasonable for some applications, like video
games, where it is better to continue execution rather than admit
programmer error.

As lot of compilers keep the asserts even in release mode. however.

And a lot don't.

So aside from writing bad code and lying to the users, you're also
relying on behaviour which simply cannot be relied upon, to handle your
validation.

Good, good. This is how you instill trust in the user, is it? This is
how you make him confident your other code is actually worth having, is
it?

 

[Richard Heathfield]

Malcolm McLean said:

Yes.
I didn't decide the assert() strategy, but it is the one chosen by the
standard library and is reasonable for some applications, like video
games, where it is better to continue execution rather than admit
programmer error.

If the programmer made an error, why not admit it? Honesty is the best
policy.

As lot of compilers keep the asserts even in release mode. however.

Implementations are required to "keep the asserts" unless NDEBUG is
defined, in which case they are required *not* to keep them. There is no
licence or flexibility there. An implementation which doesn't replace
asserts with null statements when NDEBUG is defined is a non-conforming
implementation.

 

[Malcolm McLean]

Malcolm McLean said:


If the programmer made an error, why not admit it? Honesty is the best
policy.

assert should only be triggered as a result of programmer error.
However in many video games, the only possible error is "disk dirty". The
user must close the program, remove the CD from the drive, polish it, and
restart the game. That almost always fixes the error.

Implementations are required to "keep the asserts" unless NDEBUG is
defined, in which case they are required *not* to keep them. There is no
licence or flexibility there. An implementation which doesn't replace
asserts with null statements when NDEBUG is defined is a non-conforming
implementation.

I'd have to check whether my Visual C++ 6.0 (now taken away from me by
Vista) defines NDEBUG in release mode or not. It certainly retains asserts
by default.

 

[jameskuyper]

Malcolm McLean wrote:
....

However in many video games, the only possible error is "disk dirty".

How in the world did they manage to achieve that level of coding
perfection? Are the techniques they used available for scientific
programming, or do they involve some kind of black magic? Are human
sacrifices required, or is selling your soul sufficient?

Personally, I've never seen a video game without bugs, and few if any
of those bugs were traceable to dirty disks. Maybe I'm playing the
wrong kinds of games?

 

[Richard Heathfield]

Malcolm McLean said:

assert should only be triggered as a result of programmer error.

The purpose of assertions are to test assumptions made by the programmer.
If those assumptions are false, the assertion should fire *during
development*. Every assertion in the program - and there should be a great
multitude thereof - should have at least one test case assigned to it the
purpose of which is to attempt, as valiantly as possible, to make that
assertion fire.

However in many video games, the only possible error is "disk dirty".

I disagree. There is no upper limit to the amount of possible errors in
video games. For example, my children recently discovered that they could
force Mario to get trapped in a clearly unintentional nether-world
"between" the meta-game "which magic window do you want to jump through
next?" bit and one of the actual game levels. No idiot marketroid is going
to persuade them that this is a result of a dirty disk. I understand
you've moved to academia, so I suppose there's no point in telling you to
pass this information along to *your* idiot marketroids, but the fact of
the matter is that kids are not as stupid as games merchants would like to
believe.

The
user must close the program, remove the CD from the drive, polish it, and
restart the game. That almost always fixes the error.

It sure as heck wouldn't fix this one.

I'd have to check whether my Visual C++ 6.0 (now taken away from me by
Vista) defines NDEBUG in release mode or not. It certainly retains
asserts by default.

The Standard doesn't define "release mode" or "debug mode". It only says
what behaviour implementations must provide for assert with NDEBUG
defined, and what behaviour they must provide for assert with NDEBUG not
defined. Indeed, you can #define it and #undef it yourself as often as you
like throughout your program code, if the mood should take you.

If you want NDEBUG defined, check that it is defined. If not, define it.
Don't trust an implementation's "modes" to make choices that you are
supposed to be making yourself.

 

[Richard Tobin]

The purpose of assertions are to test assumptions made by the programmer.
If those assumptions are false, the assertion should fire *during
development*. Every assertion in the program - and there should be a great
multitude thereof - should have at least one test case assigned to it the
purpose of which is to attempt, as valiantly as possible, to make that
assertion fire.

Well yes, ideally. Ideally we would find all bugs during development.
But since we don't, I'm not sure what your point is. Are you suggesting
that we should not try to detect mistaken assumptions later?

-- Richard

 

[Richard Heathfield]

Richard Tobin said:

Well yes, ideally. Ideally we would find all bugs during development.
But since we don't, I'm not sure what your point is. Are you suggesting
that we should not try to detect mistaken assumptions later?

I think of assertions as being rather like scaffolding. Whilst a building
is being built, the scaffolding is useful and indeed vital. But if the
"finished" building collapses when you take the scaffolding away, it
wasn't a very good building, was it?

Okay - all analogies are flawed, and proof by analogy is flawed.
Nevertheless, I think this one is a good illustration of my own take on
assertions. Clearly YMMV.

 

[Richard Heathfield]

Richard Heathfield said:

Okay - all analogies are flawed, and proof by analogy is
flawed.

^^^^^^

I meant "fraud" (quoting Stroustrup), not "flawed". Sorry.