Cleanup patterns

[Keith Thompson]

Remember that this subthread was prompted by someone who suggested it's a
waste of time to check malloc's return value at all! Such people never ever
run out of memory, no matter how many Gigabytes of emergency reserve they
allocate. Their programs crash randomly sometimes for no apparent reason,
but at least they never run out of memory!

But the person who made this statement presented, as an alternative to
checking malloc's return value, a wrapper function that checks
malloc's return value. (Actually he presented a URL for such a
function.) Some posters here have advocated simply ignoring malloc()
failures; the poster in this thread (sorry, I don't remember who it
was) did not.

 

[Keith Thompson]

True. Sometimes the canary dies even though the air in
the mine is still just barely breathable.

Right. But a more apt analogy here is that you bring a *really big*
canary into the mine with you. If the miners start to have trouble
breathing, you kill the canary and release the oxygen from its
bloodstream into the air, giving the miners enough time to exit
gracefully. (Ok, maybe it's just an inflatable plastic canary.) If
the miners were just about to leave anyway, bringing the canary was a
waste of time -- just as any insurance policy that doesn't pay off is
a waste of money. (Life insurance is a really bad deal if you happen
to be immortal.)

 

[CBFalconer]

.... snip ...
waste of time -- just as any insurance policy that doesn't pay off
is a waste of money. (Life insurance is a really bad deal if you
happen to be immortal.)

Now I know why I haven't had any for the past 20 years (Apart
from the Social Security death 'benefit').

 

[Richard Bos]

And what is your program supposed to do when memory is exhausted?

Depends entirely on the situation. If your word processor asks for a
large amount of memory to paste a graphic, and it doesn't get it,
putting up a message saying "sorry, no memory for graphic" is a lot more
civilised than just bombing out of the entire document. Even in the
worst case, putting up a meaningful message and exiting gracefully is
always better than crashing and burning.

Richard

 

[Richard Bos]

Isn't there an external mergesort?

Yes. But they're typically a lot slower than internal sorts, so you
don't use it until you _know_ that you need it. Hence the check on
malloc().

Richard

 

[Richard Bos]

Those are mostly hints how to reduce the dynamic memory consumption of
a program but don't answer the question. OOM usually means that you
must terminate the application. The simplest way is a xmalloc-like
function. If enough context is available some cleanup can be done
before.

The problem with xmalloc()-like functions is that they crash anyway;
they just crash with a predictable message. This _may_ be good enough,
but IMO not nearly as often as I see it done in the wild.

Richard

 

[Richard Heathfield]

Keith Thompson said:

But the person who made this statement presented, as an alternative to
checking malloc's return value, a wrapper function that checks
malloc's return value.

He did? Oh, so he did. My apologies to Mr Pilinger. Nevertheless, whilst
"detect and die" does at least beat "ignore and hope", it remains a
neophyte solution - acceptable from, say, a first-year student. I'd expect
better from a second-year.

 

[Richard Bos]

Well, it's so simple that most programmers will write their own
versions of it, if need be. The code seems to use the old style
function definition and uses unsigned instead of size_t, which isn't
the way it's done now. And calling exit() with a random value is not
portable. It's better to use EXIT_FAILURE.

Not to mention that the particular random value used in this xmalloc is,
erm, rather silly, given the comment above. Don't _all_ odd values mean
some kind of success on VMS?

Richard

 

[slebetman]

I think these are actually the three rules for pontificating about
programming instead of actually programming...years ago I read
the REAL rules for writing "correct" "fast" and "efficient" code, and
of course, as everybody who's not just pontificating knows,
it boils down to one general rule: pick the two you really want, because
a lot of times you CAN'T have all three...

I certainly wasn't talking about anything like "micro-optimization".
I'm talking about great big gobs of "MACRO-optimization", like
slowing down or speeding up what was virtually the IDENTICAL
code by a factor of up to ten-fold (I could make the same program run
in either two seconds or 20 seconds)...

On a lot of modern CPUs the difference between a function call and a
jump (as in generated by switch or if) is nowhere near ten-fold when
compiled with an properly optimising compiler. One fold at most (note
that one-fold is ten times slower) or more typically up to 4 times
slower. Indeed on at least two architectures I code for, a function
call (if not recursive) compiles to execute in exactly the same number
of CPU cycles as a regular jump/branch.

This is not exactly an academic exercise for me, since a lot of the
code I work with takes several HOURS to run on a Pentium-class
machine, and I have to run it EVERY day.

Of course, if you're programming for an architecture as archaic and
register-starved as the Pentium then function call overhead can be an
issue. Even so, improvements have been made to x86-64 which makes the
function call overhead for x86-64 even less than before.

 

[Roland Pibinger]

The problem with xmalloc()-like functions is that they crash anyway;
they just crash with a predictable message.

They don't crash the program, they call exit (which also calls
atexit).

This _may_ be good enough,
but IMO not nearly as often as I see it done in the wild.

In order to run your program primarily needs one resource, memory.
When it runs out of memory (e.g. due to a memory leak) there is hardly
anything you can do except to (more or less) gracefully terminate the
program. Not even that is always possible since some OS (IIRC, Linux)
never return NULL for malloc even when memory is exhausted.

Best wishes,
Roland Pibinger

 

[Ian Collins]

This is not exactly an academic exercise for me, since a lot of the
code I work with takes several HOURS to run on a Pentium-class
machine, and I have to run it EVERY day.

Maybe it's time for a 64 bit upgrade!

 

[Keith Thompson]

They don't crash the program, they call exit (which also calls
atexit).


In order to run your program primarily needs one resource, memory.

Along with a number of other resources.

When it runs out of memory (e.g. due to a memory leak) there is hardly
anything you can do except to (more or less) gracefully terminate the
program.

That's not always true; it depends on the application. Somebody
already posted an example of an interactive editor failing to allocate
enough memory to load an image; the proper response is to issue a
diagnostic and continue running, not to crash the program.

Not even that is always possible since some OS (IIRC, Linux)
never return NULL for malloc even when memory is exhausted.

That behavior is arguably non-conforming.

 

[santosh]

Along with a number of other resources.


That's not always true; it depends on the application. Somebody
already posted an example of an interactive editor failing to allocate
enough memory to load an image; the proper response is to issue a
diagnostic and continue running, not to crash the program.


That behavior is arguably non-conforming.

To what standard? Linux's behaviour is outside the scope of the C
standard. Nevertheless, by never returning NULL, even when memory is
not available, it does make the malloc() implementation non-conforming.

Actually since from within the standard C program you cannot find out
_if_ memory is available or not, malloc()'s behaviour under OSes like
Linux is not non-conforming. It's doing it's job correctly, it just so
happens that it's subverted by forces outside it's control, (and the
standard's.)

 

[Richard Heathfield]

Keith Thompson said:

That behavior is arguably non-conforming.

I would have no doubt about that - it's non-conforming. What I would
question is the OP's claim that Linux over-commits. It is true that it can
be configured to over-commit, but certainly my installation is not
configured that way by default.

 

[Richard Heathfield]

santosh said:

To what standard?

ISO/IEC 9899. Some people view the operating system as being part of the
implementation. This is a not unreasonable view, although it's a bit hard
on compiler-writers if they have no control over the behaviour of the OS on
their platform...

 

[santosh]

santosh said:


ISO/IEC 9899. Some people view the operating system as being part of the
implementation.

<snip>

Yes well, in that case we can add the hardware and the rest of the
physical universe as part of the implementation too.

 

[Keith Thompson]

<snip>

Yes well, in that case we can add the hardware and the rest of the
physical universe as part of the implementation too.

Yes.

An implementation that doesn't meet the requirements of the standard
is non-conforming. Having a good excuse for the non-conformance
doesn't make it conforming.

 

[CBFalconer]

Yes. But they're typically a lot slower than internal sorts, so you
don't use it until you _know_ that you need it. Hence the check on
malloc().

The relative slowness is because the external sort is primarily
dependant on the i/o processing. You can get remarkable
performance by minimizing that i/o. See Wirths "Algorithms + Data
Structures = Programs" for polyphase sort. Triggering virtual
memory operation will foul that up quite nicely.

 

[CBFalconer]

They don't crash the program, they call exit (which also calls
atexit).


In order to run your program primarily needs one resource, memory.
When it runs out of memory (e.g. due to a memory leak) there is
hardly anything you can do except to (more or less) gracefully
terminate the program. Not even that is always possible since some
OS (IIRC, Linux) never return NULL for malloc even when memory is
exhausted.

Not so. I gave an example (external sort) upthread. To misquote
Dan Pop, engage brain before aborting.

 

[CBFalconer]

To what standard? Linux's behaviour is outside the scope of the C
standard. Nevertheless, by never returning NULL, even when memory
is not available, it does make the malloc() implementation
non-conforming.

Actually since from within the standard C program you cannot find
out _if_ memory is available or not, malloc()'s behaviour under
OSes like Linux is not non-conforming. It's doing it's job
correctly, it just so happens that it's subverted by forces
outside it's control, (and the standard's.)

Again, not necessarily. There is no speed requirement, so an OS
can simply postpone program execution until the memory is actually
available. Implementing this could run into the deadly embrace
problem.