A solution for the allocation failures problem

[CBFalconer]

.... snip ...

It's also possible that a failed malloc() is not fatal.

I have a function which builds an index of records in a database.
The sort routine allocates memory in chunks. As long as the first
malloc() succeeds, it is irrelevent whether later mallocs() succeed
or not. As it wants more memory, it malloc()s another chunk for
its buffers, until either it has all the memory it needs to hold
all of the sort keys, hits a configurable maximum number of chunks,
or malloc() fails. At that point, it simply uses whatever memory
it allocated. A failed malloc() simply means "don't allocate any
more chunks", and is handled no differently than "I have already
allocated the maximum number of chunks I was told to allow".

This is similar to the action of my hashlib package [1]. It an
attempt to store a new item fails because of malloc failure, the
package cleans up (after that particular store attempt) and returns
failure (and an available error code) to the application. The
application would normally tell the user, but might simply end a
phase of the processing.

[1] <http://cbfalconer.home.att.net/download/>

 

[CBFalconer]

.... snip ...

These reasons are the ones that prompted me to add try/except to
the lcc-win C compiler. This allows for reasonable error handling.

All those people telling us otherwise are just telling us PLAIN
NONSENSE.

lcc-win tries to improve C. Most of the people here just try to
keep the language as it is without even getting rid of warts
like asctime() or similar.

No, most people here try to alter the language in the appropriate
newsgroup, known as "comp.std.c". They realize that comp.lang.c
discusses the use of the actual language, as specified in the
various ISO standards and K&R, and thus do not waste time and
bandwidth with nonsensical off-topic posts here.

 

[CBFalconer]

I disagree. It is possible. You just have to decide you want to
sweat the details bad enough and do the hard work.

What hard work? For example:

if (!(p = malloc(N * sizeof *p))) fixitup(&p, N * sizeof *p);
if (!p) exit(EXIT_FAILURE);

can handle and recover from all recoverable malloc errors. Takes 2
whole lines. You can customize the fixitup operation as needed.
If you actually want the silly xmalloc action just replace
fixitup() with exit(). That is only 1 line.

 

[ymuntyan]

What hard work? For example:

if (!(p = malloc(N * sizeof *p))) fixitup(&p, N * sizeof *p);
if (!p) exit(EXIT_FAILURE);

can handle and recover from all recoverable malloc errors. Takes 2
whole lines. You can customize the fixitup operation as needed.
If you actually want the silly xmalloc action just replace
fixitup() with exit(). That is only 1 line.

Your solution is over-complicated. Here's a nicer one:

int main (void)
{
return 0;
}

You can customize main() as needed.

 

[Kelsey Bjarnason]

[snips]

This is obviously a toy example, but it looks pretty cluttered and ugly
already. It's about 62 lines of code. Now extrapolate this out to a
non-trivial application with hundreds, thousands, or even tens of
thousands of functions that allocate memory.

Now let's compare this to a language with exceptions and garbage
collection:

No. Let's compare it to a language with magic pixie dust that creates
and installs new memory chips whenever you run out of memory.

Yes, we're all quite aware that other languages have better, simpler,
cleaner, easier ways to handle errors than C does. We're also aware that
in this group, the context is C, not those languages.

In _C_, as a developer, I adopt habits which I find work. Yes, the code
to propagate errors can make the code longer, but guess what? I'll take
longer over random crashing any day, thanks.

My intention is merely to show that handling memory in C is a
non-trivial task and that it's disingenuous to claim otherwise.

Of course it's non-trivial. That wasn't _quite_ the claim. The claim
was that it's just not as difficult as some folks make it out to be,
particularly if you develop the habit of doing it regularly.

Malcolm is getting beat up for trying to solve part of the complexity
problem of managing memory in C. For a very wide range of applications,
Malcolm's xmalloc() is just fine.

Really? Which ones?

No, seriously, which ones? Have you looked at xmalloc?

It starts by using the wrong type for the parameter: size is passed as an
int.

It is apparently designed not to allow negative parameters, as there's a
test for this.

If compiled with NDEBUG defined, that test magically disappears, meaning
that xmalloc( -3 ) now attempts to process the -3.

That -3, since it isn't trapped, now merrily attempts to allocate some
18446744073709551613 bytes, with who knows what consequences.

Setting aside my developer hat for the moment, and putting on my user
hat: There are plenty of applications where I simply don't care if the
application terminates with an out of memory error.

Sure. If logrotate does this and picks up the slack on the next hourly
run, I don't care. If my word processor does it and takes my documents
with it, I'm not gonna be happy.

 

[Kelsey Bjarnason]

Pardon me.I haven't programmed for a few years but I was involved in
some fairly complex software development. Why is it not possible to
check every malloc result?

Dunno; I've been cranking code for nigh on 30 years and I don't
understand why. Others - CBF, for example - appear to be equally baffled.

 

[Malcolm McLean]

No, you're just not reading carefully.
Go back and look for the words "once the habit has been developed".

You wonder if it is pure arrogance or just naivety from the other side.

A top-quality application will have custom error-handling for every call to
malloc(). However often, but not always, that will ultimately be to exit()
the application. The alternative of aborting the operation (but not the
program) may be unfeasible. The alternative of using a less memory-hungry
algorithm is very seldom realistic.

Now what part of the word over-engieering can't you understand? many trivial
allocations have a much lower chance of failure than a hardware bug. It is
sensible to put defences in software, if the cost is trival. The cost of a
line of code is not trivial. Average productivity of tested, validated,
production code is about ten lines per programmer day. Each line represents
about a hour of a programmer's time.

You've maybe got some sort of case that often the code is boilerplate. That
tends to be true if you are writing boilerplate applications that can be
written as a simple procedural pyramid, and don't use any complex data
structures.

No. In addition to the design flaw that makes it inappropriate for
most uses, his implementation also has several rather serious errors
that that it appears he'd rather try to defend than fix.

It uses an assert() and a signed argument. There is a real case to be made
against assert(), but it is part of the standard. The signed argument
enables invalid arguments to be destected more readily. It also emphasises
that the function is not useful for extremely large allocations.

I don't care if it says "I need more memory than I can get to do what
you asked me to".
Terminating the whole thing (including discarding any unsaved state)
because one subtask couldn't allocate as big a scratch buffer as it
wanted is an entirely different matter.

So you don't actually know what xmalloc() does. By default it exits, but
this behaviour can be oveerriden to nag the users as many times as you want
for more memory. This is the best solution for a legitimate allocation in a
multi-process, interactive environment.

 

[santosh]

It uses an assert() and a signed argument. There is a real case to be
made against assert(), but it is part of the standard. The signed
argument enables invalid arguments to be destected more readily. It
also emphasises that the function is not useful for extremely large
allocations.

That's an artificial restriction that you have imposed on xmalloc().
It's supposed to be a wrapper for malloc() and should be usable for
allocations for which malloc() itself is usable, i.e., 0 -> SIZE_MAX.

So you don't actually know what xmalloc() does. By default it exits,
but this behaviour can be oveerriden to nag the users as many times as
you want for more memory. This is the best solution for a legitimate
allocation in a multi-process, interactive environment.

I would prefer a malloc() wrapper that accepted a control block that
could be used to implement a variety of recovery strategies on a per
allocation basis. This way the same function can be used where
xmalloc() type "abort-on-error" behaviour is applicable or when
recovery is necessary or plain vanilla malloc()'s behaviour is needed
or...

It's usually the case that malloc() is a relatively high overhead
function, so a complex customisable wrapper would not significantly
affect performance but it would improve code reuse.

 

[Bartc]

What hard work? For example:

if (!(p = malloc(N * sizeof *p))) fixitup(&p, N * sizeof *p);
if (!p) exit(EXIT_FAILURE);

Hmm, perhaps not quite as eloquent as:

p=heap(N);

Then you can get on with the next thing. Of course this is a made-up
language, but this form clearly has some attraction and I think trying to
achieve something similar in C was the aim of this and the xmalloc thread.

(I keep seeing this form, usually as 'new', in other languages; what do
/they/ do when out-of-memory?)

 

[Richard Bos]

That's my usual practice, too, but sometimes I'll make
multiple malloc() calls and check all the results at once:

p = malloc(N * sizeof *p);
q = malloc(N * M * sizeof *q);
if (p == NULL || q == NULL) {
free (p);
free (q);
return SORRY_DAVE;
}

The disadvantage of this approach is that if the first
malloc() fails *and* if malloc() sets errno, any information
it may have placed in errno has probably been lost by the
time the results are tested.

Only if another error occurred in the mean time. Functions which succeed
should not (IIRC are not allowed to) set errno.

Richard

 

[Malcolm McLean]

Malcolm McLean wrote:

I would prefer a malloc() wrapper that accepted a control block that
could be used to implement a variety of recovery strategies on a per
allocation basis. This way the same function can be used where
xmalloc() type "abort-on-error" behaviour is applicable or when
recovery is necessary or plain vanilla malloc()'s behaviour is needed
or...

It's usually the case that malloc() is a relatively high overhead
function, so a complex customisable wrapper would not significantly
affect performance but it would improve code reuse.

Allocators are getting a lot better at fast allocation of small amounts of
memory.
The problem is that every library function needs the xmalloc() control block
passed to it.
I did consider a scheme whereby objects could be created on a stack or on
the heap, by passing in a flag. The stack allocated objects have to be
destroyed in inverse order. However it gets rid of virutally all the
malloc() overhead. salloc() is a check, and increment, and return. I
abandoned the idea, however, simply because of the burden on caller. He
wants the results from his function call, he doesn't want to concern himself
with the memory strategy it uses to get them.

 

[Ralf Damaschke]

Functions
which succeed should not (IIRC are not allowed to) set errno.

Generally, any function may set errno to a nonzero value, even
if it succeeded.

However, no C library function may set errno to zero.

-- Ralf

 

[Kelsey Bjarnason]

[snips]

Straw man. I never advocated not checking for allocation failures and
allowing programs to crash.

Oh, so then we're agreed: the code to do this in C is exactly as easy to
read as the code to do this in C, thus making all this nonsense about it
being difficult a complete load of twaddle.

This is (as noted) not any harder to write (or read) than code without
-> error checking once the habit has been developed

Gosh, "not any harder to write (or read)". That's quite a claim!

Given that you apparently have trouble reading English, it's not
surprising you'd have trouble reading something such as C. I trimmed the
quoted part so that you can ponder how "once the habit has been
developed", being dropped from your rejoinder, makes your response look
somewhat foolish.

I
think I proved otherwise, but if you feel memory management in C is "not
any harder to write (or read)", you go on believing it, Kelsey.

Perhaps, if you learn to read, we can continue this. Until then, with
both English and C apparently beyond your ken, there's little point.

 

[Eric Sosman]

Only if another error occurred in the mean time. Functions which succeed
should not (IIRC are not allowed to) set errno.

See Question 12.14 in the comp.lang.c Frequently Asked
Questions (FAQ) list at http://www.c-faq.com/. Alternatively,
see 7.5p3:

"[...] The value of errno may be set to nonzero by a
library function call whether or not there is an error,
provided the use of errno is not documented in the
description of the function [...]"

 

[Kelsey Bjarnason]

[snips]

Mr Bjarnason is found of saying how clever he is.

I'm sure you'll post links to _any_ post in which I claim to be clever,
to back that up, right?

Whoops, no, you won't, because you're lying. Why, I'm not sure, but you
are.

verifiable here. He can claim he never makes allocation mistakes but
this sounds very hollow to me.

Where did I say that?

No, seriously, where did I say I never make allocation mistakes? That
would include, among other things, allocating more memory than is
required, less than is optimal, a whole range of cases.

I'm sure you'll post a link to any post where I said I never make any
allocation mistakes, to back up your claim, right?

Whoops, no, you won't, because you're lying. Why, I'm not sure, but you
are.

 

[Ed Jensen]

No, you're just not reading carefully.
Go back and look for the words "once the habit has been developed".

I read it just fine. Even once you "develop the habit", it's still
not zero effort to write (or read). And the extra code still
introduces additional potential for bugs.

Code that has cleanly written checks for allocation failures is easier
to read and write than code that constantly makes me ask "What if that
fails?".

Exceptions were invented and are now a popular feature of programming
languages precisely because you're wrong about that. Moving
exceptional condition processing out of the main processing makes the
code more readable.

Leaving out checks for errors GUARANTEES more bugs. I'll take the
additional code and the risk of getting it wrong over the certainty
that the shorter code is wrong, thanks.

Straw man. I never said leaving out checks for allocation failures
was appropriate. I said it was not zero effort. Try to read more
carefully.

It gets less cluttered and ugly if you use a less cluttered and ugly
error handling strategy.
For one thing, trying to back out all of your allocations so far at
every point where one of them can fail blows up pretty quickly when
more than one thing can fail.

To be honest, I'm not pleased with the solution I coded up almost
without thought *or* your solution. I find them both cluttered.

By the way, I'm not trying to accuse you of writing ugly code. What
we're seeing is just the side effect of using a language without
garbage collection or exceptions.

And to be clear, I'm not suggesting C should add garbage collection or
exceptions, either. If you care to use them, languages with those
features already exist. C is good in its niche and shouldn't try to
become one of those other languages.

How many lines of code do the "do some work" comments represent? I bet
it's more than ten or twelve, and quite possibly enough to make ten or
twelve look vanishingly small in comparison.

I don't care. This is completely unrelated to the point I was making.

One line per allocation in the part of the function that does the
interesting bits.
Error recovery code that cleanly backs out the allocations, and can
sometimes be combined with normal cleanup in functions that allocate
resources to work with instead of allocating them for higher-level
functions.

Not exactly a great cognitive burden. If I'm reading that code, the
amount of time I spend on the error recovery will probably be lost in
the noise when you put it up against the time I'll spend wrapping my
brain around how hundreds, thousands, or even tens of thousands of
functions work together.

The fact that it is a cognitive burden is well known, which is why
people experience application failures daily, including but not
limited to:

* Memory or other resource leaks.

* Segmentation faults.

* Bugs caused by wild pointers.

* Malicious input or programs that leverage these common bugs in order
to do "bad things" with your computer *or* to your computer.

But this is all well known, and to deny it is foolishness. It's
exactly why other programming languages are now popular for many
applications for which C was traditionally used.

And when you fill out the "do some work" comments, suddenly you're
comparing something like 362 vs. 337 lines of code, and for some reason
it doesn't look like such a big difference anymore.
If you're trying to simplify it using lines of code as your metric,
leaving out the error handling is Just Not Worth Your Time.

Sometimes, the smallest amount of code can cause the greatest amount
of headaches.

Well, yeah, if all you do is allocate and deallocate resources, using a
language that handle the deallocation for you means you only have to
write half as much code. That should be obvious.
If you actually do something with the resources you allocate, then
suddenly you're writing maybe 5% less code? 10%? 25% if you're doing
something trivial?
The error handling code is going to be some of the easiest bits of that
code to write, because once you've gotten into the habit of doing it,
it practically writes itself.

It might be a small amount of code, but that doesn't mean it's not
very prone to mistakes commonly made by developers.

Checking for and failing cleanly on simple resource allocation errors
instead of aborting the entire program is not exactly what I would call
"non-trivial".

Straw man. I never said aborting on allocation failure was appopriate
for all applications, but it is acceptable for some applications.

[SNIP]

The rest of your post seems to be trying to force me to defend a
stance I never took in the first place, so I'll ignore it.

 

[Ed Jensen]

Yes, we're all quite aware that other languages have better, simpler,
cleaner, easier ways to handle errors than C does. We're also aware that
in this group, the context is C, not those languages.

And I'm aware that you're aware of those things, Kelsey. I made it
very clear that my example was merely to demonstrate that memory
management in C isn't nearly as trivial as someone was making it out
to be.

In _C_, as a developer, I adopt habits which I find work. Yes, the code
to propagate errors can make the code longer, but guess what? I'll take
longer over random crashing any day, thanks.

Straw man. I never advocated not checking for allocation failures and
allowing programs to crash.

Of course it's non-trivial. That wasn't _quite_ the claim. The claim
was that it's just not as difficult as some folks make it out to be,
particularly if you develop the habit of doing it regularly.

Let's examine the exact claim:

-> Almost all of my code (well, the parts of it that allocate resources)
-> handles failure by doing "What, I can't have it? Well, then, I'd
-> better clean up after myself and report a failure up the call chain".
-> This is (as noted) not any harder to write (or read) than code without
-> error checking once the habit has been developed, and it guarantees
-> that when I do write code that has a sensible error-handling strategy,
-> all I'll need to do to decide whether that strategy needs to be
-> invoked is a single check of the result of a lower-level setup function.

Gosh, "not any harder to write (or read)". That's quite a claim! I
think I proved otherwise, but if you feel memory management in C is
"not any harder to write (or read)", you go on believing it, Kelsey.

You're simply far too obnoxious and argumentative for me to have any
desire to follow up to any further replies you have on this subject,
so feel free to get in the last word.

 

[jacob navia]

And to be clear, I'm not suggesting C should add garbage collection or
exceptions, either. If you care to use them, languages with those
features already exist. C is good in its niche and shouldn't try to
become one of those other languages.

C can use garbage collection and exceptions. For an implementation of
those features in C see the lcc-win compiler system (see URL below)

[snip]

The fact that it is a cognitive burden is well known, which is why
people experience application failures daily, including but not
limited to:

* Memory or other resource leaks.

* Segmentation faults.

* Bugs caused by wild pointers.

* Malicious input or programs that leverage these common bugs in order
to do "bad things" with your computer *or* to your computer.

But this is all well known, and to deny it is foolishness.

This is well known to all but the "regulars" here, that will deny
and fight any improvement of C. But there are a lot of users
that like the idea of a simple language that has all
the essential features.

 

[Keith Thompson]

Hmm, perhaps not quite as eloquent as:

p=heap(N);

Then you can get on with the next thing. Of course this is a made-up
language, but this form clearly has some attraction and I think trying to
achieve something similar in C was the aim of this and the xmalloc thread.

Changing the syntax for memory allocation doesn't magically solve
anything.

(I keep seeing this form, usually as 'new', in other languages; what do
/they/ do when out-of-memory?)

It varies from language to language. Typically either the allocator
returns a null pointer on failure (exactly what C's malloc() does), or
it throws/raises an exception (if the language supports exceptions).

It's *always* possible for an attempted memory allocation to fail.
The possible ways to handle this possibility are:

1. Ignore it and keep going (dangerous).
2. Use an allocator that immediately aborts the program on failure.
3. Check for failure on each allocation. On failure:
3a. Immediately abort the program (equivalent to 2).
3b. Clean up and abort the program.
3c. Clean up and continue processing (this can be difficult, but
it's important if you want the program to be robust).
4. If the language supports exceptions (C doesn't), catch the error at
whatever level is appropriate, not necessarily immediately after
the attempted allocation. Perform appropriate cleanup and recovery
in the exception handler.

 

[jacob navia]

Let's examine the exact claim:

-> Almost all of my code (well, the parts of it that allocate resources)
-> handles failure by doing "What, I can't have it? Well, then, I'd
-> better clean up after myself and report a failure up the call chain".
-> This is (as noted) not any harder to write (or read) than code without
-> error checking once the habit has been developed, and it guarantees
-> that when I do write code that has a sensible error-handling strategy,
-> all I'll need to do to decide whether that strategy needs to be
-> invoked is a single check of the result of a lower-level setup function.

Gosh, "not any harder to write (or read)". That's quite a claim! I
think I proved otherwise, but if you feel memory management in C is
"not any harder to write (or read)", you go on believing it, Kelsey.

You're simply far too obnoxious and argumentative for me to have any
desire to follow up to any further replies you have on this subject,
so feel free to get in the last word.

Mr Bjarnason is found of saying how clever he is. This is of course
not verifiable here. He can claim he never makes allocation mistakes
but this sounds very hollow to me. Like other claims done by the
"regulars" here, they sound pathetic, like the one of the famous
Dan Pop that claimed that he never had a crash of a program written by
him.

Boasting here is very easy.