E
ediebur
jacob said:
Pardon me.I haven't programmed for a few years but I was involved in
some fairly complex software development. Why is it not possible to
check every malloc result?
Pardon me.I haven't programmed for a few years but I was involved in
some fairly complex software development. Why is it not possible to
check every malloc result?
M
Martien Verbruggen
Why not?
Martien
C
CBFalconer
Don't believe everything you find written here.
S
Stan Milam
jacob said:
I disagree. It is possible. You just have to decide you want to sweat
the details bad enough and do the hard work.
Regards,
Stan Milam
K
Kelsey Bjarnason
[snips]
I think that's the key to it. I keep hearing how it's so hard to check
such things, the code is messy, yadda yadda yadda, yet it's not. Why,
then, do some of us do this and find it relatively straightforward, while
some don't?
I suspect it's because we developed habits - design habits - which
incorporate such checking from the outset, while others don't. We, by
sheer habit, write functions such that they are capable of reporting
failure to their callers and, also by sheer habit, write callers which
check for those errors.
If we were to write code where the called function _could_ fail, but the
caller didn't bother to check for it, we would be faced with three
choices:
1) Just accept our code will fail in unpredictable ways
2) Try to go back and find every place such things can fail and bolt on
the error handling code after the fact
3) Do something like xmalloc - if the allocation fails, just abort
I suspect this is exactly what's happening - that some folks are writing
the code *without* the error handling, then going back in to add it on
later. Obviously, this will mean it's nigh-on impossible to ensure, in a
complex application, that every case is checked. Obviously, it's going
to make things ugly. Obviously, it's going to be a whole heck of a lot
of work.
I don't know that this is what they're doing, but it sounds like it. It
would explain why they think something like xmalloc makes sense, why they
think that "checking every allocation" is so hard to ensure.
It really is just a matter of design, but even simpler, just a matter of
habit. If you're writing code which opens a file or allocates memory, or
some similar operation which can fail in a manner which you can both
detect and deal with, add the code to check it. Now.
It's a habit one can develop, and once it becomes habit, it is very
little effort indeed to ensure you do, in fact, check such things - each
and every time.
I think that's the key to it. I keep hearing how it's so hard to check
such things, the code is messy, yadda yadda yadda, yet it's not. Why,
then, do some of us do this and find it relatively straightforward, while
some don't?
I suspect it's because we developed habits - design habits - which
incorporate such checking from the outset, while others don't. We, by
sheer habit, write functions such that they are capable of reporting
failure to their callers and, also by sheer habit, write callers which
check for those errors.
If we were to write code where the called function _could_ fail, but the
caller didn't bother to check for it, we would be faced with three
choices:
1) Just accept our code will fail in unpredictable ways
2) Try to go back and find every place such things can fail and bolt on
the error handling code after the fact
3) Do something like xmalloc - if the allocation fails, just abort
I suspect this is exactly what's happening - that some folks are writing
the code *without* the error handling, then going back in to add it on
later. Obviously, this will mean it's nigh-on impossible to ensure, in a
complex application, that every case is checked. Obviously, it's going
to make things ugly. Obviously, it's going to be a whole heck of a lot
of work.
I don't know that this is what they're doing, but it sounds like it. It
would explain why they think something like xmalloc makes sense, why they
think that "checking every allocation" is so hard to ensure.
It really is just a matter of design, but even simpler, just a matter of
habit. If you're writing code which opens a file or allocates memory, or
some similar operation which can fail in a manner which you can both
detect and deal with, add the code to check it. Now.
It's a habit one can develop, and once it becomes habit, it is very
little effort indeed to ensure you do, in fact, check such things - each
and every time.
K
Kelsey Bjarnason
I already asked why once, but I'll ask again - WHY is it not possible?
It's actually very simple. When I write code which calls malloc - that
is, *immediately after writing the call to malloc* - the very next thing
I do is write the code which checks whether it failed.
ptr = malloc(x);
if ( ! ptr ) { handle }
It's a simple habit. If you develop the habit, it's not merely possible,
but trivial.
Nor is it limited to malloc:
MyWindow *win = createWindow();
if ( ! win ) { handle }
Do I know whether createWindow calls malloc? No. Nor do I care. What I
care about is that it can fail, and on failure, returns NULL. Nor is
this limited to allocations:
int sock = socket(...);
if ( sock == -1 ) { handle }
c = fgetc(fp);
if ( c == EOF ) { handle }
Why is it more difficult to check for allocation failures than to check
that you got a legitimate (non-EOF) character from a file, or a
legitimate socket handle, or any other case where the value returned may
be an indicator of a failure state?
K
Kelsey Bjarnason
Kinda the point to the whole discussion. The tools - xmalloc et al -
treat failure as fatal, when it often isn't, and even if it is, *I* - the
developer - want to decide what to do about it.
Bingo. It is *not* a situation where the only possible response is to
scream and die.
K
Kenneth Brody
Kelsey said:
It's also possible that a failed malloc() is not fatal.
I have a function which builds an index of records in a database.
The sort routine allocates memory in chunks. As long as the first
malloc() succeeds, it is irrelevent whether later mallocs() succeed
or not. As it wants more memory, it malloc()s another chunk for
its buffers, until either it has all the memory it needs to hold
all of the sort keys, hits a configurable maximum number of chunks,
or malloc() fails. At that point, it simply uses whatever memory
it allocated. A failed malloc() simply means "don't allocate any
more chunks", and is handled no differently than "I have already
allocated the maximum number of chunks I was told to allow".
[...]
--
+-------------------------+--------------------+-----------------------+
| Kenneth J. Brody | www.hvcomputer.com | #include |
| kenbrody/at\spamcop.net | www.fptech.com | <std_disclaimer.h> |
+-------------------------+--------------------+-----------------------+
Don't e-mail me at: <mailto:[email protected]>
K
Kelsey Bjarnason
[snips]
Right. If there absolutely is nothing else you can do, you have to
abort. Sometimes - rarely, but sometimes - that is, in fact, the only
sensible _first_ thing to do. It should not be the first thing as a
matter of policy or habit, though; if anything it should be the last.
As you describe it - "I won't have anything to do but abort anyways" -
there's little point in your trying, as you wouldn't buy anything by it;
it makes sense in such a case. As a habit, policy or general design
decision? No.
Right. If there absolutely is nothing else you can do, you have to
abort. Sometimes - rarely, but sometimes - that is, in fact, the only
sensible _first_ thing to do. It should not be the first thing as a
matter of policy or habit, though; if anything it should be the last.
As you describe it - "I won't have anything to do but abort anyways" -
there's little point in your trying, as you wouldn't buy anything by it;
it makes sense in such a case. As a habit, policy or general design
decision? No.
M
Morris Dovey
William said:
Sometimes yes and sometimes no, depending on the work being done.
Allocation failure in one application I developed required
(automagical) powering up another network node to run another
instance of the application, and moving a portion of the
unexpectedly large body of information to the new node without
significantly impacting throughput. The recovery process was
complicated by the need to implement the reverse when loading
shrank.
/Sometimes/ it's not difficult.
E
Eric Sosman
Kelsey said:
That's my usual practice, too, but sometimes I'll make
multiple malloc() calls and check all the results at once:
p = malloc(N * sizeof *p);
q = malloc(N * M * sizeof *q);
if (p == NULL || q == NULL) {
free (p);
free (q);
return SORRY_DAVE;
}
The disadvantage of this approach is that if the first
malloc() fails *and* if malloc() sets errno, any information
it may have placed in errno has probably been lost by the
time the results are tested. In my limited experience, few
malloc() implementations set errno to describe the failure
(usually if I call perror() after a failed malloc() I get
"No error"), so the disadvantage isn't crippling.
M
Malcolm McLean
Maybe its because the programs you write have certain characteristics.Kelsey Bjarnason said:
I've only written xmalloc() in the past few months, and it was designed for
Baby X. However not using some sort of central error-handling for out of
memory conditions would entail that essentially every function - the IDE
consists of function pointers strung together with dynamically-allocated
structures - would have to both accept and report null out of memory
conditions. These are not just functions within the library, these are user
functions the library user is calling. Every window he creates can fail.
Every time he wants to change the text in a window that can fail. It's just
an unreasonable burden. Especially since the whole point of the toolkit is
that it is easy to use. What can you do if a window doesn't come up,
realistically? Ask the user to close something down to give you more memory,
or kill the app. That;s what Baby X does.
P
Paul Hsieh
I'm not sure what you mean by this. Are you saying that large complex
projects will inevitably have bugs of not checking? Or are you saying
that this is a literal impossibility?
One could, for example, change the API for malloc to be something
like:
void * safemalloc(label,size_t sz);
which would jump to label if the allocation failed. Of course there
isn't a way to do this in standard C either with the pre-processor or
by function declarations. But some other sort of pre-processor, or
LINT check could do the equivalent of this.
I think a case could be made for extending the language standard to
include the above.
In any event, the *standard* approach of forcing your programmers to
painstakingly do the equivalent of this is still possible, and
essentially mandatory in C.
Well, fully general GC is not possible in the C language, because of
the way the language is. However, GC as a new feature added to the
language, is certainly possible and *useful* as you and Hans Boehm
know full well.
In most situations errors should be propagated *up* the call stack,
not downward. (*cough* idiots who proposed/endorsed TR 24731 when C
does not have exception handling *cough*)
Ok, I see what you are doing here, but this is a desperate strategy
that I don't think fully works as well as you are hoping.
The above 3 lines of code would *HAVE* to be in a critical section.
(You need some kind of OS lock.) As I have said before, and I will
say again: Memory allocation in C is all about multi-threading.
The above, I think, is a poor decision. You've just freed your big
block; if the guy didn't set the mallocfailedHandler, you should just
go ahead and give the guy his memory, to let him be compatible with
his prior behavior.
What *I* would do is if the mallocfailedHandler() returned with some
particular return value (like < 0) then do the failure, abort, or
whatever. Otherwise go to restart, and if you still can't get memory,
then return NULL.
But that's with the idea of making a malloc() compatible function
rather than something to try to save xmalloc() (which I don't think is
doable.)
Yeah ok, except what you are trying to do, in the end is save
Malcolm's xmalloc() idea. But I don't think it can be saved. What
one should take as a lesson from all this is the *desire* and the
*direction* you have tried to go to avoid "out of memory" scenarios
and try to apply them to the original C semantics.
I propose the following:
size_t maxAllocAvail();
size_t totFreeAvail();
So that the call site can estimate the amount of memory available to
its small objects as maxAllocAvail() + totFreeAvail(), and the maximum
size of its large allocations as maxAllocAvail(). This can allow the
developer to choose different strategies depending on available
memory. Then we add:
enum incident {
MEM_INCIDENT_SYSTEM_ALLOC, /* sbrk() or VirtualAlloc() is
called */
MEM_INCIDENT_SYSTEM_FREE, /* returning memory to the OS */
MEM_INCIDENT_DEPLETED_FREE_LIST, /* The next alloc will need
OS mem */
MEM_INCIDENT_LAST_ALLOC_GUARD, /* Prediction that no new
allocations are possible after this */
MEM_INCIDENT_OUT_OF_MEMORY /* We're out! We have to return
NULL now */
};
int (* allocEvent) (void * ctx, enum incident, size_t);
int setAllocEvent (void * ctx, allocEvent, enum incident);
So this sets up a callback function that is called in the event of
each kind of memory condition during a memory allocation or free.
Your pre-allocation idea then, would be reduced to simply being the
MEM_INCIDENT_LAST_ALLOC_GUARD event, after which the application could
make some kind of decision about future memory use (knowing it will
likely not be able to get it in subsequent calls to malloc.)
MEM_INCIDENT_OUT_OF_MEMORY would be there so that Malcolm and people
who believe that memory should be dealt with in the strategies he
suggests could also be satisfied (by simply aborting on this event, or
if you happen to be really running a C++ compiler, you might consider
*throw()*ing here).
The point of all this being, that there is clearly lots of room for
improvement of the C standard with respect to memory allocation.
Personally, I lean more towards better "analyze what you've got" kind
of extensions. The above is a compromised between what I see people
are posting here and what I have in mind.
Either way, we have to keep in mind that none of these ideas
*guarantees* anything in a multi-threaded environment.
D
dj3vande
Kelsey Bjarnason said:
Seconded.
Almost all of my code (well, the parts of it that allocate resources)
handles failure by doing "What, I can't have it? Well, then, I'd
better clean up after myself and report a failure up the call chain".
This is (as noted) not any harder to write (or read) than code without
error checking once the habit has been developed, and it guarantees
that when I do write code that has a sensible error-handling strategy,
all I'll need to do to decide whether that strategy needs to be invoked
is a single check of the result of a lower-level setup function.
That said, I'm currently writing a code generator that uses xmalloc-
style wrappers. But that's a design decision ("I won't have anything
to do but abort anyways, so I might as well do it at the bottom instead
of at the top"), not ignorance of the issues ("Error recovery is Hard,
so I won't bother").
dave
J
jacob navia
Paul said:
There are many situations when you can't add at each level a complex
unwinding code to take care of an allocation failure in the middle of
the construction of a complex data structure. It is much more sensible
to set an a priori strategy and not check at each call.
No. lcc-win implements try/catch, but of course the regulars here will
start crying "heresy heresy" so I did not mention that.
[snip]
It is one way of trying to cope with this. The same strategy is
implemented for the stack under windows. You get a stack overflow
exception with one LAST page still free to be allocated for
the stack. You can then still call some functions and you have
a stack of 4096 bytes reserved for this purpose. This is the
same strategy.
This is just a model OBVIOUSLY. No multi-threading considerations are in
this code. It is just an outline of how this could be solved.
This is a good point.
It must implement a counter so that it doesn't get into an infinite loop.
This are quite reasonable specs. I will try to think them over
and maybe implement them in the lcc-win library.
Thanks for your time Paul.
E
Ed Jensen
I think you're being dishonest (or perhaps naive). It's not zero
effort to write error checking code, and it's not zero effort to read
code with lots of extra error checking cluttering up the code.
Consider a scenario in which A() calls B(), B() calls C(), C() calls
D(), and D() calls E(). At each level, memory allocations may occur.
That means, in every function, code must be explicitly written to
check if any given allocation failed, and if it has, to pass that
information to the caller.
More code is usually harder to write.
More code is usually harder to read.
More code usually introduces the potential for more bugs.
In any case, let's see how this scenario might look in C:
int A(void) {
void *a = malloc(...);
if (!a) { goto OutOfMemoryError; }
/* do some work */
if (B() == ENOMEM) {
free(a);
goto OutOfMemoryError;
}
/* do some more work */
free(a);
return SUCCESS;
OutOfMemoryError:
/* handle it */
}
int B(void) {
void *b = malloc(...);
if (!b) { return ENOMEM; };
/* do some work */
if (C() == ENOMEM) {
free(b);
return ENOMEM;
}
/* do some more work */
free(b);
return SUCCESS;
}
int C(void) {
void *c = malloc(...);
if (!c) { return ENOMEM; };
/* do some work */
if (D() == ENOMEM) {
free(c);
return ENOMEM;
}
/* do some more work */
free(c);
return SUCCESS;
}
int D(void) {
void *d = malloc(...);
if (!d) { return ENOMEM; };
/* do some work */
if (E() == ENOMEM) {
free(d);
return ENOMEM;
}
/* do some more work */
free(d);
return SUCCESS;
}
int E(void) {
void *e = malloc(...);
if (!e) { return ENOMEM; };
/* do some work */
free(e);
return SUCCESS;
}
This is obviously a toy example, but it looks pretty cluttered and
ugly already. It's about 62 lines of code. Now extrapolate this out
to a non-trivial application with hundreds, thousands, or even tens of
thousands of functions that allocate memory.
Now let's compare this to a language with exceptions and garbage
collection:
void A() {
try {
A a = new ...;
/* do some work */
B();
/* do some more work */
}
catch (OutOfMemoryError e) {
/* handle it */
}
}
void B() {
B b = new ...;
/* do some work */
C();
/* do some more work */
}
void C() {
C c = new ...;
/* do some work */
D();
/* do some more work */
}
void D() {
D d = new ...;
/* do some work */
E();
/* do some more work */
}
void E() {
E e = new ...;
/* do some work */
}
This is about 37 lines of code. It was easier to write (after all,
there's a lot less code), it's easier to read (less visual clutter),
and it's more reliable/dependable (less code usually means less
opportunities for bugs).
Before you all jump down my throat, my intention is not to turn this
into a "C vs. Some Other Language" war. I recognize that languages
with exceptions and garbage collection have their own sets of pros and
cons. I also recognize that very explicit memory management is
appropriate for some applications.
My intention is merely to show that handling memory in C is a
non-trivial task and that it's disingenuous to claim otherwise.
Malcolm is getting beat up for trying to solve part of the complexity
problem of managing memory in C. For a very wide range of
applications, Malcolm's xmalloc() is just fine. He's also been
forthcoming by admitting it's not appropriate for all uses.
Setting aside my developer hat for the moment, and putting on my user
hat: There are plenty of applications where I simply don't care if the
application terminates with an out of memory error. For this set of
applications, I'd rather the developers add features, rather than
apply obscene levels of Obsessive Compulsive Disorder to every single
allocation in the entire system.
Y
ymuntyan
[snip]
Welcome to comp.lang.c, the place where it is easy to handle
memory. Just a matter of habit really.
And welcome to the club of those who do not care about
user data.
Anyway, well said!
Yevgen
Welcome to comp.lang.c, the place where it is easy to handle
memory. Just a matter of habit really.
And welcome to the club of those who do not care about
user data.
Anyway, well said!
Yevgen
J
jacob navia
Ed said:
These reasons are the ones that prompted me to add try/except to
the lcc-win C compiler. This allows for reasonable error handling.
All those people telling us otherwise are just telling us PLAIN
NONSENSE.
lcc-win tries to improve C. Most of the people here just try to
keep the language as it is without even getting rid of warts
like asctime() or similar.
K
Keith Thompson
jacob navia said:
You seem to be under the impression that the people who post here have
the ability to fix or get rid of asctime(). We don't.
But you know that.
D
dj3vande
No, you're just not reading carefully.
Go back and look for the words "once the habit has been developed".
Code that has cleanly written checks for allocation failures is easier
to read and write than code that constantly makes me ask "What if that
fails?".
Leaving out checks for errors GUARANTEES more bugs. I'll take the
additional code and the risk of getting it wrong over the certainty
that the shorter code is wrong, thanks.
int A(void)
{
void *a=malloc(size_of_a);
if(!a) goto FAIL_FIRST_MALLOC;
/*do some work*/
if(B()==ENOMEM) goto FAIL_B_FAILED;
free(a);
return SUCCESS;
FAIL_B_FAILED:
free(a);
FAIL_FIRST_MALLOC:
/*Handle failure*/
}
int B(void)
{
void *b=malloc(size_of_b);
if(!b) goto FAIL_FIRST_MALLOC;
/*do some work*/
if(C()==ENOMEM) goto FAIL_C_FAILED;
/*do some more work*/
free(b);
return SUCCESS;
FAIL_C_FAILED:
free(b);
FAIL_FIRST_MALLOC:
return ENOMEM;
}
Or maybe:
int B(void)
{
int ret=SUCCESS;
void *b=malloc(size_of_b);
if(!b) { ret=ENOMEM; goto FAIL_FIRST_MALLOC; }
/*do some work*/
if(C()==ENOMEM) { ret=ENOMEM; goto FAIL_C_FAILED; }
/*do some more work*/
FAIL_C_FAILED:
free(b);
FAIL_FIRST_MALLOC:
return ret;
}
[snip C(),D(),E() isomorphic to B()]
It gets less cluttered and ugly if you use a less cluttered and ugly
error handling strategy.
For one thing, trying to back out all of your allocations so far at
every point where one of them can fail blows up pretty quickly when
more than one thing can fail.
I count ten or twelve non-blank, non-comment lines in each of A() and
B() (both yours and mine).
If the error handling is done sensibly, it generates one additional
line of code in the main flow of control per check. Hardly "cluttered
and ugly". The actual handling of errors is done at the end of the
function, after the main flow of control returns, where you can decide
whether it's important to look at it or not. It's also easy to verify
that resources are only cleaned up on failure if they were allocated
before the failure occurred.
(This model obviously breaks when your error recovery strategy is more
than just "clean up and push the failure up the call stack", but that's
not the case under discussion here.)
How many lines of code do the "do some work" comments represent? I bet
it's more than ten or twelve, and quite possibly enough to make ten or
twelve look vanishingly small in comparison.
One line per allocation in the part of the function that does the
interesting bits.
Error recovery code that cleanly backs out the allocations, and can
sometimes be combined with normal cleanup in functions that allocate
resources to work with instead of allocating them for higher-level
functions.
Not exactly a great cognitive burden. If I'm reading that code, the
amount of time I spend on the error recovery will probably be lost in
the noise when you put it up against the time I'll spend wrapping my
brain around how hundreds, thousands, or even tens of thousands of
functions work together.
And when you fill out the "do some work" comments, suddenly you're
comparing something like 362 vs. 337 lines of code, and for some reason
it doesn't look like such a big difference anymore.
If you're trying to simplify it using lines of code as your metric,
leaving out the error handling is Just Not Worth Your Time.
Well, yeah, if all you do is allocate and deallocate resources, using a
language that handle the deallocation for you means you only have to
write half as much code. That should be obvious.
If you actually do something with the resources you allocate, then
suddenly you're writing maybe 5% less code? 10%? 25% if you're doing
something trivial?
The error handling code is going to be some of the easiest bits of that
code to write, because once you've gotten into the habit of doing it,
it practically writes itself.
Checking for and failing cleanly on simple resource allocation errors
instead of aborting the entire program is not exactly what I would call
"non-trivial".
Nobody is claiming that a partial recovery from a resource failure at
the level where it makes sense to make error-handling decisions is
trivial, but that's not what we're talking about here, and that's no
more nontrivial in C than it is in a language with GC and exceptions.
(If anything, having fewer unknowns in C might make it easier.)
No, he's "getting beat up" because his proposed solution introduces
more problems than it solves.
No. In addition to the design flaw that makes it inappropriate for
most uses, his implementation also has several rather serious errors
that that it appears he'd rather try to defend than fix.
I don't care if it says "I need more memory than I can get to do what
you asked me to".
Terminating the whole thing (including discarding any unsaved state)
because one subtask couldn't allocate as big a scratch buffer as it
wanted is an entirely different matter.
I would rather the developers of ANY software get the features they
already have right before they add new features.
If you really think that "more b0rken features" is preferable over
"stuff that actually works", I think we have nothing further to
discuss.
dave