E
Eric
Hi What is a good way to respond to a failed Malloc (returns Null)? I do
not want to call Abort or Exit immediately.
Thanks
not want to call Abort or Exit immediately.
Thanks
X
Xavier Roche
Le 12/09/2012 22:07, Eric a écrit :
IMHO, except for large allocations, failure to allocate can lead to abort()
In many cases, if you fail to allocate few bytes with malloc(),
continuing is anyway difficult as the next new operator or guard page
stack allocation will also lead to road end.
IMHO, except for large allocations, failure to allocate can lead to abort()
In many cases, if you fail to allocate few bytes with malloc(),
continuing is anyway difficult as the next new operator or guard page
stack allocation will also lead to road end.
A
Andrew Cooper
Well - you should probably try to flush any files which are open, return
eternal changes back to defaults (changes to the terminal etc), and find
a way of letting your user see, or at least work out why they no longer
have a running process.
Failing gracefully is a lot more effort than you might imagine at first.
~Andrew
E
Eric
Sorry maybe I wasn't too clear. I would like to get the Malloc call to
succeed if I redo it. Or get some alternative memory eg from the stack or
swop.
P
Paul N
If your program has an alternative way of working which uses less
memory (say, for example, a slower method) you could put in a new
malloc asking for less memory and see if that works. Or you could
allocate more memory than you need early in the program, so you've got
a bit already to hand when you run a crucial bit of the program. Or
you could tell the user that unfortunately the program can't do what
was requested.
Getting more memory from the stack seems a little risky, as there are
normally less checks whether it worked. Swapping your own program's
data out to disk might be a possibility if you keep track of what you
have done. Swapping other programs out to disk is system-specific, if
you want to try this then other newsgroups will be able to give better
help.
Hope that helps.
Paul.
J
John Gordon
In said:
If malloc is returning null, there isn't a lot you can do.
You can wait a short interval and pray that more memory becomes available
(perhaps some other program just exited, freeing up a lot of memory.)
You can free some memory that your program is using elsewhere.
J
James Kuyper
That depends entirely upon what your program is doing. There's only one
general piece of advice that's applicable: do NOT do anything with that
null pointer value other than copying it or comparing it to other pointers.
My own code would generate an error message identifying the fact that
malloc() failed, along with the name of the function that called it, and
the name of the file containing that function. That much is required by
the coding standards for the project I'm working on. Everything else I
will say about this reflects my own personal requirements. If the
failure occurred inside a loop, the error message would indicate the
value of the loop index at the time the problem was identified. If the
message would otherwise be identical to another message that could be
produced in the same block of code, something will be added to it to
allow unique identification of the particular call to malloc() that
triggered the problem. That function would then clean up anything that
it needed to clean up, and then return an error code to the calling
function.
The calling function would respond to that error code by generating
another error message containing the same type of information for the
calling function as was described above for the function in which the
problem was originally detected. This continues all the way up the call
chain to main(), which would generate a final error message before
returning EXIT_FAILURE. The net effect is an annotated reverse
stack-trace. That information isn't particularly useful when dealing
with malloc() failures, but it's part of a general architecture for
dealing with error conditions that helps me get a running start on
diagnosing the underlying problem.
This is a totally inappropriate way of handling error conditions for
many contexts, such as interactive GUI programs. However, most of my
code exclusively runs inside a batch processing environment, where this
is quite a reasonable approach.
You need to figure out the right approach for your purposes.
I
Ian Collins
I can understand this approach on a single use embedded system where
such a failure may be repeatable. Do error messages provide any useful
data on a multi-user system given the generally random memory usage
patterns?
K
Kaz Kylheku
Strategies for out of memory (OOM) handling are a fairly tough problem, with a fairly
broad scope.
Calling exit is not adequate in nontrivial programs, because the module in which OOM
occured doesn't have the authority to shut down the program, let alone abruptly.
One way is to implement exception handling, so that your code can take recovery actions
in an orderly way based on speculations along the lines of "what if out of memory happens
in this dynamic contour? or this one?"
You can set a side a chunk of reserve memory so that you can when OOM happens, it's actually
just a warning: the program is informed that OOM has occured, but is able to continue
thanks to the reserved memory (which is hopefully enough for recovery actions).
If the program is able to fully recover, so that it does not have to terminate, then
it can reset this machinery for the sake of future occurences.
No matter what you do, this is difficult to *test*, because programs make memory allocation
requests in many places and in many states. So the different possible failure scenarios
are huge. What if the first N allocation requests succeed, and the N+1-th fails?
Now consider that for a large range of N, multiplied by variation in the
program's execution based on inputs ...
In general, all error checking and recovery type code which rarely executes only under
special conditions is difficult to test. If it is not tested (or else formally
verified), it only expresses the coder's best intentions to do something about
the situation.
J
James Kuyper
On 09/12/2012 04:34 PM, Eric wrote:
....
When it is possible to do something like that (which is seldom the
case), it is essential that you have one or more blocks of memory
somewhere that you can free(), so that you can retry the malloc() with
some chance of having it succeed.
Problem: if you allocated those other blocks of memory, you either need
to write to them, or you need access to information that has already
been written to them. If neither of those are the case, those other
blocks should either a) not have been allocated yet or b) should have
already been free()d.
Therefore, you're going to have to figure out some alternative way of
dealing with whatever it is that you allocated those other memory blocks
for, an alternative that does not require as much memory. That's the
hard part, and is one reason why this approach is seldom used.
An example that could come up in my own code (if it were radically
re-designed). Several of my programs process one scan of data (13,540
pixels) at a time. Those programs could, instead, have been written to
process only one frame of data (10 pixels) at a time. There's some
inefficiency to doing so, so it would run a bit slower, but it would
also require a lot less memory.
....
When it is possible to do something like that (which is seldom the
case), it is essential that you have one or more blocks of memory
somewhere that you can free(), so that you can retry the malloc() with
some chance of having it succeed.
Problem: if you allocated those other blocks of memory, you either need
to write to them, or you need access to information that has already
been written to them. If neither of those are the case, those other
blocks should either a) not have been allocated yet or b) should have
already been free()d.
Therefore, you're going to have to figure out some alternative way of
dealing with whatever it is that you allocated those other memory blocks
for, an alternative that does not require as much memory. That's the
hard part, and is one reason why this approach is seldom used.
An example that could come up in my own code (if it were radically
re-designed). Several of my programs process one scan of data (13,540
pixels) at a time. Those programs could, instead, have been written to
process only one frame of data (10 pixels) at a time. There's some
inefficiency to doing so, so it would run a bit slower, but it would
also require a lot less memory.
E
Eric Sosman
(Aside: You mean `malloc' and `NULL' and `abort' and `exit';
letter case matters in C.)
Usually, a malloc() failure means the program will not be
able to continue: It wouldn't have requested the memory if it
weren't needed, and if it's needed but not there the program is
in trouble. However, this does not mean the program must or
should halt abruptly: It may want to do some kind of "orderly
shutdown" before quitting, like saving in-memory data to files.
To make this work, the function whose malloc() call failed should
do its own cleanup, then report "*I* failed" to its caller, which
will do its bit of cleanup, then report failure to its caller, and
so on upward to a function that's at a high enough level to make
reasonable decisions on behalf of the entire program. (Variations
of this scheme can involve setjmp() and longjmp(), but they have
their own perils and are not to be used lightly.)
Sometimes the orderly shutdown activities will themselves need
dynamic memory, which may be in short supply if malloc() is failing.
One strategy to supply this need is to malloc() a chunk of "In Case
Of Emergency Break Glass" memory at program startup, then free()
it after malloc() failure to give the last-gasp code some room
to breathe. I'd be tempted to package this inside a wrapper
to be called instead of malloc(), e.g.
void *myMalloc(size_t bytes) {
static int initialized;
static void *emergency_stash;
if (!initialized) {
// Reserve a stash on the first call.
emergency_stash = malloc(STASH_SIZE);
if (emergency_stash == NULL) {
// Ooh, bad: Couldn't even get started!
return NULL;
}
initialized = 1;
}
void *new = malloc(bytes);
if (new == NULL && bytes > 0) {
// Trouble! Raid the rainy day fund -- not for
// this failing request, but for future requests
// the shutdown code may make.
free(emergency_stash);
emergency_stash = NULL; // Don't double-free.
}
return new;
}
Some programs can proceed normally even after malloc() failure,
perhaps giving up on a sub-task but keeping the program alive
to do others. "Sorry, Dave, I haven't got enough memory to load
yet another cute kitten video right now; maybe if you closed a
few that you've already got open we could try again."
J
James Kuyper
While the systems my programs normally run on are nominally multi-user,
there's one user-id that is used to start the production manager
program. All other programs in the production system (including mine)
are started by the manager, directly or indirectly, and therefore all
run using the same user id. For practical purposes, it's a single-user
system (and definitely NOT an embedded one).
As I said, for failed malloc() calls the information provided by those
error messages is seldom of much use - it's just part of a general
framework for error handling that I've found useful. However, most of
the problems that come up are data-driven, and are unaffected by the
"random memory use patterns" of other programs that might be running at
the same time on the same system. For those kinds of problems, I've
found it a very useful framework.
This is example of how the right way to deal with errors depends very
much on the context in which those errors have to be dealt with, which
is precisely the point I was making.
I was recently assigned to help another group with their work - when it
fails, their software provides much less information in the error
messages than mine does. I keenly miss that information.
J
Joe Pfeiffer
Eric said:
Your options are to either end your program, decide you don't really
need the memory you just tried to allocate after all, or free() something
else up so you can use it.
But... if you're on a 32 bit machine, you can malloc() a *lot* of
memory before it'll fail. Are you sure you haven't corrupted the free
space pool?
I
Ian Collins
I understand your pain. I use a similar approach to resource allocation
failures to yours in C++ with exceptions that convey file and line
number information from the call site. These can ether propagate up or
be caught and augmented and re-thrown higher up the stack. I guess I
haven't done anything similar in C because it takes more work and lot
more discipline!
J
James Kuyper
On 09/12/2012 06:28 PM, Ian Collins wrote:
....
On my new task, I'm actually using C++, and I've been trying to figure
out how to make use of exceptions to produce the same kind of error
output I get from the my C programs. It's a fantasy, since the new group
I'm working with has it's own well-entrenched error handling conventions
that would not mix well with my preferred style.
However, when I did indulge that fantasy, I found that one key problem
was that exceptions discard information about where they came from, as
the call stack unwinds. Unless I catch and rethrow around every function
call that could throw an exception (including the implicit function
calls due to ctors, dtors, and operator overloads), it can't duplicate
the information generated by my error messages.
If I did catch and rethrow everywhere, it would end up looking very
similar to my current C code, which constantly checks for error codes.
More than one person has looked at my code and come away with the
impression that its primary purpose was error handling. One person even
submitted a test plan for a module that triggered every single error
message that could be generated by that module, but failed to cover
normal processing.
I'd hoped to find a way to use C++ exceptions to reduce the error
message clutter, but I can't figure out how to do so without a
corresponding reduction in the amount of information conveyed by the
error messages.
....
On my new task, I'm actually using C++, and I've been trying to figure
out how to make use of exceptions to produce the same kind of error
output I get from the my C programs. It's a fantasy, since the new group
I'm working with has it's own well-entrenched error handling conventions
that would not mix well with my preferred style.
However, when I did indulge that fantasy, I found that one key problem
was that exceptions discard information about where they came from, as
the call stack unwinds. Unless I catch and rethrow around every function
call that could throw an exception (including the implicit function
calls due to ctors, dtors, and operator overloads), it can't duplicate
the information generated by my error messages.
If I did catch and rethrow everywhere, it would end up looking very
similar to my current C code, which constantly checks for error codes.
More than one person has looked at my code and come away with the
impression that its primary purpose was error handling. One person even
submitted a test plan for a module that triggered every single error
message that could be generated by that module, but failed to cover
normal processing.
I'd hoped to find a way to use C++ exceptions to reduce the error
message clutter, but I can't figure out how to do so without a
corresponding reduction in the amount of information conveyed by the
error messages.
S
Stefan Ram
Eric said:
This cannot be answered in general, it depends on the requirements
to the software. A similar question that might make this more clear
is to think of someone having asked »What are some good texts to
print using Printf? I do not always want to print "Hello world!".«
I
Ian Collins
I use the OS provided stack trace functions to build a stack trace to
include in the exception, so there's no need to pick anything use up
along the way.
S
Stefan Ram
For example, you can see below how I used »malloc« in a
recent post.
Message-ID: <[email protected]>
vaysagekv said:
The following code does this with at most one call to malloc
per read operation and no call of realloc. The length of
the input is still limited by the size of »size_t«,
»unsigned length«, and especially the size of the memory
for objects with automatic storage duration. However,
all those limitations are specified by the implementation
- not by the program.
#include <stdio.h>
#include <stdlib.h>
size_t count = 0;
unsigned long depth = 0;
char * doread( void )
{ int const c = getchar();
if( c == EOF || c == '\n' || c == '\r' )
{ char * const result = malloc( count + 1 );
if( result )result[ count ]= 0;
return result; }
else
{ ++count; if( count <= 0 )abort();
++depth; if( depth == 0 )abort();
char * const buffer = doread();
--depth;
if( buffer )buffer[ depth ] = c;
return buffer; }}
char * read( void )
{ count = 0;
depth = 0;
return doread(); }
int main( void )
{ char * result = read();
if( result )
{ printf( "text = \"%s\".\n", result );
printf( "count = (%lu).\n",( unsigned long )count );
if( result[ 0 ])printf( "first = '%c'.\n", result[ 0 ]);
if( result[ count > 0 ? count - 1 : 0 ])
printf( "last = '%c'.\n", result[ count > 0 ? count - 1 : 0 ]);
free( result );
result = 0; }}
J
Johann Klammer
Eric said:
Some ideas, most of them difficult to do:
Do it like linux kernel...
If you are using in-memory caches for various things(file i/o, expensive
calculations), you can try reclaiming those.
If you hack up your own datastructures, you could serialize those to
disk on-demand.
Both of these would require you to override the allocation functions at
least. Also, your allocator would have to know of all the reclaimable
areas. Possibly prioritize things... File caching may fail, other things
must succeed.
Never tried any of this myself, but it is surely possible...
M
Malcolm McLean
בת×ריך ×™×•× ×¨×‘×™×¢×™, 12 בספטמבר 2012 21:07:51 UTC+1, מ×ת Eric:
If there's not, you have to decide whether it's worth complicating the interface. Normally the answer is yes, so it's traditional to return -1. Then caller decides what to do.
So typically an allocation failure deep down in the program will be passed up
to main(). main() will then print out "Out of memory" and either abort or
skip to the next item in its list.
If there's a "natural" error return, pass it up to caller. So if you're returning a pointer to dynamic memory, return null.
If there's not, you have to decide whether it's worth complicating the interface. Normally the answer is yes, so it's traditional to return -1. Then caller decides what to do.
So typically an allocation failure deep down in the program will be passed up
to main(). main() will then print out "Out of memory" and either abort or
skip to the next item in its list.