A solution for the allocation failures problem

Discussion in 'C Programming' started by Kelsey Bjarnason, Jan 29, 2008.

  1. On Tue, 29 Jan 2008 12:47:27 +0100, jacob navia wrote:

    > 1:
    > It is not possible to check EVERY malloc result within complex software.


    Says who? There are a limited number of places memory is allocated, thus
    a limited number of places checks need to be put in.

    > 3:
    > A solution like the one proposed by Mr McLean (aborting) is not possible
    > for software quality reasons. The program must decide if it is possible
    > to just abort() or not.


    Oh, this solution is always *possible*, but I'd be very much annoyed with
    any developer stupid enough to adopt this as a default strategy in any
    application I actually use.


    > Solution:
    >
    > 1) At program start, allocate a big buffer that is not used elsewhere in
    > the program.


    Note that in at least one app, I do exactly this - and I *expect* that
    the allocation will fail on a pretty regular basis. Of course, if I'm
    stupid enough to be using something like Malcolm's xmalloc, it's too
    late, the app has already bombed by the time I would normally just detect
    the NULL and reduce the size of the request.

    'Sides, it's a messy approach, one which ties up resources you don't
    really need, resources best left alone for other things. Just because it
    makes *your* app easier to write doesn't mean it's a good idea - ponder a
    word processor grabbing 128MB for document space, with no documents
    loaded, on an otherwise busy machine.
    Kelsey Bjarnason, Jan 29, 2008
    #1
    1. Advertising

  2. Kelsey Bjarnason

    jacob navia Guest

    1:
    It is not possible to check EVERY malloc result within complex software.

    2:
    The reasonable solution (use a garbage collector) is not possible for
    whatever reasons.

    3:
    A solution like the one proposed by Mr McLean (aborting) is not
    possible for software quality reasons. The program must decide
    if it is possible to just abort() or not.

    Solution:

    1) At program start, allocate a big buffer that is not used
    elsewhere in the program. This big buffer will be freed when
    a memory exhaustion situation arises, to give enough memory
    to the error reporting routines to close files, or otherwise
    do housekeeping chores.

    2) xmalloc()

    static int (*mallocfailedHandler)(int);
    void *xmalloc(size_t nbytes)
    {
    restart:
    void *r = malloc(nbytes);
    if (r)
    return r;
    // Memory exhaustion situation.
    // Release some memory to the malloc/free system.
    if (BigUnusedBuffer)
    free(BigUnusedBuffer);
    BigUnusedBuffer = NULL;
    if (mallocfailedHandler == NULL) {
    // The handler has not been set. This means
    // this application does not care about this
    // situation. We exit.
    fprintf(stderr,
    "Allocation failure of %u bytes\n",
    nbytes);
    fprintf(stderr,"Program exit\n");
    exit(EXIT_FAILURE);
    }
    // The malloc handler has been set. Call it.
    if (mallocfailedHandler(nbytes)) {
    goto restart;
    }
    // The handler failed to solve the problem.
    // Exit without any messages.
    exit(EXIT_FAILURE);
    }

    4:
    Using the above solution the application can abort if needed, or
    make a long jump to a recovery point, where the program can continue.

    The recovery handler is supposed to free memory, and reallocate the
    BigUnusedBuffer, that has been set to NULL;


    --
    jacob navia
    jacob at jacob point remcomp point fr
    logiciels/informatique
    http://www.cs.virginia.edu/~lcc-win32
    jacob navia, Jan 29, 2008
    #2
    1. Advertising

  3. In article <fnn3og$o40$>, jacob navia <> wrote:

    >It is not possible to check EVERY malloc result within complex software.


    I assume you're just setting up a scenario here rather than making
    this claim yourself.

    >1) At program start, allocate a big buffer that is not used
    >elsewhere in the program. This big buffer will be freed when
    >a memory exhaustion situation arises, to give enough memory
    >to the error reporting routines to close files, or otherwise
    >do housekeeping chores.


    This is not guaranteed to help with all malloc() implementations.
    Some use different areas of memory for different sized blocks, and may
    not have code to re-purpose the freed block. Or they may free the
    memory back to the OS, in which case it may get gobbled up by
    something else before they have the chance to re-use it.

    -- Richard
    --
    :wq
    Richard Tobin, Jan 29, 2008
    #3
  4. Kelsey Bjarnason

    Guest

    On Jan 29, 1:47 pm, jacob navia <> wrote:
    > 1:
    > It is not possible to check EVERY malloc result within complex software.

    It is, with proper design it becomes easier too.

    > 3:
    > A solution like the one proposed by Mr McLean (aborting) is not
    > possible for software quality reasons. The program must decide
    > if it is possible to just abort() or not.

    That was not a solution.

    > Solution:
    >
    > 1) At program start, allocate a big buffer that is not used
    > elsewhere in the program. This big buffer will be freed when
    > a memory exhaustion situation arises, to give enough memory
    > to the error reporting routines to close files, or otherwise
    > do housekeeping chores.

    That does not guarantee anything, it depends on the OS, and the
    implementation of free/malloc.
    Also it makes your program slower, and what if the big buffer
    allocation fails?
    Would you exit because your program failed to allocate resources it
    would not use?
    What if no allocation fails? The buffer is only a waste of resources.

    > 2) xmalloc()
    >
    > static int (*mallocfailedHandler)(int);
    > void *xmalloc(size_t nbytes)
    > {
    > restart:
    > void *r = malloc(nbytes);
    > if (r)
    > return r;
    > // Memory exhaustion situation.
    > // Release some memory to the malloc/free system.
    > if (BigUnusedBuffer)
    > free(BigUnusedBuffer);

    free it anyway, free(NULL); does nothing.

    > BigUnusedBuffer = NULL;
    > if (mallocfailedHandler == NULL) {
    > // The handler has not been set. This means
    > // this application does not care about this
    > // situation. We exit.
    > fprintf(stderr,
    > "Allocation failure of %u bytes\n",
    > nbytes);

    Undefined behavior, you pass size_t where a variadic function expects
    unsigned int.

    > fprintf(stderr,"Program exit\n");
    > exit(EXIT_FAILURE);
    > }

    What about previous allocations that have been done with xmalloc?
    memory leak.

    > // The malloc handler has been set. Call it.
    > if (mallocfailedHandler(nbytes)) {
    > goto restart;
    > }
    > // The handler failed to solve the problem.
    > // Exit without any messages.
    > exit(EXIT_FAILURE);

    Memory leak because you don't give the user a chance to free previous
    allocations

    > The recovery handler is supposed to free memory, and reallocate the
    > BigUnusedBuffer, that has been set to NULL;

    So if the allocation for BigUnusedBuffer succeeds but the allocation
    after the callback fails, we will enter a loop of free-ing/allocating
    the big buffer, great.

    mr Jacob, I suggest you read a book or two on program structure &
    designing.
    , Jan 29, 2008
    #4
  5. Kelsey Bjarnason

    CJ Guest

    On 29 Jan 2008 at 11:47, jacob navia wrote:
    > 1:
    > It is not possible to check EVERY malloc result within complex software.
    >
    > 2:
    > The reasonable solution (use a garbage collector) is not possible for
    > whatever reasons.
    >
    > 3:
    > A solution like the one proposed by Mr McLean (aborting) is not
    > possible for software quality reasons. The program must decide
    > if it is possible to just abort() or not.
    >
    > Solution:
    >
    > 1) At program start, allocate a big buffer that is not used
    > elsewhere in the program. This big buffer will be freed when
    > a memory exhaustion situation arises, to give enough memory
    > to the error reporting routines to close files, or otherwise
    > do housekeeping chores.
    >
    > 2) xmalloc()


    Why xmalloc? There's already a mechanism for dealing with such problems,
    viz signals.

    #include <signal.h>

    void alloc_failure_handle(int x)
    {
    fputs("Dis baby gon crash and burn!\n", stderr);
    exit(EXIT_FAILURE);
    /* or insert your free-buffer-and-clean-up stuff if you prefer */
    }

    main()
    {
    signal(SIGSEGV, alloc_failure_handle);
    /* ... */
    }
    CJ, Jan 29, 2008
    #5
  6. Kelsey Bjarnason

    cr88192 Guest

    "jacob navia" <> wrote in message
    news:fnn3og$o40$...
    > 1:
    > It is not possible to check EVERY malloc result within complex software.
    >


    <snip>

    >
    > 4:
    > Using the above solution the application can abort if needed, or
    > make a long jump to a recovery point, where the program can continue.
    >
    > The recovery handler is supposed to free memory, and reallocate the
    > BigUnusedBuffer, that has been set to NULL;
    >


    5:
    add exception handling (ie, as a library feature, or, possibly as a compiler
    extension).

    this way, we can throw, and see if the app has any nifty ideas...
    I propose implementing something combining both signal handling and
    unwinding features.

    sza=(1<<31)-1;
    while(1)
    {
    i=exBegin();
    if(!i)
    {
    //try...
    pa=malloc_throw(sza);
    }else if(i==EX_NO_MEM)
    {
    //catch...
    sza=(sza/3)*2;
    continue;
    }else
    {
    //cleanup...
    exRelay(i); //relay to next handler
    }
    break;
    }


    or something...


    (ok, nevermind that this idea is stupid, as directly handling the NULL
    pointer is simpler).

    sza=(1<<31)-1;
    pa=malloc(sza);
    while(!pa)
    {
    sza=(sza/3)*2;
    pa=malloc(sza);
    }


    >
    > --
    > jacob navia
    > jacob at jacob point remcomp point fr
    > logiciels/informatique
    > http://www.cs.virginia.edu/~lcc-win32
    cr88192, Jan 29, 2008
    #6
  7. Kelsey Bjarnason

    Guest

    On Jan 29, 2:26 pm, wrote:
    > On Jan 29, 1:47 pm, jacob navia <> wrote:> 1:
    > > It is not possible to check EVERY malloc result within complex software.

    > mr Jacob, I suggest you read a book or two on program structure &
    > designing.


    Here's a solution I propose:

    Have a linked list for each allocation done and three functions,
    --
    xmalloc_start(); /* initializes a global linked list for future
    allocations */
    if(atexit(xmalloc_end)) /* handle error */
    /* ... */
    p = xmalloc(123, 0); /* false = 0, if the allocation fails, xmalloc
    exits. */
    s = xmalloc(123, 1); /* true = !0, if the allocation fails, xmalloc
    returns NULL */
    --
    , Jan 29, 2008
    #7
  8. In article <>,
    CJ <> wrote:

    >Why xmalloc? There's already a mechanism for dealing with such problems,
    >viz signals.


    > signal(SIGSEGV, alloc_failure_handle);


    An ignored malloc() failure is not guaranteed to lead to a SIGSEGV.
    It may just result in memory corruption.

    -- Richard

    --
    :wq
    Richard Tobin, Jan 29, 2008
    #8
  9. Kelsey Bjarnason

    Richard Bos Guest

    jacob navia <> wrote:

    > 1:
    > It is not possible to check EVERY malloc result within complex software.


    Bollocks from the start. Well done.

    Richard
    Richard Bos, Jan 29, 2008
    #9
  10. Kelsey Bjarnason

    santosh Guest

    CJ wrote:

    > On 29 Jan 2008 at 11:47, jacob navia wrote:
    >> 1:
    >> It is not possible to check EVERY malloc result within complex
    >> software.
    >>
    >> 2:
    >> The reasonable solution (use a garbage collector) is not possible for
    >> whatever reasons.
    >>
    >> 3:
    >> A solution like the one proposed by Mr McLean (aborting) is not
    >> possible for software quality reasons. The program must decide
    >> if it is possible to just abort() or not.
    >>
    >> Solution:
    >>
    >> 1) At program start, allocate a big buffer that is not used
    >> elsewhere in the program. This big buffer will be freed when
    >> a memory exhaustion situation arises, to give enough memory
    >> to the error reporting routines to close files, or otherwise
    >> do housekeeping chores.
    >>
    >> 2) xmalloc()

    >
    > Why xmalloc? There's already a mechanism for dealing with such
    > problems, viz signals.
    >
    > #include <signal.h>
    >
    > void alloc_failure_handle(int x)
    > {
    > fputs("Dis baby gon crash and burn!\n", stderr);


    The Standard library functions are not guaranteed to be reentrant and
    you cannot safely call them from a signal handler.

    > exit(EXIT_FAILURE);
    > /* or insert your free-buffer-and-clean-up stuff if you prefer */
    > }
    >
    > main()
    > {
    > signal(SIGSEGV, alloc_failure_handle);
    > /* ... */
    > }


    The state of a program after it has handled a SIGSEGV is undefined as
    per the Standard.

    IMHO, it's far better to handle out-of-memory conditions within normal
    code rather than invoking all the complexity and fragility of signals.
    santosh, Jan 29, 2008
    #10
  11. Kelsey Bjarnason

    santosh Guest

    Richard Tobin wrote:

    > In article <fnn3og$o40$>, jacob navia <>
    > wrote:
    >
    >>It is not possible to check EVERY malloc result within complex
    >>software.

    >
    > I assume you're just setting up a scenario here rather than making
    > this claim yourself.
    >
    >>1) At program start, allocate a big buffer that is not used
    >>elsewhere in the program. This big buffer will be freed when
    >>a memory exhaustion situation arises, to give enough memory
    >>to the error reporting routines to close files, or otherwise
    >>do housekeeping chores.

    >
    > This is not guaranteed to help with all malloc() implementations.
    > Some use different areas of memory for different sized blocks, and may
    > not have code to re-purpose the freed block. Or they may free the
    > memory back to the OS, in which case it may get gobbled up by
    > something else before they have the chance to re-use it.


    Yes. A better strategy is to use this buffer directly. No need to invoke
    all the uncertainty involved in freeing and allocating again.
    santosh, Jan 29, 2008
    #11
  12. "jacob navia" <> wrote in message
    > 3:
    > A solution like the one proposed by Mr McLean (aborting) is not
    > possible for software quality reasons. The program must decide
    > if it is possible to just abort() or not.
    >

    xmalloc() calls a caller-defined emergency function. It can do several
    things, one of which is to simply exit. Don't confuse the requirement to
    return the memory requested or (inherently) abort with the requirement to
    abort.

    --
    Free games and programming goodies.
    http://www.personal.leeds.ac.uk/~bgy1mm
    Malcolm McLean, Jan 29, 2008
    #12
  13. Kelsey Bjarnason

    Bartc Guest

    jacob navia wrote:

    >A solution for the allocation failures problem


    You may have noticed this is somewhat controversial here. Doubt you will get
    much agreement. It's one of those things where everyone has their own ideas.

    --
    Bart
    Bartc, Jan 29, 2008
    #13
  14. Kelsey Bjarnason

    Eric Sosman Guest

    jacob navia wrote:
    > 1:
    > It is not possible to check EVERY malloc result within complex software.


    Why not? It's no harder than checking every fopen() for
    success -- or is that another impossibility?

    > 2:
    > The reasonable solution (use a garbage collector) is not possible for
    > whatever reasons.


    Garbage collection (when it works) eases the memory management
    problem by relieving the programmer of the need to call free().
    But collecting all the garbage does not imply that every malloc()
    will succeed! In fact, a collector that cannot relocate the non-
    garbage is likely to increase fragmentation and produce allocation
    failures when a free()-as-soon-as-possible strategy would not.

    > 3:
    > A solution like the one proposed by Mr McLean (aborting) is not
    > possible for software quality reasons. The program must decide
    > if it is possible to just abort() or not.
    >
    > Solution:
    >
    > 1) At program start, allocate a big buffer that is not used
    > elsewhere in the program. This big buffer will be freed when
    > a memory exhaustion situation arises, to give enough memory
    > to the error reporting routines to close files, or otherwise
    > do housekeeping chores.


    This is a reasonable thing to try, and has been tried
    often. The hard part is choosing a value for "big:" too little
    and there's not enough for the cleanup activity, too much and
    you provoke allocation failures that don't need to happen.

    > 4:
    > Using the above solution the application can abort if needed, or
    > make a long jump to a recovery point, where the program can continue.


    The problems of longjmp() are well understood. It's possible
    to use it effectively, but the programmers who write functions
    "between" the setjmp() and the longjmp() must be constantly aware
    that they might not get a chance to clean up:

    char *this = dup(getenv("THIS"));
    char *that = dup(getenv("THAT"));
    printf ("THIS = %s, THAT = %s\n", this, that);
    free (this);
    free (that);

    See the memory leak? If the second dup() eventually calls
    longjmp() and returns to an ancestor of this code, the
    memory allocated to `this' is never freed.

    It's possible to work around this problem: various people
    have put together macro packages that imitate a try/finally
    discipline, for example. But the language itself gives little
    help, and the compilers won't warn if somebody forgets (for
    example, when a function that originally didn't need cleanup
    acquires such a need during maintenance). Error recovery
    based on longjmp() is do-able, but difficult.

    --
    Eric Sosman, Jan 29, 2008
    #14
  15. jacob navia <> writes:
    > 1:
    > It is not possible to check EVERY malloc result within complex software.

    [...]

    It's not only possible, it's required.

    --
    Keith Thompson (The_Other_Keith) <>
    Nokia
    "We must do something. This is something. Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"
    Keith Thompson, Jan 29, 2008
    #15
  16. "Richard Bos" <> schreef in bericht
    news:4all.nl...
    > jacob navia <> wrote:
    >
    >> 1:
    >> It is not possible to check EVERY malloc result within complex software.

    >
    > Bollocks from the start. Well done.
    >
    > Richard


    Think he meant it's not possible to HANDLE every malloc failure. Indeed its
    pretty easy putting an if in there, but not so easy to actually gracefully
    handle it.
    Serve Laurijssen, Jan 29, 2008
    #16
  17. "Serve Laurijssen" <> writes:
    > "Richard Bos" <> schreef in bericht
    > news:4all.nl...
    >> jacob navia <> wrote:
    >>
    >>> 1:
    >>> It is not possible to check EVERY malloc result within complex software.

    >>
    >> Bollocks from the start. Well done.

    >
    > Think he meant it's not possible to HANDLE every malloc
    > failure. Indeed its pretty easy putting an if in there, but not so
    > easy to actually gracefully handle it.


    Yeah, programming is hard.

    --
    Keith Thompson (The_Other_Keith) <>
    Nokia
    "We must do something. This is something. Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"
    Keith Thompson, Jan 29, 2008
    #17
  18. Serve Laurijssen <> wrote:

    > "Richard Bos" <> schreef in bericht
    > news:4all.nl...
    > > jacob navia <> wrote:
    > >
    > >> 1:
    > >> It is not possible to check EVERY malloc result within complex software.

    > >
    > > Bollocks from the start. Well done.
    > >
    > > Richard


    > Think he meant it's not possible to HANDLE every malloc failure. Indeed its
    > pretty easy putting an if in there, but not so easy to actually gracefully
    > handle it.


    If you write software not intending to recover from a malloc failure, then,
    yes, upon inspection your code will prove to be very difficult to modify to
    handle such recovery.

    But, that's beside the point. If at outset you write your code intending to
    handle recovery, its not difficult at all. I don't remember (and granted I
    can't remember what I wrote when I first began programming in C) ever being
    in a situation where I found it difficult to recover or unwind from a path
    because of a failed malloc call. Of course, I have developed a very
    structured, almost rote method for writing software which suits me. But I
    did so by necessity, because from very early on I never accepted the premise
    that memory failure could or should be ignored.

    I originally learned to program with Perl, then JavaScript (Navigator 3.0).
    Subsequently, to me memory management was a _feature_. I found it much more
    satisfying to have the capacity (if not perfectly realized or utilized) to
    write more resilient programs.

    When I don't wish to exercise that feature, and if C is otherwise not
    particularly suited to a task, I use another language.

    Aside from glib, I can't off-hand think of any widely used Free Software C
    library which doesn't check and propogate memory allocation failures.
    William Ahern, Jan 29, 2008
    #18
  19. "William Ahern" <william@wilbur.25thandClement.com> wrote in message
    > But, that's beside the point. If at outset you write your code intending
    > to
    > handle recovery, its not difficult at all. I don't remember (and granted I
    > can't remember what I wrote when I first began programming in C) ever
    > being in a situation where I found it difficult to recover or unwind from
    > a
    > path because of a failed malloc call. Of course, I have developed a very
    > structured, almost rote method for writing software which suits me. But I
    > did so by necessity, because from very early on I never accepted the
    > premise that memory failure could or should be ignored.
    >

    I wrote xmalloc() for Baby X, which is a minimalist X windows toolkit.
    The problem is that the whole IDE is strung together with function pointers,
    so there is no high level to propagate to. Most of the allocations are for
    tiny structures associated with windows, anyway. It is also not clear what
    you should do if unable to put up a window. My solution is to nag the user
    for more memory until he gives it or kills the application.

    --
    Free games and programming goodies.
    http://www.personal.leeds.ac.uk/~bgy1mm
    Malcolm McLean, Jan 29, 2008
    #19
  20. Kelsey Bjarnason

    Jack Klein Guest

    On Tue, 29 Jan 2008 12:47:27 +0100, jacob navia <>
    wrote in comp.lang.c:

    > 1:
    > It is not possible to check EVERY malloc result within complex software.


    Yes, it is. Poor programmers might not wish to do so, and might
    produce arguments defending their choice, but that does not change the
    fact that it is possible.

    --
    Jack Klein
    Home: http://JK-Technology.Com
    FAQs for
    comp.lang.c http://c-faq.com/
    comp.lang.c++ http://www.parashift.com/c -faq-lite/
    alt.comp.lang.learn.c-c++
    http://www.club.cc.cmu.edu/~ajo/docs/FAQ-acllc.html
    Jack Klein, Jan 29, 2008
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kevin Spencer

    Re: So many failures early on

    Kevin Spencer, Aug 13, 2003, in forum: ASP .Net
    Replies:
    4
    Views:
    333
    Kevin Spencer
    Aug 27, 2003
  2. Ken
    Replies:
    24
    Views:
    3,856
    Ben Bacarisse
    Nov 30, 2006
  3. chris
    Replies:
    6
    Views:
    985
    chris
    Oct 28, 2005
  4. Richard Tobin

    Handling allocation failures?

    Richard Tobin, Sep 28, 2007, in forum: C Programming
    Replies:
    7
    Views:
    350
    Malcolm McLean
    Sep 29, 2007
  5. Bjarke Hammersholt Roune
    Replies:
    14
    Views:
    1,180
    Bjarke Hammersholt Roune
    Mar 6, 2011
Loading...

Share This Page