A solution for the allocation failures problem

[santosh]

As I read the description of free(), it releases memory for subsequent
allocation. Bear in mind, C has no concept of multitasking and the
like, thus "subsequent allocation" *must* be for the application.

I'm not sure that the Standard says so. After memory has been
deallocated with free() it could be given back to the OS.

So, if my first program action is to allocate a gig of ram, which I
then free immediately, that gig is still "reserved" for my application
to reallocate. Failure to "reserve" it means it is *not* being made
available for subsequent allocation; it is being handed off to
something else - the OS - instead, in direct contradiction of what
it's supposed to do.

But I think the Standard allows this. VM OSes will reclaim unused
memory.

 

[santosh]

The author appears to turn purple at the very mention of C++! His
operator overloading implementation is nothing like C++.

Okay, I'll take your word for this. I have used lcc-win32 but only as an
ISO C compiler.

 

[Kelsey Bjarnason]

The standard uses the term "made available for further allocation". It
doesn't say to whom.

As relates to free, the standard expressly says what is conforming: the
memory is made available for further allocation. Your conforming code
can absolutely rely on that memory being there, as the definition of free
*expressly* asserts that the memory is available for further allocation,
an assurance it can *only* make to the conforming application in a
conforming implementation.

If this guarantee is violated, the implementation is non-conforming. If
this guarantee wasn't intended, it's a DR. Either way, your application
- by the standard - has every right to expect that memory is available,
*not* poached by the OS.

I'll grant that in the real world, you cannot rely on this - but that
simply means the implementations out there are, on the whole, non-
conforming.

 

[Ian Collins]

As relates to free, the standard expressly says what is conforming: the
memory is made available for further allocation. Your conforming code
can absolutely rely on that memory being there, as the definition of free
*expressly* asserts that the memory is available for further allocation,
an assurance it can *only* make to the conforming application in a
conforming implementation.

It does not, if it did, it would explicitly state that and it does not.

Take the to comp.std.c if you want to discuss it further.

 

[Richard Heathfield]

Ioannis Vranos said:

Well, it may sound somehow unusual, but since C and C++ are something
like siblings, why not just adopt a subset of the C++ exception
mechanism?

If you want C++, you know where to find it.

The solution to the "allocation failures problem" is, believe it or not,
*not* to lock our code into an untrusted proprietary solution but rather
to check that our allocation requests succeeded and deal with them if they
didn't. This is not rocket science. Those who struggle with if() will
struggle even more with try/catch.

 

[jacob navia]

Well, it may sound somehow unusual, but since C and C++ are something
like siblings, why not just adopt a subset of the C++ exception
mechanism? Like throwing specific structs and catch them?

For those not familiar with C++ something like:


void some_function(void) try
{
/* ... */
}

catch(bad_alloc)
{
/* Error handling code here */
}



void somefunc(void)
{
try
{
int *p= malloc(sizeof(*p));
}

catch(bad_alloc)
{
/* Error handling code here */
}

/* ... function code continues... */
}

The lcc-win C compiler adopts a try/catch mechanism, copied from
the proposal of Microsoft Corp's MSVC. This feature allows
for easier programming. If you want to stay within standard C,
you can use the solution that I proposed at the start of this
thread.

My complete proposal is to adopt the namespace mechanism and the
exception handling mechanism of C++, in a way they will be a subset of
C++ equivalent functionalities (that is only for functions, struct
definitions and objects, and variables) - so there can be compatibility
between C and C++ interfaces, making it easier for the implementers to
provide both C and C++ compilers, and making the already existing C++
mechanisms available to C easily and quickly.

That would be really nice but with the current mindset in official
C circles it is a tough fight to do any changes to the language.

They look frozen.

 

[jacob navia]

Ioannis Vranos said:


If you want C++, you know where to find it.

The solution to the "allocation failures problem" is, believe it or not,
*not* to lock our code into an untrusted proprietary solution but rather
to check that our allocation requests succeeded and deal with them if they
didn't.

This is not possible for each allocation at each step without
introducing too many bugs, as the proponents of the other side
have repeatedly pointed out.

When an allocation fails it is better to code an exception
that jumps to a recovery point. This simplifies the code
and makes for LESS bugs. Obviously this solution is not
meant for geniuses like you and the other people here
that boast of their infallible powers when programming.

You never (of course) make any mistake when writing the
hundreds of lines of recovery code. My solution is meant for
people that do make mistakes (like me).


This is not rocket science. Those who struggle with if() will

struggle even more with try/catch.

This is just rubbish. Nobody "struggles with if". But it is
a proved fact that the more lines of code you have to write
the more the possibility of making mistakes.

 

[Ian Collins]

This is not possible for each allocation at each step without
introducing too many bugs, as the proponents of the other side
have repeatedly pointed out.

When an allocation fails it is better to code an exception
that jumps to a recovery point. This simplifies the code
and makes for LESS bugs.

That's not as easy as it sounds in C. C++ has the built in mechanisms
to write exception safe code. Such code is hard to write in C (the same
issues as multiple points of return).

Consider

char* p0 = malloc(100);
char* p1 = malloc(200);
char* p2 = malloc(300);

What do you do if the second malloc throws?

You'd end up with as much if not more code for a series of try/catch
blocks as you would for a series of if() blocks.

 

[Richard Heathfield]

[Piggybacking]

Ian Collins said:

I have demonstrated to many people's satisfaction that there is no point in
my arguing with Jacob Navia, so I will not do so here. Such arguments
swiftly become acrimonious, generating far more heat than light. Suffice
it to say that I do not share his pessimism about the inability of good C
programmers to write good C code. What he finds impossible may not be
impossible for others.

That's not as easy as it sounds in C.

Nor has the case been proved (that it simplifies the code and makes for
fewer bugs). If someone whose opinion I respect would care to support that
case, it might make for a reasonable and informative discussion.

Consider

char* p0 = malloc(100);
char* p1 = malloc(200);
char* p2 = malloc(300);

What do you do if the second malloc throws?

Write a bug report.

You'd end up with as much if not more code for a series of try/catch
blocks as you would for a series of if() blocks.

Right. It is certainly the case that nobody writes perfect code, but
there's nothing particularly special about malloc - it's just a way of
requesting a resource, very similar in that respect to fopen. I don't hear
anyone arguing that it's impossible to deal with fopen failures.

 

[jacob navia]

That's not as easy as it sounds in C. C++ has the built in mechanisms
to write exception safe code. Such code is hard to write in C (the same
issues as multiple points of return).

Consider

char* p0 = malloc(100);
char* p1 = malloc(200);
char* p2 = malloc(300);

What do you do if the second malloc throws?

There are many possibilities:

1) You modify malloc so that it keeps in memory the last allocations.
If there is a failure, it will jump to a recovery point where the
last n allocations are undone.
This can be coupled with calls to mark()/release() like in Pascal,
where you can free() the last n allocations.
2) You just USE A GARBAGE COLLECTOR like the one lcc-win provides!
This solution is the best, since it frees() the mind from this
hyper tedious ACCOUNTING of each memory piece and leaves room for
the interesting stuff of programming.

Yes, I know. There are some programmers like those called here
"good C programmers" that like writing thousands of times
if (result == NULL) {
// cleanup code.
}

I am not one of those.

You'd end up with as much if not more code for a series of try/catch
blocks as you would for a series of if() blocks.

If you program with good tools and use a bit of creativity you do not
have to.

 

[Robert Latest]

Consider

char* p0 = malloc(100);
char* p1 = malloc(200);
char* p2 = malloc(300);

What do you do if the second malloc throws?

You'd end up with as much if not more code for a series of try/catch
blocks as you would for a series of if() blocks.

Here's how I've dealt with such a situation once:

------------------------
void *my_malloc(size_t size, int *err)
{
void *buf;
if (*err) {
++*err;
return NULL;
}
buf = malloc(size);
*err = (buf == NULL);
return buf;
}

int main(void)
{
char *a, *b, *c;
int err = 0;

a = my_malloc(100, &err);
b = my_malloc(200, &err);
c = my_malloc(300, &err);
return err;
}
--------------------

That wasn't for acual malloc()s, but I've used this for demonstration.
The idea is that you can lump subsequent calls to error-prone functions
together and check once for an error condition at the end.

I recently implemented this scheme in a program that bombarded a
measurement system with requests over a network. If one of those calls
fails it doesn't make any sense (in fact in this case it could be
harmful to expensive equipment) to keep soldering on, so the program
simply falls through the rest of the calls, checks for an error
condition and then deals with it. The increment of *err inside the
called function even allows localization of the error.

Not an universal recipe, but in this case it worked for me.

robert

 

[Chris Dollin]

Robert Latest wrote:

I recently implemented this scheme in a program that bombarded a
measurement system with requests over a network. If one of those calls
fails it doesn't make any sense (in fact in this case it could be
harmful to expensive equipment) to keep soldering on,

I can see that excess solder could be a problem.

 

[Ian Collins]

There are many possibilities:

1) You modify malloc so that it keeps in memory the last allocations.
If there is a failure, it will jump to a recovery point where the
last n allocations are undone.
This can be coupled with calls to mark()/release() like in Pascal,
where you can free() the last n allocations.
2) You just USE A GARBAGE COLLECTOR like the one lcc-win provides!
This solution is the best, since it frees() the mind from this
hyper tedious ACCOUNTING of each memory piece and leaves room for
the interesting stuff of programming.

That's fine for memory, but say another resource was claimed between the
first two mallocs, you would have to use multiple try blocks.

There's no getting away form the need for exception safe programming
once you add exceptions.

 

[Robert Latest]

Consider

char* p0 = malloc(100);
char* p1 = malloc(200);
char* p2 = malloc(300);

What do you do if the second malloc throws?

There are many possibilities:
[...]

2) You just USE A GARBAGE COLLECTOR like the one lcc-win provides!

Sounds like some interesting garbage collector. If the first call above
ate up the last 100 available bytes, how is a GC going to do anything
about subsequent calls?

The fact remains that the memory management of modern OSes sacrifices a
certain amount of reliablility for speed. Non-critical apps should be
aware of this fact. If your software is part of a commercial airliner's
avionics, don't use malloc() and free() at all, let alone a GC.
Statically hog all necessary RAM at start-up, and write
your own memory manager around it.

(I don't care much for military aircraft and space travel. For all I
care their firmware may be written with lcc-win)

Yes, I know. There are some programmers like those called here
"good C programmers" that like writing thousands of times
if (result == NULL) {
// cleanup code.
}

I wonder where you see all those people against extensions. Nobody here
writes exclusively Standard C without non-standard libraries. If you
want a GC, get one and link against it. Just don't come to CLC if it
doesn't work as expected. As with any third-party library.

robert

 

[jacob navia]

Consider

char* p0 = malloc(100);
char* p1 = malloc(200);
char* p2 = malloc(300);

What do you do if the second malloc throws?

There are many possibilities:
[...]

2) You just USE A GARBAGE COLLECTOR like the one lcc-win provides!

Sounds like some interesting garbage collector. If the first call above
ate up the last 100 available bytes, how is a GC going to do anything
about subsequent calls?

No, you just jump to the recovery point, then you call

gc()

and all the intermediate allocations are automatically freed.
No need for costly recovery measures.

The fact remains that the memory management of modern OSes sacrifices a
certain amount of reliablility for speed. Non-critical apps should be
aware of this fact. If your software is part of a commercial airliner's
avionics, don't use malloc() and free() at all, let alone a GC.

I wouldn't even use malloc()!

malloc CAN'T be reliably used in an avionics settings. It CAN
fail and that is not an option when landing an aircraft for
instance

Statically hog all necessary RAM at start-up, and write
your own memory manager around it.

Or use static buffers

(I don't care much for military aircraft and space travel. For all I
care their firmware may be written with lcc-win)


I wonder where you see all those people against extensions. Nobody here
writes exclusively Standard C without non-standard libraries. If you
want a GC, get one and link against it. Just don't come to CLC if it
doesn't work as expected. As with any third-party library.

robert

exactly

 

[Herbert Rosenau]

[snips]

In practise it does not - because immediately after free() has oned its
work the sheduler gives another thread the CPU and that thread has
nothing better to do than to eat the released memory for its own job.

Scheduler or no, it seems to me than an implementation which does this
may in fact be non-conforming.

Why? The standard says absolute nothing about multiprocessing and
multithreading. As it does neither allow or forbid them it lefts it to
the implementation to (not) handle that.

Or functions, plural. No reason that if you're 9 levels down the call
tree, seven of them might not have things to do in the recovery process.

Right, some will flag other processes/treads about that event, some
will only theyr own task with informing theyr caller, some will have
to clean up theyr own work, some do all of them, some do nothing
except to return the error......

No, there are two options: do exactly that, which is what I suggested, or
do what Malcolm et al seem bent on doing: crashing and burning.

It really is an option. Careless, sloppy, ill-conceived and potentially
disastrous, but it is still an option.


Unless one has actually exhausted the alternatives: no UI, so you can't
alert the user. No disk space left, so you can't save. Etc. At some
point you do have to simply give up; I just can't agree with it being the
_first_ choice, rather than the _last_.

And even then there is no cause to give up the application. I had an
project where such app was sitting as an extension inside a server.
Yes, on some error situations there is nothing one can do because
after receiving the order the connection gets broken, so it was
nothing to do as to clean up and wait for new order. Break down?
Forbidden mission because other connections are active.

--
Tschau/Bye
Herbert

Visit http://www.ecomstation.de the home of german eComStation
eComStation 1.2R Deutsch ist da!

 

[Herbert Rosenau]

As I read the description of free(), it releases memory for subsequent
allocation. Bear in mind, C has no concept of multitasking and the like,
thus "subsequent allocation" *must* be for the application.

Chapter and verse please wher the standard requires that free()d
memory can't given back to the OS.

So, if my first program action is to allocate a gig of ram, which I then
free immediately, that gig is still "reserved" for my application to
reallocate. Failure to "reserve" it means it is *not* being made
available for subsequent allocation; it is being handed off to something
else - the OS - instead, in direct contradiction of what it's supposed to
do.

Such implementation is itself buggy when it is running in an
multiprocess environment. True, yes, it will aquire a block of memory
from the OS that is never given back until the program gets closed,
but when it needs more it will request another block and will give
that back when it is completely unused.

The standard does not require that a block requested from malloc() can
never given back to the OS when completely unused.

malloc() will request a block of memory from the OS and split it up in
one or more chuncs of memory to fullify requeuests from its user. If
inside that block is no more a chunk free that is big enough to
fullify the questet it will request another big enough block from the
OS - when this fails it returns NULL. free() will return the chunk to
the list of unused chunks and when the block itself contains now no
used chunk it may give that block back to the OS, giving another app.
the chance to to use that. This does not mean that free() is requirded
togive the blcok back immediately after it is emty.

As malloc() delivers chunks it is highly possible that at some time a
block is mostenly unused but contains no chunk big enough to fullify
the request of the caller, so another block must be requested from the
OS. A single chunk of e.g. 10 bytes can be the cause not been able to
hinder malloc on give you 100 MB because the block holding that chunk
hinders the OS on giving malloc() the requested amout of blocksize to
deliver a chunk big enough to hold the 100 MB.

malloc() is a bit more than only a wrapper around the system APIs to
the memory manager. The system handles memory in sizes of full pages
or segments compounded to a logical continous block of memory.
malloc() breaks this down to chunks of 1 byte upwards + some bytes for
its own memory management.


--
Tschau/Bye
Herbert

Visit http://www.ecomstation.de the home of german eComStation
eComStation 1.2R Deutsch ist da!

 

[Bartc]

Consider

char* p0 = malloc(100);
char* p1 = malloc(200);
char* p2 = malloc(300);

What do you do if the second malloc throws?

You'd end up with as much if not more code for a series of try/catch
blocks as you would for a series of if() blocks.

OK, here's one idea of my own. It's not a complete solution of course.

* Allocation a big chunk of memory at the start of execution (failure here
is not critical)
* Use that as a private heap area for some allocations
* For all allocations in the application which are known to be small and
well-managed, use this private heap.
* The size of this heap is chosen to be fit comfortably all the allocations
which are likely to be active at one time.
* If this private heap overflows, while it's easy to keep expanding it, it's
better to treat this as an error in the program which uses it.

This heap allocator is not expected to fail. Therefore error returns don't
need to be checked; it will either return a valid pointer or show an error
message. (This is not that much different from expecting array indices to be
in range.)

A lot of small allocations can then make use of this instead:

char* p0 = heap(100);
char* p1 = heap(200);
char* p2 = heap(300);
....
freeheap(p0); etc..

Otherwise, if you need to use malloc() for a trivial allocation like a few
dozen bytes, then you are exposing the program to the risk of major failure;
you *must* check this allocation succeeded, and if not then you may have a
decidedly non-trivial task of cleaning up and recovering:

char *filespec;

filespec=malloc(strlen(file)+strlen(ext)+1);
if (filespec=NULL) then /* big headache of what to do about this */
....
strcopy(filespec,file);
strcat(filespec,ext);
status=checkfileexists(filespec);

free(filespec);
return status;

The fact that malloc() failed on this tiny allocation means there is a major
problem, it's not just not being able to complete this file operation (or
whatever); rather than just failing this operation, it should signal
somewhere else better able to handle this.

(And, for this function that returns 1: the file exists, 0:doesn't exist;
should a malloc() failure just return 0? That would be misleading. Perhaps
introduce 2:don't know?!)

Solution: use heap()/freeheap() instead.

(The real scenario comes from a program which is an interpreter for another
language; memory allocation/deallocation is implicit, in fact the programmer
can't even intervene:

return checkfileexists(file+ext);

The programmer may not even be aware that he narrowly missed catastrophic
failure of his application!)

 

[Ioannis Vranos]

That's not as easy as it sounds in C. C++ has the built in mechanisms
to write exception safe code. Such code is hard to write in C (the same
issues as multiple points of return).

Consider

char* p0 = malloc(100);
char* p1 = malloc(200);
char* p2 = malloc(300);

What do you do if the second malloc throws?

You'd end up with as much if not more code for a series of try/catch
blocks as you would for a series of if() blocks.

You can retrieve information about what failed from the exception itself.

 

[Ioannis Vranos]

Right. It is certainly the case that nobody writes perfect code, but
there's nothing particularly special about malloc - it's just a way of
requesting a resource, very similar in that respect to fopen. I don't hear
anyone arguing that it's impossible to deal with fopen failures.

Exceptions can be used for all kinds of resource allocations.