xmalloc string functions

[Kelsey Bjarnason]

[snips]

Now I got it. You don't care about how the library actually works, you
only need one bogus sentence from docs to be able to talk about how the
library works

You *almost* have it right, but not quite.

The documentation tells me how the library is _supposed_ to work. In the
case of glib, it is _supposed_ to terminate the application on allocation
failure. In fact, it says so, right there in the documentation.

It *also* says that there is no point even *checking* whether an
allocation succeeded or not, because it _will_ succeed or the app will
terminate.

Thus, if I write code according to how glib is _supposed_ to work, I will
find myself, sooner or later, in a situation such as this:

type_t *ptr = g_allocation_function(...);
/* no check for ptr being NULL */
memset( ptr, 0, size ); /* or some equivalent */

If glib does *not* work as documented, then it *can* return a NULL, and
the code written to the documented guarantee that a NULL will not be
returned is now going to fail.

So it either fails on allocation failure, which, frankly, I find
unacceptable, or it violates its own documented guarantees, breaking the
code that uses it. I don't see how either of these makes it a good idea
to use this library.

 

[Nick Keighley]

On 29 Jan, 09:23, (e-mail address removed) wrote:


You're mixing up execute with fork.

I assure you I'm not! Where did the term "fork" appear in my message.
I don't believe I have *ever* called fork().

Your policy is to always fork on every
call to malloc().

so who is the "you" in "your". It isn't me! Do you mean "ones policy"?

The question is, do you then execute both branches of the
fork in every debug run?

x = malloc();

if (fork() == ORIGINAL_PROCESS)
do_old_process_stuff();
else
do_new_process_stuff();

I'm afraid I don't know enough about the semantics of malloc()
in this case. Do both processes see the failure? In that case they
both run appropriate recovery code.

 

[Malcolm McLean]

xmalloc() crashes on error. It always crashes on error. Therefore you
cannot recover from an error if you use xmalloc().

Not it doesn't. It will exit() if it is impossible to honour the request for
more memory, but it might even susepend the app, save it to disk, and
automatically email an order for a larger PC to PC World. It can do
anything, except refuse to honour the request.

We seem to be agreeing that glib cannot be safely used for most
programs.

glib uses exactly the same strategy, or as near as makes no difference. Now
if glib didn't, the situation would be exactly the same, my arguments would
still have the same validity. But the fact that another major system uses
exactly the same strategy shows that it can be used safely, for most
programs. Not for every program. For programs that put windows up on
screens, or programs that will fail if the computer they are running on
loses power, or programs that run on machines with more than a few megabytes
of memory, or programs that can have no meaningful recovery strategy,
xmalloc() is fine. There's a set of programs that don't meet those
characteristics, there you wouldn't use glib or xmalloc().

 

[Malcolm McLean]

I have first-hand knowledge that VMware Workstation, Server, and
ESX Server contain wrappers over malloc() and related functions
that simply abort the entire program (including the running VM)
when memory is exhausted, and that it uses them in many places.

Thank you. The rubbish that some people are posting to this thread is quite
unbelievable.

 

[Kelsey Bjarnason]

Some of the functions - including the one mentioned - explcitly say that
they *don't* exit. Not to mention the clue in the function name.

Now obviously it's a flaw in the documentation that it talks as if all
the functions aborted, but you don't want to rest your whole argument on
that flaw, do you? One mistaken statement, and you damn the whole
thing?

Let's ponder the whole issue a moment.



Let's look at an example, g_completion_complete. As follows:

GList* g_completion_complete (GCompletion *cmp,
const gchar *prefix,
gchar **new_prefix);

Parameters are:

cmp - a "completion context" containing the list of objects to be tested
for matches and a few other odds and ends

prefix - the prefix string (what to match on)

new_prefix - a pointer to pointer to gchar which is actually a "return" -
NULL on no match, or the longest prefix common to all matching items

And it returns a GList *, the list of strings which begin with prefix.


So let's start the process:

GCompletion *cmp = g_completion_new( NULL );

This says to allocate a new completion context object; the NULL can be
replaced with a function pointer if you're not examining strings.

Okay, problem: it does *not* document what happens when there's
insufficient memory for an allocation; does it return NULL? Or does it
not return, just kills the app? No clue, as it's not documented there.

Okay, fine, we have a successful GCompletion object. Or we don't, but
from the documentation there, there's no way to tell whether we do or
not, so we'll just assume we do. So we use it, in the call to
g_completion_complete.

In g_completion_complete, it apparently allocates the longest common
prefix, handing it back as new_prefix. Sounds good. And if the
allocator fails here, it does what, precisely?

It says it "returns" (in new_prefix) NULL if no matches were found, or
the longest common prefix if one or more was, but it says _nothing_ about
what it does on allocation failure.


So, what *does* happen on allocation failure? From the documentation of
_these_ functions, there is absolutely no way to tell. If they use the
"report NULL on failure" allocator, they don't document NULL as a
possible result. If they use the "crash and burn" allocator, it makes
sense they don't document what happens on allocation failure, as it's
documented elsewhere - it crashes and burns.

It seems likely they use the crash-and-burn allocator, given the complete
lack of any documentation about returning allocation failures in the
functions.

Yet they have this nifty new "report on error" allocator, too. Why?

No, seriously, why? If they "fix" the library functions, existing code
using them is going to puke, because it's built on the guarantee that
allocation failure means death.

New library functions could use the new allocator, sure, but now you've
got an inconsistent library: any two functions which return a pointer may
involve allocation, but which one dies on failure, which one reports NULL?

Maybe you duplicate the _entire_ library? Have the original function
g_completion_complete, but add a gs_completion_complete, with "gs" for
"glib safe"?

That's a hell of a lot of work, and I don't expect it to happen.


The very fact that there even exists a "different" allocation function,
one which _does_ return NULL on failure, actually makes the situation
worse, not better, as it means you're now faced with two conflicting
error handling strategies, one bad, the other potentially forcing you
into enormous rewrite efforts.

So yeah, I *do* think a flaw in the documentation warrants this sort of
damning, as that flaw reveals a much deeper flaw, one which makes using
the library even *less* safe, *less* reliable, than if the inconsistency
didn't exist and the single absolute rule were adhered to.

Neither is good, this is simply worse.

 

[Kelsey Bjarnason]

[snips]

Maybe for you. I, on the other hand, can be running three VMs eating up
a bit over three quarters of the memory, Lotus Notes, another email
client (yes, I have good reasons for using multiple email clients
simultaneously) and several other applications.

Likewise. This box has 2GB RAM installed, and it runs fairly often at
near capacity.

Not for me they don't, if I find any application aborting simply due to
lack of memory it will be replaced.

Well, an out of memory error is one thing; I don't mind if my word
processor tells me it can't allocate enough memory for another document.
I *would* be a tad upset if on doing something as trivial as pulling down
a menu, it ran out of memory and died, taking my edits with it.

 

[Malcolm McLean]

He persists in ignoring the simple example: a word processor with three
documents open and insufficient resources to open a fourth. When the
open fails - due to inability to allocate resources for it - simply
saying "can't allocate memory to open a new document" and returning to
the existing three poses _zero_ danger except in extreme cases.

The fact I can't allocate 3MB - or even 30K - for the new document does
*not* mean I cannot allocate 30 bytes for a FILE structure to save the
next already opened document, unless I use his xmalloc, in which case it
*does* mean I can't allocate 30 bytes for a FILE structure to save the
existing document, as the application has died.

/*
This one can fail, quite forseeably. Return null on out of memory and set
the err flag.
*/
Document *LoadDocument(char *path, int *err);

now to call it

Document *newdoc;

newdoc = LoadDocument(userEntry, &err);
if(newdoc == 0)
{
switch(err)
{
case OUT_OF_MEMORY:
SendOOMMessage();
abortLoad();
break;
case NO_SUCH_PATH:
/* similar processing here */
case DOCUMENT_CORRUPT:
/* similar here */
}
}

See how much code this is generating? We need effectively an entire function
to manage the call to LoadDocument() and its error returns. We also need to
define symbol after symbol. For such a fundamental function, this is
justified. But you are saying that every function must be like that.

The fact you cannot make a program absolutely proof against all failure
conditions is insufficient justification for simply throwing in the towel
without making even the simplest attempt.

You don't call xmalloc() for a 3MB allocation. You call it for your trivial
allocations, like filenames to pass to fopen. If you can't generate a 100
byte path to hold a filename, and user cannot shut down anything else to
supply more memory, then unless your system is rocksolid - i.e. not Windows,
not Linux - it's going to seize up anyway.
You make simple attempts to recover on allocations that are likely to fail.
You do not make complex attempts to recover on allocations that are most
unlikely to fail, unless developments costs are not an issue for you, and
unless you have such good error-checking systems that the added complexity
is not more likely to add bugs to the program, thus giving data less safety.
In other words, you do not over-engineer your solutions.

 

[Richard Bos]

Oh, good God. They didn't. Tell me they didn't.

Sure they did. It's all part of the drive to make the Lyenooks
Ekspeeriens more familiar to Windows users.

Richard

 

[Malcolm McLean]

(e-mail address removed) wrote:
Why does g_malloc and g_try_malloc ever arise? Is typing:

if (!(ptr = malloc(N * sizeof *ptr))) fixitup();

so hard? Feel free to play with the fixitup() function call. You
might want to pass it the size required and a pointer to ptr.

The problem is, what is fixitup() going to do? Probably it needs access to
objects within the function's local scope to clean up. So it cannot be a
general-purpose function.
It is a few lines of custom code, which probably destroy the object being
created, and pass an error condition to the caller. The code is only of the
slightest use if caller also has a fixitup() function, which has similar
characterisitics. Eventually you will reach a level at which the error can
be handled, assuimg that the program is a structured one with a strict
hierarchy of funcntions.

 

[Richard Tobin]

Linux VMWare uses Gtk for its UI, isn't it funny? That "out of resources
message" corresponds to a failed g_try_malloc(), so what?

[/QUOTE]

According to the glib memory allocation documentation page, this cannot
be true, as the page clearly states, right near the top, that allocation
either *works* or the application *terminates*.

Some of the functions - including the one mentioned - explcitly say
that they *don't* exit. Not to mention the clue in the function name.

Now obviously it's a flaw in the documentation that it talks as if all
the functions aborted, but you don't want to rest your whole argument
on that flaw, do you? One mistaken statement, and you damn the whole
thing?

Given that this has already been pointed out, I am starting to doubt
your good faith.

-- Richard

 

[ymuntyan]

[snips]

It is just impossible to ask for a smaller block in that situation. But
if you know that you will be able to work with a smaller block then you
don't call g_malloc in the first place, you call g_try_malloc instead.

Doesn't matter. The documentation, as you can see for yourself,
*clearly* states... no, here, I'll quote:

"If any call to allocate memory fails, the application is terminated.
This also means that there is no need to check if the call succeeded."

There's no point using g_try_malloc, as the documentation tells you,
right up front, that if it fails to allocate the memory, the program will
be terminated.

Now I got it. You don't care about how the library actually
works, you only need one bogus sentence from docs to be able
to talk about how the library works (and ignore what you're
told repeatedly since it wouldn't let you talk about good
software design). Here's a puzzle for you: the same docs say

g_try_malloc (gulong n_bytes);
Attempts to allocate n_bytes, and returns NULL on failure.
Contrast with g_malloc(), which aborts the program on failure.

A contradiction. Now what?

Gosh. It doesn't matter what it does, "because the documentation..."
Pathetic. Next you will quote wikipedia or what?
How about you start talking about how glib uses 32 bits
gsize? It's there in the docs!

 

[ymuntyan]

we seem to be going round in circles. Ok I thought you were proposing
an xmalloc() that wrappered malloc(). If malloc() fails it calls
abort().

Do you what? I wasn't proposing xmalloc(), nor was I talking
about xmalloc().

If such an xmalloc() is used to replace a call to malloc() then the
caller
can take no recovery action if malloc() fails.

True. This conditional statement, that is.

Recovery includes such things as saving important data then
terminating
the application.
Right.



ok "test" could either mean "test the value" eg. with an if
*or* it could mean "run a test". So where do these segfaults come
from.
If every malloc is followed by a test(1)?

Segfault is here:

if (!(ptr = malloc(size)))
{
// Right here, in the untested code.
// Not because you access NULL, no,
// just a plain normal bug.
}

not if you use xmalloc()

Right, not if you use xmalloc() (the version which
aborts immediately).

xmalloc() does
Yes.



xmalloc() crashes on error. It always crashes on error. Therefore you
cannot recover from an error if you use xmalloc().
Right.



it might not be your application that leaks. Don't you
run your software in multitasking OSs?

I didn't realize that "long run" was irrelevant, sorry.

you apparently. So do you agree that an application
can do more than crash when memory runs out. You
contradict yourself.

Do I? Care to quote?

there could be. Look I work on comms applications.
Packets get dropped and they then get repeated. A small embedded
system at the end of a crappy line *might* quite reasonably
discard a packet if it was short of memory.

Sorry, I wasn't talking about "comms applications".

good grief. What is a "desk top system". Excell? Word? A Network
Management
System. is it ok for GUIs to crash? That's about the only thing I can
see as
reasonable. And that's a pure GUI that doesn't hold any user data.

We seem to be agreeing that glib cannot be safely used for most
programs.

glib? Weren't you talking about xmalloc()?

 

[Kelsey Bjarnason]

[snips]

If g_allocation_function() may return NULL

It cannot. See the documentation at the top of the page. I'll quote it
for you:

"If any call to allocate memory fails, the application is terminated.
This also means that there is no need to check if the call succeeded."

Any glib allocation function which returns NULL on failure violates the
guarantee given in the documentation.

then it's your bug.

Actually, it's theirs, as it violates the guarantee they make: if *ANY*
call to allocate memory fails, the application is terminated.


Yes, I know they document other behaviours, and that's kinda the point.
You can't trust the documentation, for starters, and given their initial
design decision to abort on allocation failure, the existence of this
NULL-returning allocator is nothing more than a flashing red light,
warning that they've got a massive inconsistency either happening or
about to happen, with some functions aborting on error, others returning
NULLs and consistency not even a concept to be considered.

It's bad, any way you slice it, and apparently getting worse.

 

[Kelsey Bjarnason]

Kelsey Bjarnason wrote, On 30/01/08 11:43:

[snips]

Maybe for you. I, on the other hand, can be running three VMs eating
up a bit over three quarters of the memory, Lotus Notes, another email
client (yes, I have good reasons for using multiple email clients
simultaneously) and several other applications.

Likewise. This box has 2GB RAM installed, and it runs fairly often at
near capacity.

I'm glad I'm not the only one filling the RAM of a 2GB machine.
Actually, one of my colleagues is also finding the 2GB of RAM is rather
tight.

Depends what you do with it.

I do development on it. Sadly, at the moment, I'm doing Windows-based
development, meaning I'm running XP in a VM window. Slow as snail snot,
but it works - but it does suck ram.

Besides that, I've got Apache, MySQL, Postgres, leafnode, I often have
postfix/cyrus/amavis/spamassasin plus their attendant doo-dads (pyzor,
razor, etc, etc) sucking up who knows how much memory, plus KDE which
usually ends up running umpteen apps - kmail, moz, pan, kdevelop, amarok,
kopete, skype and more, plus OpenOffice unless I'm really cramped for
space.

Adds up.

It telling me it cannot open another windows is fine, killing the word
processor and all the other documents would not be.

Indeed. It'd make me want to pay a little visit to the developer, and
bring along my buddy Guido and his pet iguana.

I *expect* to find I
don't have the memory to do some things sometimes and have to close down
one of the many things I am doing.

Zackly. It's one thing to run out of memory; it's another entirely to
crash and burn because of it.

 

[ymuntyan]

[snips]

Segfault is here:

if (!(ptr = malloc(size)))
{
// Right here, in the untested code.
// Not because you access NULL, no,
// just a plain normal bug.
}

Why would you write your code that way?

Let's see:

FILE *fp = fopen...

if ( ! fp )
fwrite( ..., fp );

Do you generally write to (or read from) files you can't open? No? So
why would you write to or read from a pointer you can't allocate?

As someone else said, it's a habit you develop. You write the code to
allocate the data, you check the allocation, and you do not use the
variable _at all_ in your error code. Why would you? It has no
meaningful value; you _know_ it's NULL, there's nothing to test or
display or process.

Who said anything about writing or reading from NULL?

It is just not that hard to do this stuff. People do it day in, day out.



What long run? Even if your app never leaks a single byte, it can still
face allocation failures due to _other_ apps consuming available
resources. That can happen ten milliseconds from now or ten days from
now.

Right. Which is why "long run" is irrelevant.

They are, to all intents and purposes, the same thing as pertains to the
discussion. Each has the same fundamental flaws.

No, they are not the same.

 

[ymuntyan]

[snips]

Now I got it. You don't care about how the library actually works, you
only need one bogus sentence from docs to be able to talk about how the
library works

You *almost* have it right, but not quite.

The documentation tells me how the library is _supposed_ to work. In the
case of glib, it is _supposed_ to terminate the application on allocation
failure. In fact, it says so, right there in the documentation.

It *also* says that there is no point even *checking* whether an
allocation succeeded or not, because it _will_ succeed or the app will
terminate.

Thus, if I write code according to how glib is _supposed_ to work, I will
find myself, sooner or later, in a situation such as this:

type_t *ptr = g_allocation_function(...);
/* no check for ptr being NULL */
memset( ptr, 0, size ); /* or some equivalent */

If g_allocation_function() may return NULL, then it's your
bug. You can blame docs if you like. But then you should
have read the docs for that particular function, no? Docs
which say that it may return NULL.

If glib does *not* work as documented, then it *can* return a NULL, and
the code written to the documented guarantee that a NULL will not be
returned is now going to fail.

So it either fails on allocation failure, which, frankly, I find
unacceptable, or it violates its own documented guarantees, breaking the
code that uses it. I don't see how either of these makes it a good idea
to use this library.

Heh. Man, you have a super argument why it's not a good idea to use
any open source library. As you like, I don't care. I was talking
about another issue: glib memory allocation. What you think
about using it because of how you like the documentation is a
different
story, more suitable for comp.linux.advocacy or alt.linux.sucks or
whatever.

Yevgen

 

[Flash Gordon]

Ben Pfaff wrote, On 30/01/08 03:15:

I have first-hand knowledge that VMware Workstation, Server, and
ESX Server contain wrappers over malloc() and related functions
that simply abort the entire program (including the running VM)
when memory is exhausted, and that it uses them in many places.

Note that I stated "out of resource message". This is because
1) I have not bothered to note the exact message
2) I think it is something along those lines

It is also possible that VMware has changed since I am using Workstation
version 6 latest build. Our Server and ESX machines have enough physical
RAM for all the machines they run so there is no reason for them to run out.

Of course, I might have been lucky and only hit the instances where it
is trapped. However if it starts aborting on my I will be very quick to
complain to VMware and also investigate the alternative.

Lotus Notes has actually reported an "out of memory" error on several
occasions. Again, for all I know there could be lots of places it does
not check and I've been lucky.

 

[Flash Gordon]

It doesn't do the same as glib. But it doesn't guard
event processing against oom exceptions either. Which
in effect means abort (again, you can catch the exception,
save data, and quit nicely, as you can do with glib; just
can't continue working as if nothing happened). Since
"ALL malloc calls are checked!"

People have not been saying "continue as if nothing happened".

Don't forget to check their code.

I don't have time. I'll use whatever I choose until I'm aware of a
severe enough problem to make me change or I find something better.

Oh, and while you're at
it, don't use X. It aborts on oom.

There are several reasons I am not fond of X already.

 

[Flash Gordon]

Kelsey Bjarnason wrote, On 30/01/08 11:43:

[snips]

Maybe for you. I, on the other hand, can be running three VMs eating up
a bit over three quarters of the memory, Lotus Notes, another email
client (yes, I have good reasons for using multiple email clients
simultaneously) and several other applications.

Likewise. This box has 2GB RAM installed, and it runs fairly often at
near capacity.

I'm glad I'm not the only one filling the RAM of a 2GB machine.
Actually, one of my colleagues is also finding the 2GB of RAM is rather
tight.

Well, an out of memory error is one thing; I don't mind if my word
processor tells me it can't allocate enough memory for another document.

It telling me it cannot open another windows is fine, killing the word
processor and all the other documents would not be. I *expect* to find I
don't have the memory to do some things sometimes and have to close down
one of the many things I am doing.

I *would* be a tad upset if on doing something as trivial as pulling down
a menu, it ran out of memory and died, taking my edits with it.

Indeed.

 

[ymuntyan]

[snips]

If g_allocation_function() may return NULL

It cannot. See the documentation at the top of the page. I'll quote it
for you:

"If any call to allocate memory fails, the application is terminated.
This also means that there is no need to check if the call succeeded."

Any glib allocation function which returns NULL on failure violates the
guarantee given in the documentation.

Right, the quoted sentence is not completely correct. It fails
to say that it refers only to certain functions, not all of them.
Docs bug, absolutely.

But, it's easy to avoid a bug in your application caused
by that bug, once you actually do read docs. If you can
think, you'll notice the obvious contradictions. If you
choose to blindly rely on one sentence, it's your choice.

Actually, it's theirs, as it violates the guarantee they make: if *ANY*
call to allocate memory fails, the application is terminated.

Yes, it is a bug in documentation.

Yes, I know they document other behaviours, and that's kinda the point.
You can't trust the documentation, for starters, and given their initial
design decision to abort on allocation failure, the existence of this
NULL-returning allocator is nothing more than a flashing red light,
warning that they've got a massive inconsistency

Massive inconsistency, huh? This is not true. What *is* there
is a wrong sentence in docs (which isn't even too bad once
you are *willing* to read), and you take that every glib user
is as dumb or hostile as you. *You* certainly shouldn't use
glib, since you won't read documentation of the functions you
are using, and you won't try to apply common sense when using
its API.

either happening or
about to happen, with some functions aborting on error, others returning
NULLs and consistency not even a concept to be considered.

BS. Imaginary inconsistency.

It's bad, any way you slice it, and apparently getting worse.

Heh, it's getting worse now. God save humanity!

Yevgen