xmalloc string functions

[ymuntyan]

Malcolm McLean wrote, On 29/01/08 10:06:





It did not cause me significant extra work. The most likely reason for
the difference is that I designed the entire system knowing that
out-of-resource errors *do* occur so it was part of the entire design
concept rather than an extra I had to try and fit in.

This is a very good point. In those stupid GUI applications
out-of-memory errors occur less often than crashes caused
by usual bugs. In that update-mime-database application
memory errors never occur in observable practice. For
many applications, out-of-memory error is a disaster
(not for Lotus, I admit). Yet they do work.

Oh, and my real-time applications degraded gracefully rather than
failing on out-of-time errors as well. Again it was a case of
considering how to do the job properly during the design of the
application not during coding.


Well, I would not have used your BabyX library anyway, but now I have
even more reason to avoid it.

I have no idea what BabyX does, and I have no idea how good it is,
but if the only bad thing about it is xmalloc(), then it's better
than both qt and gtk.

So, you have a reason to avoid QT, Gtk, Python, shell. Are you
using any unix? Just curious.

Um, I need to tell Kelsey about QT, it must make him swith
to windows immediately.

Yevgen

 

[CBFalconer]

The point is that they don't. If fopen() fails, there is nothing
to recover from. Though if all your application does is fopen(),
then you can safely abort() when fopen() fails.

Very good. After Josephine Q Worker has typed in 16 file names for
her xyz app, she mistypes the 17th. xyz immediately aborts. No
chance to re-enter. xyz is even more sloppily written, because it
has gobbled in data from the first 16 files, destroyed the files,
and now will destroy the data. Now Big Bad Boss can yell at
Josephine for impardonable carelessness. He may even fire her.

Maybe xyz isn't quite that bad, it just collects filenames and only
does anything after it has them all opened. However, it aborted,
so Josy is re-entering.

 

[CBFalconer]

.... snip ...

I don't see how an allocation failure can corrupt data on a well
written program. At most a malloc() failure stops you doing any
more work, but how can it corrupt what already exists? Incomplete
data, maybe, but corrupt?

Point 1: A program that aborts on malloc failure is not a well
written program.

Point 2: Well, point 1 is enough.

 

[CBFalconer]

.... snip ...

Err, for allocations which can "normally" fail, for those big
allocations, you don't use g_malloc and use g_try_malloc instead.
Which can return NULL. What you don't handle in a normal way
(that is you don't check return value because it won't be NULL,
let's not start again about losing user data) is failure on
allocation of a small chunk of memory, like when you are making
a string, or appending a node to a list, or creating a widget.

Why does g_malloc and g_try_malloc ever arise? Is typing:

if (!(ptr = malloc(N * sizeof *ptr))) fixitup();

so hard? Feel free to play with the fixitup() function call. You
might want to pass it the size required and a pointer to ptr.

 

[CBFalconer]

G_no, g_I g_don't.

g_ag, G_ag, g_Ag, G_Ag, G_AG.

 

[Malcolm McLean]

I have no idea what BabyX does, and I have no idea how good it is,
but if the only bad thing about it is xmalloc(), then it's better
than both qt and gtk.

It's a minimal X tookit, so far unreleased. May never be released, because I
can't give it any kind of priority.

 

[Flash Gordon]

g_malloc() is not a raw wrapper around malloc. You can get into
it, and do stuff where malloc() fails. And *then* return NULL
to glib, and it will abort (or quit application yourself). Or
even fancier, return memory from some emergency pool, and shut
down the application later, as you would after a signal is caught.
What your imagination allows.

This is the first time I've seen it clearly stated that you can do this
rather than simple assertions that despite the document saying it will
abort. Still makes it rather difficult to do the nice simple options of
requesting a smaller block instead (as there is no mechanism to signal
you have done that) and so limits your options when building on top of
code built on glib.

Linux VMWare uses Gtk for its UI, isn't it funny? That "out of
resources message" corresponds to a failed g_try_malloc(), so
what? Yes, you can call g_try_malloc.

Actually I get it on the Windows version of VMWare. The only Linux
versions of it I use are server and ESX.

As to "frequently", here is another piece of usage data: when
applications crash here, they crash because of random bugs,

If an application crashes regularly I bin it. Even if it only crashes
when I am tight on resources, because when I am tight on resources it is
because I am doing a lot of heavy duty work which I don't want to loose.

never because of malloc failure. Does that mean malloc() never
fails? I mean, yeah, Lotus is great.

No, it is horrible. However, that is for rather different reasons.

So what? What losses do
actually happen because g_malloc() will abort by default when
malloc() fails? How often, how much?

For me not very often because if they did then I would stop using them
and find replacements. I have too much work to do to use any application
which is not robust under the conditions I need to use it, which
includes running out of memory.

 

[Flash Gordon]

This is a very good point. In those stupid GUI applications
out-of-memory errors occur less often than crashes caused
by usual bugs.

Maybe for you. I, on the other hand, can be running three VMs eating up
a bit over three quarters of the memory, Lotus Notes, another email
client (yes, I have good reasons for using multiple email clients
simultaneously) and several other applications.

In that update-mime-database application
memory errors never occur in observable practice.

I'm frequently tight on memory, so...

For
many applications, out-of-memory error is a disaster
(not for Lotus, I admit). Yet they do work.

Not for me they don't, if I find any application aborting simply due to
lack of memory it will be replaced. It will also be replaced (apart from
testing purposes if a customer uses it) if it crashes regularly for any
other reasons.

I have no idea what BabyX does, and I have no idea how good it is,
but if the only bad thing about it is xmalloc(), then it's better
than both qt and gtk.

It also sounds like a mess based on the descriptions so far given.

So, you have a reason to avoid QT, Gtk, Python, shell. Are you
using any unix? Just curious.

I use AIX, Linux and several versions of Windows. I used to use SCO but
managed to get that dropped. I do C, the odd bits of Java and Perl,
XSLT, shell scripting, odd bits of SQL, some DBA work, systems
administration...

Then there are all the languages and systems I no longer use for various
reasons.

Um, I need to tell Kelsey about QT, it must make him swith
to windows immediately.

If QT does the same as glib then I'm sure people will say that is
braindead as well. I might switch back to one of the lighter weight GUIs
on my personal notebook anyway.

 

[Ian Collins]

I'm frequently tight on memory, so...

Buy some more!

Not for me they don't, if I find any application aborting simply due to
lack of memory it will be replaced.

I guess I, like so many others have become lazy and just add more memory.

It's an unfortunate reality these days that a lot of desktop
applications assume the user has enough memory and simply don't bother
with contingencies.

 

[ymuntyan]

(e-mail address removed) wrote, On 29/01/08 22:14:





This is the first time I've seen it clearly stated that you can do this
rather than simple assertions that despite the document saying it will
abort.

Well, I tried to say that (and actually did say).

Still makes it rather difficult to do the nice simple options of
requesting a smaller block instead (as there is no mechanism to signal
you have done that) and so limits your options when building on top of
code built on glib.

It is just impossible to ask for a smaller block in that
situation. But if you know that you will be able to work
with a smaller block then you don't call g_malloc in the
first place, you call g_try_malloc instead. You can't
work with a block smaller than sizeof(YourStruct) if you
need YourStruct on heap, that's where you call g_malloc.
You don't request a smaller block if malloc(sizeof(YourStruct))
failes, do you?

Yevgen

 

[ymuntyan]

(e-mail address removed) wrote, On 29/01/08 22:56:





Maybe for you. I, on the other hand, can be running three VMs eating up
a bit over three quarters of the memory, Lotus Notes, another email
client (yes, I have good reasons for using multiple email clients
simultaneously) and several other applications.


I'm frequently tight on memory, so...


Not for me they don't, if I find any application aborting simply due to
lack of memory it will be replaced. It will also be replaced (apart from
testing purposes if a customer uses it) if it crashes regularly for any
other reasons.





It also sounds like a mess based on the descriptions so far given.


I use AIX, Linux and several versions of Windows. I used to use SCO but
managed to get that dropped. I do C, the odd bits of Java and Perl,
XSLT, shell scripting, odd bits of SQL, some DBA work, systems
administration...

Then there are all the languages and systems I no longer use for various
reasons.


If QT does the same as glib then I'm sure people will say that is
braindead as well.

It doesn't do the same as glib. But it doesn't guard
event processing against oom exceptions either. Which
in effect means abort (again, you can catch the exception,
save data, and quit nicely, as you can do with glib; just
can't continue working as if nothing happened). Since
"ALL malloc calls are checked!"

I might switch back to one of the lighter weight GUIs
on my personal notebook anyway.

Don't forget to check their code. Oh, and while you're at
it, don't use X. It aborts on oom.

Yevgen

 

[Ben Pfaff]

I also get VMWare reporting an out of resources message a lot
more often than it crashes on me (I've not bothered verifying
what the resource shortage is, but the only resource I am
normally short of is memory). So there you go, real apps that
really do handle out-of-memory situations.

I have first-hand knowledge that VMware Workstation, Server, and
ESX Server contain wrappers over malloc() and related functions
that simply abort the entire program (including the running VM)
when memory is exhausted, and that it uses them in many places.

 

[Kelsey Bjarnason]

[snips]

Not necessarily corrupt. There are many cases where the incomplete data
set can be saved and processed at another program run.

He persists in ignoring the simple example: a word processor with three
documents open and insufficient resources to open a fourth. When the
open fails - due to inability to allocate resources for it - simply
saying "can't allocate memory to open a new document" and returning to
the existing three poses _zero_ danger except in extreme cases.

The fact I can't allocate 3MB - or even 30K - for the new document does
*not* mean I cannot allocate 30 bytes for a FILE structure to save the
next already opened document, unless I use his xmalloc, in which case it
*does* mean I can't allocate 30 bytes for a FILE structure to save the
existing document, as the application has died.

An allocation failure usually leaves the program with incomplete data.

Or complete data from an incomplete set, as in all the data for three
documents, none for the fourth.

Yes. Really well engineered programs are apparently rare but that's no
reason to get discouraged into writing slack code.

Nor is *imperfect* error handling a justification for *no* error
handling. Even if you only correctly handle, say, half the failures,
that's still data saved in half the cases, rather than none.

Meanwhile, one wonders why he persists in this bizarre notion that such
handling is terribly difficult to write. It's not. There are, of
course, limits to it: if memory is exhausted, the file system is full and
you can't alert the user or save, you're pretty much screwed.

The fact you cannot make a program absolutely proof against all failure
conditions is insufficient justification for simply throwing in the towel
without making even the simplest attempt.

 

[Kelsey Bjarnason]

Malcolm McLean said:



Yes, but you can design the apparatus in software, and you can design it
in such a way that it fails only when you want it to, and then you can
design your tests to make it fail at each point in turn. This is not
even particularly difficult to do. (I have done it myself, so how hard
could it be?)

Yeah, if *you* can do it, *anybody* can! <snicker>

 

[Kelsey Bjarnason]

[snips]

Yes, but it also has functions that don't do that.

Right up top, on the page, it says clearly that any failure to allocate
results in the application being terminated, and thus there is no point
even *bothering* with code to see if the allocation failed.

You know, no point in a try/catch block. No point in if ( ptr ==
NULL ). No point in anything, as - according to the notes on the page -
*none* of the allocation routines survive an allocation failure, the app
simple terminates.

According to the documentation, there are exactly _zero_ functions where
this is not the case.

g_try_new()
...
Attempts to allocate n_structs elements of type struct_type, and
returns NULL on failure. Contrast with g_new(), which aborts the
program on failure.

This is in direct contradiction to the notes heading the allocation
documentation, which clearly states the program _will_ terminate on
allocation failure. Period. In all cases.

Since I can't trust them to write sensible code, and they now apparently
have documentation with conflicting statements, I'm supposed to believe
they did the right thing here?

Losing a user's data is obviously undesirable. But it happens even with
the most careful programming.

Obviously, there are limits to any error handling mechanism. If a
required allocation has failed and you can't continue, you try to save
the data. If the file system is full, try to alert the user. If the
alert mechanism fails, well, you can't continue, can't save, can't fix
the problem, you're screwed.

This is _not_ justification for simply throwing in the towel at the first
sign of a problem, without even trying.

Is the rate of loss that results from
these functions significant compared with that from power failures,
hardware faults, user error and so on?

Yes, because we cannot control those things. Nothing in our code can
prevent a power failure. Nothing in our code can say "Oops, the machine
has no power, better save the data". (I'm ignoring the odd case where
there's a UPS and it alerts the system).

We _can_ however detect an allocation failure and take steps to ensure a
proper, graceful recovery from this.

Is it outside the range of normal risks that one takes every day, such
as using a frying pan without gloves on?

You tell me a way, in C, to detect such a situation and warn about it or
prevent it, I'm happy to give it a go. What implementation generates
code for your frying pan?

 

[Kelsey Bjarnason]

[snips]

g_malloc() is not a raw wrapper around malloc. You can get into it, and
do stuff where malloc() fails. And *then* return NULL to glib, and it
will abort (or quit application yourself).

Sounds good. Let's see.

I have a function which has created an object to store, oh, a page of
modified text.

It calls my UI handling functions which, in turn, create a window,
assorted doo-dads and so forth and attempt to create a usable display
environment.

About 10 levels down the call tree, these functions invoke a glib call
which allocates memory... and fails.

If glib were properly designed, it would pass back the allocation
failure, which would propagate up the tree to the original function, the
one with the modified data, which could then do any number of things,
including saving the document while tagging the modifications as
"provisional" before exiting, such that on the next run, the app can
display them as unconfirmed edits.

Unfortunately, according to the glib documentation, this doesn't happen.
According to the documentation, the allocations work or die and as a
result there's no reason to even check if allocation succeeded.

So my code never sees the returned NULL, or equivalent failure notice,
never gets a chance to save the edited data, poof, it's gone.

Linux VMWare uses Gtk for its UI, isn't it funny? That "out of resources
message" corresponds to a failed g_try_malloc(), so what?

According to the glib memory allocation documentation page, this cannot
be true, as the page clearly states, right near the top, that allocation
either *works* or the application *terminates*. If vmware is reporting
an allocation failure _and not terminating_ it cannot, therefore, be
using the glib allocation routines.

Here's the exact quote from the page:

"If any call to allocate memory fails, the application is terminated."

Terminated. Not recovered. There is *no* ability to recover, period,
the application is terminated. VMWare recovers, therefore the failure is
not in allocation, or, if it is, it is _not_ using glib's allocation
functions.

Also from that same page:

"This also means that there is no need to check if the call succeeded."

Right there, it says there's no reason whatsoever to check for NULLs, to
use a catch or any other sort of attempt to detect allocation failures.
Why? Because glib doesn't do that, it simply terminates the application
on allocation failure.

No recovery. No reporting. No way to stop it. No "g_try_new reports
failure without exiting". Nope, according to their documentation, it
either works or the application dies. Period.

VMWare runs out of resources and _recovers_? Then it *cannot* be using
the glib allocation functions. Period.

Well, either that, or the glib documentation lies through its teeth, and
in fact allocation failure does *not* result in termination, there *is* a
point to checking the return, but then, if the documentation lies to the
developer, how much can you trust it, or the code?

As to "frequently", here is another piece of usage data: when
applications crash here, they crash because of random bugs, never
because of malloc failure.

My apps don't crash as a result of malloc failure either. They detect it
and handle it appropriately, instead. Something not possible if one uses
the glib allocation routines.

Does that mean malloc() never fails? I mean,
yeah, Lotus is great. So what? What losses do actually happen because
g_malloc() will abort by default when malloc() fails? How often, how
much?

It doesn't matter if it's only once, if that once is critical. No, you
cannot prevent all failures and losses, we know this. You *can*,
however, prevent many losses, and the notion of designing a library where
the response to such an error is to simply give up and throw in the towel
is frankly irresponsible.

 

[Kelsey Bjarnason]

[snips]

It is just impossible to ask for a smaller block in that situation. But
if you know that you will be able to work with a smaller block then you
don't call g_malloc in the first place, you call g_try_malloc instead.

Doesn't matter. The documentation, as you can see for yourself,
*clearly* states... no, here, I'll quote:

"If any call to allocate memory fails, the application is terminated.
This also means that there is no need to check if the call succeeded."

There's no point using g_try_malloc, as the documentation tells you,
right up front, that if it fails to allocate the memory, the program will
be terminated.

 

[Flash Gordon]

Ian Collins wrote, On 30/01/08 01:11:

Buy some more!

This is a company machine, not my personal machine. When it was bought
getting any more RAM in it would have cost an extra 400UKP or so and I
was not going to get away with that.

I guess I, like so many others have become lazy and just add more memory.

Not always possible either because you don't own the purse strings (a
company machine) or the cost is prohibitive, or the machine already has
the maximum the hardware supports.

I could probably now get the company to upgrade the machine from 2GB to
4GB, but then I would probably just get the machine doing more things
and still run out of memory.

It's an unfortunate reality these days that a lot of desktop
applications assume the user has enough memory and simply don't bother
with contingencies.

Fortunately not all and fortunately the ones I need to use have not
collapsed due to simple memory exhaustion.

 

[Kelsey Bjarnason]

[snips]

Segfault is here:

if (!(ptr = malloc(size)))
{
// Right here, in the untested code.
// Not because you access NULL, no,
// just a plain normal bug.
}

Why would you write your code that way?

Let's see:

FILE *fp = fopen...

if ( ! fp )
fwrite( ..., fp );

Do you generally write to (or read from) files you can't open? No? So
why would you write to or read from a pointer you can't allocate?

As someone else said, it's a habit you develop. You write the code to
allocate the data, you check the allocation, and you do not use the
variable _at all_ in your error code. Why would you? It has no
meaningful value; you _know_ it's NULL, there's nothing to test or
display or process.

It is just not that hard to do this stuff. People do it day in, day out.

I didn't realize that "long run" was irrelevant, sorry.

What long run? Even if your app never leaks a single byte, it can still
face allocation failures due to _other_ apps consuming available
resources. That can happen ten milliseconds from now or ten days from
now.

glib? Weren't you talking about xmalloc()?

They are, to all intents and purposes, the same thing as pertains to the
discussion. Each has the same fundamental flaws.

 

[Nick Keighley]

we seem to be going round in circles. Ok I thought you were proposing
an xmalloc() that wrappered malloc(). If malloc() fails it calls
abort().

If such an xmalloc() is used to replace a call to malloc() then the
caller
can take no recovery action if malloc() fails.

Recovery includes such things as saving important data then
terminating
the application.

You missed the meaning of "test".

ok "test" could either mean "test the value" eg. with an if
*or* it could mean "run a test". So where do these segfaults come
from.
If every malloc is followed by a test(1)?

I don't know why not. You can do that.

not if you use xmalloc()

malloc doesn't crash.

xmalloc() does

It rarely crashes, indeed. What are you talking about?

xmalloc() crashes on error. It always crashes on error. Therefore you
cannot recover from an error if you use xmalloc().

So you write the application in a different way. Gtk
toolkit is not for such an application (not for the
application which leaks so much that it runs out of
memory on the long run).

it might not be your application that leaks. Don't you
run your software in multitasking OSs?

Sometimes yes. Who objects?

you apparently. So do you agree that an application
can do more than crash when memory runs out. You
contradict yourself.

There won't be repeat.

there could be. Look I work on comms applications.
Packets get dropped and they then get repeated. A small embedded
system at the end of a crappy line *might* quite reasonably
discard a packet if it was short of memory.

Desktop applications.

good grief. What is a "desk top system". Excell? Word? A Network
Management
System. is it ok for GUIs to crash? That's about the only thing I can
see as
reasonable. And that's a pure GUI that doesn't hold any user data.

We seem to be agreeing that glib cannot be safely used for most
programs.