xmalloc string functions

[Kelsey Bjarnason]

[snips]

That's exactly how xmalloc() works. Except that the user callback can
never say "return null". User must either supply some memory or
terminate the program.

So the user - in this case, the coder using your function - has
absolutely no option to do something as basic as reducing the memory
allocation request and trying again, or scheduling the operation for
later processing and continuing on to some other action.

Yeah, useful, that. Totally aside from the fact it lies about what it
does, then actually fails to even meet the functionality stated in the
lie.

 

[Kelsey Bjarnason]

[snips]

The specifications are too weak to drive implementations.
What does xmalloc() do when unable to allocate memory?

Calls an optionally-user-specified function to figure out how to free up
some memory (eg opening a window to say "Please close another
application", which, of course, assumes there's enough memory to do so),
or, failing that, aborts the application.

Yes, that's right, it's a trap door function: you can enter, but you can
never leave, unless and until the memory is available. No error
reporting, no recovery, no options at all. That is the intent of the
code, as Malcolm has expressed more than once, that you *cannot* recover
from any allocation failure. You get the memory, or you exit. Period.

Does replace()
replace all occurrences, or just the first, or just the last, or what?
What does getquote() do with unbalanced quotes?

And does it do single quotes? Double quotes? Both?

... and massively complicates error handling.

No, no, it makes error handling *easier*, because you don't ever need to
*do* error handling. xmalloc does that for you - by ensuring you get the
memory, or the app aborts. See how wonderful it is? No need for all
that error handling code at the calling point. Also no way to ever do
anything like, oh, reduce the allocation request and try again, or
reschedule the operation for later, etc, etc, etc, but hey, it's not like
any application would ever actually do things like that.

If xmalloc()
can unilaterally terminate the program

Not "can", by design it is *required to*, unless the allocation
succeeds. One or the other: you get the memory, or the app dies.

the whole suite is unusable
except in toy programs.

Except it's of little use in toy apps. Such an app probably doesn't do a
hell of a lot of complex allocations in the first place, so wouldn't
benefit significantly from "easy malloc". A complex app where there may
be thousands or millions of allocations done from hundreds or thousands
of places in the code, now this would benefit from "easy malloc" - if it
were actually useful.

The xmalloc notion presented is of little interest in a toy app, and is
unusable in a real app. One wonders where, exactly, it would actually be
used. One place it won't be used: my code. I actually like to be able
to choose a course of action on allocation failure, not be stuck with
some asinine notion that allocation will succeed or there's no point in
continuing.

 

[Malcolm McLean]

Here are six functions implemented on top of xmalloc(). No C programmer
should have any triouble providing the implemetations, though replace and
getquote are non-trivial.

char *dup(const char *str);
char *cat(const char *str1, const char *str2);
char *catandkill(char *str1, const char *str2);
char *tok(const char *str, const char *delims, char **end);
char *midstr(const char *str, int idx, int len);
char *replace(const char *str, const char *pattern, const char *rep);
char *getquote(const char *str, char quote, char escape, char **end);
char *getline(FILE *fp);

All return strings allocated with xmalloc().

dup() is strdup(), and duplicates a string. cat() concatenates two strings.
catandkill concatenagtes two strings, and frees the first. It also returns
dup(str2) is the first pointer is null. It is designed for building strings
in conveneint loops.
tok() is strtok() that takes a string argument instead of using a global.
midstr() takes out a central portion of the string. replace() is a find and
replace, getquote() retrives the next quoted string, allowing for the quotes
themselves to be escaped. You will have to escape non-quote escaped
characters manually, of course. getline() will be a familiar friend.

I've think we've got something quite powerful here, purely because none of
these functions can ever return null for out of memory conditions. It
massively simplifies string handling.
The library is not terribly efficient and doesn't pretend to be. If you are
doing heavy string processing you might want to look at Paul Hsieh's better
string library, that stores a length parameter with strings. It is meant to
be a few simple functions for easy manipulation.

 

[Eric Sosman]

Here are six functions implemented on top of xmalloc(). No C programmer
should have any triouble providing the implemetations, though replace
and getquote are non-trivial.

char *dup(const char *str);
char *cat(const char *str1, const char *str2);
char *catandkill(char *str1, const char *str2);
char *tok(const char *str, const char *delims, char **end);
char *midstr(const char *str, int idx, int len);
char *replace(const char *str, const char *pattern, const char *rep);
char *getquote(const char *str, char quote, char escape, char **end);
char *getline(FILE *fp);

All return strings allocated with xmalloc().

dup() is strdup(), and duplicates a string. cat() concatenates two
strings. catandkill concatenagtes two strings, and frees the first. It
also returns dup(str2) is the first pointer is null. It is designed for
building strings in conveneint loops.
tok() is strtok() that takes a string argument instead of using a
global. midstr() takes out a central portion of the string. replace() is
a find and replace, getquote() retrives the next quoted string, allowing
for the quotes themselves to be escaped. You will have to escape
non-quote escaped characters manually, of course. getline() will be a
familiar friend.

The specifications are too weak to drive implementations.
What does xmalloc() do when unable to allocate memory? Does
replace() replace all occurrences, or just the first, or just
the last, or what? What does getquote() do with unbalanced
quotes? Does getline() keep or discard the '\n', and what does
it do on an I/O error? (See the recent thread on design of line-
input functions to see how divergent people's opinions are on
how this "familiar friend" should behave.)

I've think we've got something quite powerful here, purely because none
of these functions can ever return null for out of memory conditions. It
massively simplifies string handling.

... and massively complicates error handling. If xmalloc()
can unilaterally terminate the program, the whole suite is
unusable except in toy programs. If there's a fancier framework,
all functions that use the suite or that call other functions
that might use the suite need to be plugged in to the framework.

Design consists of more than writing a few declarations.

 

[Kelsey Bjarnason]

[snips]

Exactly. Which is what these string functions will do. They will never
return null for out of memory conditions.

Rendering them instantly useless for any real-world code. You know, code
where something as trivial as an allocation failure does *not*
necessarily imply a requirement to abort, code where if an allocation
fails and is detected, can be dealt with any number of ways, up to and
including shrinking another allocated region to free up space so the new
allocation can succeed.

The quote character is passed,

So if I have both single and double quotes included, it doesn't work?
Not terribly useful, particularly in a function whose sole purpose is to
get quotes.

You've got it. If machine won't give us a hundred bytes

100 bytes? Excuse? Where did you get the asinine notion that a call to
malloc is limited to 100 bytes? Last I checked, it was limited to
SIZE_MAX, which with the tools I use is either 4294967295 bytes or
18446744073709551615 bytes depending on whether you're running a 32-bit
or a 64-bit OS. Hell, even on a 16-bit machine, you can generally
allocate up to 64K. 100 bytes? Where did that bit of nonsense come from?

, and the end
user cannot free up any more applications, what can we do, seriously?

All *sorts* of things, unless he uses your code in which case he's
screwed.

Single user apps.

You mean like, oh, a word processor which might not have enough resources
available to run the spell checking module, but which has *absolutely no
justification* for realizing it can't allocate the memory, choosing to
die a graceless death, taking your entire document with it?

No freaking thank you. That sort of idiocy went out with... er... no,
actually, I've been coding for 29 years now and I can't recall a time it
was *ever* an acceptable practise.

Or database apps where there isn't a robust memory
recovery strategy in place

Ah, right, so if it can't allocate enough memory to process that large
select you just issued, it makes perfect sense to crash and die, taking
with it any unstored data, rather than report back that there's
insufficient memory. Yeah, well, not like *data* matters, not in a
*database* app.

- e.g. anything which uses an X server.

WTF does an X server have to do with a database?

You persist in demonstrating you have no concept at all how to design
software.

 

[Malcolm McLean]

Just one suggestion. Usually it's not typical for a /library/ function
to take the liberty of terminating your program. I suggest that
xmalloc() calls a user supplied call-back function if malloc() fails.
The address of the call back need not be passed with every function in
the library. Instead the user could call a call-back register function
before using the other library routines.

That's exactly how xmalloc() works. Except that the user callback can never
say "return null". User must either supply some memory or terminate the
program. He can of course ask the end user to close down other programs, and
ha can conduct an emergency save/backup.

 

[Kelsey Bjarnason]

Here are six functions implemented on top of xmalloc(). No C programmer
should have any triouble providing the implemetations, though replace
and getquote are non-trivial.
[snip]

I've think we've got something quite powerful here, purely because none
of these functions can ever return null for out of memory conditions.
It massively simplifies string handling.

Take a look at glib,
http://library.gnome.org/devel/glib/2.14/glib-Memory-Allocation.html

Oh, good God. They didn't. Tell me they didn't.

One wonders how many applications they've screwed over with that bit of
asinine idiocy.

Malcolm should go work for them. He'd love it. "Errors? We don't do
errors, we just crash the app. Enjoy another piece of quality software
from the Gnome team."

 

[vippstar]

That's exactly how xmalloc() works. Except that the user callback can never
say "return null". User must either supply some memory or terminate the
program. He can of course ask the end user to close down other programs, and
ha can conduct an emergency save/backup.

And you consider that powerful mr Malcom McLean?
That right there is very weak and a very bad practise.
Example:

-- snip.c --
/* no headers, we are using imaginary functions anyway */

int main(void) {

FILE *fp = fopen("/dev/zero", "r"); /* assume UNIX & fopen doesn't
fail */
char *foo, *bar;
foo = dup("Hello, World"); /* assume success */
bar = getline(fp); /* obviously this will fail to allocate enough
memory, will quit and a memory leak will occur because the return
value of dup() was not freed */
free(bar); free(foo);
return 0;
}
-- snip.c --

I can think of a *lot* more reasons why I would not want my program to
terminate because an allocation failed.
These solutions are horrible, I strongly suggest that you avoid using
them in your own programs.

 

[Ian Collins]

[snips]

That's exactly how xmalloc() works. Except that the user callback can
never say "return null". User must either supply some memory or
terminate the program.

So the user - in this case, the coder using your function - has
absolutely no option to do something as basic as reducing the memory
allocation request and trying again, or scheduling the operation for
later processing and continuing on to some other action.

Doesn't that meet the requirements for "supply some memory"? That
requirement appears sufficiently vague to cover just about anything!

 

[Yevgen Muntyan]

Here are six functions implemented on top of xmalloc(). No C programmer
should have any triouble providing the implemetations, though replace
and getquote are non-trivial.
[snip]

I've think we've got something quite powerful here, purely because none
of these functions can ever return null for out of memory conditions. It
massively simplifies string handling.

Take a look at glib,
http://library.gnome.org/devel/glib/2.14/glib-Memory-Allocation.html

Best regards,
Yevgen

 

[Malcolm McLean]

[snips]

That's exactly how xmalloc() works. Except that the user callback can
never say "return null". User must either supply some memory or
terminate the program.

So the user - in this case, the coder using your function - has
absolutely no option to do something as basic as reducing the memory
allocation request and trying again, or scheduling the operation for
later processing and continuing on to some other action.

Doesn't that meet the requirements for "supply some memory"? That
requirement appears sufficiently vague to cover just about anything!

Rescheduling would be possible. Reducing the amount allocated not, because
then caller would have to fall back on some alternative algorithm. That
tends not to make sense except in special situations, where malloc() is the
appropriate call.

 

[Malcolm McLean]

Yes, that's right, it's a trap door function: you can enter, but you can
never leave, unless and until the memory is available. No error
reporting, no recovery, no options at all. That is the intent of the
code, as Malcolm has expressed more than once, that you *cannot*
recover from any allocation failure. You get the memory, or you exit.
Period.

Exactly. Which is what these string functions will do. They will never
return null for out of memory conditions.

And does it do single quotes? Double quotes? Both?

The quote character is passed, together with the character to escape the
quotes (which can be the quote itself). It doesn't support different opening
and closing quotes, which maybe it ought.

No, no, it makes error handling *easier*, because you don't ever need to
*do* error handling. xmalloc does that for you - by ensuring you get the
memory, or the app aborts. See how wonderful it is? No need for all
that error handling code at the calling point. Also no way to ever do
anything like, oh, reduce the allocation request and try again, or
reschedule the operation for later, etc, etc, etc, but hey, it's not like
any application would ever actually do things like that.

You've got it. If machine won't give us a hundred bytes, and the end user
cannot free up any more applications, what can we do, seriously? The
ambitious might attempt an emergency save in the callback function/

Except it's of little use in toy apps. Such an app probably doesn't do a
hell of a lot of complex allocations in the first place, so wouldn't
benefit significantly from "easy malloc". A complex app where there may
be thousands or millions of allocations done from hundreds or thousands
of places in the code, now this would benefit from "easy malloc" - if it
were actually useful.

The xmalloc notion presented is of little interest in a toy app, and is
unusable in a real app. One wonders where, exactly, it would actually be
used. One place it won't be used: my code. I actually like to be able
to choose a course of action on allocation failure, not be stuck with
some asinine notion that allocation will succeed or there's no point in
continuing.

Single user apps. Or database apps where there isn't a robust memory
recovery strategy in place - e.g. anything which uses an X server. Not for
an app where it doesn;t matter if you crash or not on the null pointer
dereference that will follow a failed, unchecked malloc(), not for an app
which absolutely must not terminate but can abort jobs if memory gets low.
Between that low end and high end there's an enormous number of programs.

 

[William Ahern]

Here are six functions implemented on top of xmalloc(). No C programmer
should have any triouble providing the implemetations, though replace
and getquote are non-trivial.

[snip]

I've think we've got something quite powerful here, purely because none
of these functions can ever return null for out of memory conditions. It
massively simplifies string handling.

Take a look at glib,
http://library.gnome.org/devel/glib/2.14/glib-Memory-Allocation.html

glib is where bad ideas go to die. Now, if somebody just had the nerve to
tell them....

 

[Yevgen Muntyan]

Here are six functions implemented on top of xmalloc(). No C programmer
should have any triouble providing the implemetations, though replace
and getquote are non-trivial.

[snip]

I've think we've got something quite powerful here, purely because none
of these functions can ever return null for out of memory conditions. It
massively simplifies string handling.

Take a look at glib,
http://library.gnome.org/devel/glib/2.14/glib-Memory-Allocation.html

glib is where bad ideas go to die. Now, if somebody just had the nerve to
tell them....

There are at least two desktop environments built on glib, and
bunch of independent applications, and they actually work. Have
you got something more substantial than "bad ideas"?

Best regards,
Yevgen

 

[William Ahern]

Malcolm McLean wrote:
Here are six functions implemented on top of xmalloc(). No C programmer
should have any triouble providing the implemetations, though replace
and getquote are non-trivial.

[snip]

I've think we've got something quite powerful here, purely because none
of these functions can ever return null for out of memory conditions. It
massively simplifies string handling.

Take a look at glib,
http://library.gnome.org/devel/glib/2.14/glib-Memory-Allocation.html

glib is where bad ideas go to die. Now, if somebody just had the nerve to
tell them....

There are at least two desktop environments built on glib, and
bunch of independent applications, and they actually work. Have
you got something more substantial than "bad ideas"?

I only brought it up because glib uses an xmalloc() wrapper which dies on
allocation failure, and Evolution and similar glib-based applications loved
to crash when, for instance, I would [try to] open a message with an
especially long marked-up diff. Of course, it was worse (as in more
frequent) before I disabled the Linux OOM killer. Never infrequent enough,
however.

In my experience, glib-based applications "mostly" work. Not sure whether
that has more to do w/ the xmalloc() wrapper, or other issues. I'll grant
you that the organization and design of glib source interfaces is... a
matter of taste.

One of my rules of thumb is that if a network daemon uses glib, I
automatically exclude it from consideration. I can deal when an application
crashes and destroys my work. I don't want to be responsible for installing
an application which crashes and destroys or interrupts _other_ people's
work.

 

[Randy Howard]

Here are six functions implemented on top of xmalloc(). No C programmer
should have any triouble providing the implemetations, though replace
and getquote are non-trivial.

[snip]

I've think we've got something quite powerful here, purely because none
of these functions can ever return null for out of memory conditions. It
massively simplifies string handling.

Take a look at glib,
http://library.gnome.org/devel/glib/2.14/glib-Memory-Allocation.html

glib is where bad ideas go to die. Now, if somebody just had the nerve to
tell them....

You gdon't glike ghaving gall gyour gvariables gprexfed gwith g?

 

[Yevgen Muntyan]

Malcolm McLean wrote:
Here are six functions implemented on top of xmalloc(). No C programmer
should have any triouble providing the implemetations, though replace
and getquote are non-trivial.
[snip]
I've think we've got something quite powerful here, purely because none
of these functions can ever return null for out of memory conditions. It
massively simplifies string handling.
Take a look at glib,
http://library.gnome.org/devel/glib/2.14/glib-Memory-Allocation.html
glib is where bad ideas go to die. Now, if somebody just had the nerve to
tell them....

There are at least two desktop environments built on glib, and
bunch of independent applications, and they actually work. Have
you got something more substantial than "bad ideas"?

I only brought it up because glib uses an xmalloc() wrapper which dies on
allocation failure, and Evolution and similar glib-based applications loved
to crash when, for instance, I would [try to] open a message with an
especially long marked-up diff. Of course, it was worse (as in more
frequent) before I disabled the Linux OOM killer. Never infrequent enough,
however.

I'd think evolution crashed because it crashed. You can look at
its list of bugs to see what I mean. If it was abort() inside
g_malloc(), then it probably leaked so much that it indeed has
eaten everything and asked for more. In which case you can't
really blame glib
(Here I'd be actually grateful if such applications were killed
immediately, because they first freeze X, and I've got to wait
for ten minutes to switch to console and kill the application.
They just don't die themselves.)

In my experience, glib-based applications "mostly" work. Not sure whether
that has more to do w/ the xmalloc() wrapper, or other issues.

There is no xmalloc() wrapper in glib. Anyway, have you never
seen "mostly working" application which do not use glib? Bugs
are everywhere, on all platforms. Do you have a real base for
saying that g_malloc() is somehow responsible for crashes you
have seen? Something other than "evolution crashed", that is,
or "similar applications" (similar glib-based applications
which open messages, huh?).

You know, I have heard things like you are saying only from
people who talk about xmalloc() and related things. Never from
users. Why is that? Perhaps because buggy applications are
buggy applications, not some poor creatures crashing because
glib memory handling is broken?

I'll grant
you that the organization and design of glib source interfaces is... a
matter of taste.

Yes it is, certainly. And it's irrelevant.

One of my rules of thumb is that if a network daemon uses glib, I
automatically exclude it from consideration. I can deal when an application
crashes and destroys my work. I don't want to be responsible for installing
an application which crashes and destroys or interrupts _other_ people's
work.

It's fine, nobody promiced glib will work for any program. It certainly
won't; but nevertheless it doesn't make abort-on-failing-malloc less
sensible strategy for a whole class of applications.

Besides, a glib application can set up some sort of emergency memory
pool or something, so that failed malloc doesn't necessarily lead to
immediate abort(). Same sort of science fiction as "graceful exit with
saving data on *any* failed malloc() call in any possible application
in any possible situation" which seems to be so popular here

Best regards,
Yevgen

 

[Yevgen Muntyan]

Malcolm McLean wrote:
Here are six functions implemented on top of xmalloc(). No C programmer
should have any triouble providing the implemetations, though replace
and getquote are non-trivial.
[snip]
I've think we've got something quite powerful here, purely because none
of these functions can ever return null for out of memory conditions. It
massively simplifies string handling.
Take a look at glib,
http://library.gnome.org/devel/glib/2.14/glib-Memory-Allocation.html

glib is where bad ideas go to die. Now, if somebody just had the nerve to
tell them....

You gdon't glike ghaving gall gyour gvariables gprexfed gwith g?

Why, you don't like the following code?

#include <glib.h>

gint main (gint argc, gchar **argv)
{
gchar *s = g_strdup ("Hello there!");
g_print ("%s\n", s);
g_free (s);
}

 

[Yevgen Muntyan]

Here are six functions implemented on top of xmalloc(). No C programmer
should have any triouble providing the implemetations, though replace
and getquote are non-trivial. [snip]

I've think we've got something quite powerful here, purely because none
of these functions can ever return null for out of memory conditions.
It massively simplifies string handling.

Take a look at glib,
http://library.gnome.org/devel/glib/2.14/glib-Memory-Allocation.html

Oh, good God. They didn't. Tell me they didn't.

One wonders how many applications they've screwed over with that bit of
asinine idiocy.

One wonders why one wonders about that only after he learns about
g_malloc. Perhaps because those applications aren't actually screwed?

 

[cr88192]

The specifications are too weak to drive implementations.
What does xmalloc() do when unable to allocate memory? Does
replace() replace all occurrences, or just the first, or just
the last, or what? What does getquote() do with unbalanced
quotes? Does getline() keep or discard the '\n', and what does
it do on an I/O error? (See the recent thread on design of line-
input functions to see how divergent people's opinions are on
how this "familiar friend" should behave.)


... and massively complicates error handling. If xmalloc()
can unilaterally terminate the program, the whole suite is
unusable except in toy programs. If there's a fancier framework,
all functions that use the suite or that call other functions
that might use the suite need to be plugged in to the framework.

Design consists of more than writing a few declarations.

I usually use a garbage collector.


if the app runs out of memory (rare, since I usually use primarily static
memory-management approaches, but I can be lazy sometimes), the allocator
returns NULL.

now, crashing the app is just lazy, but it is possibly justified on the
grounds that on modern systems, running out of memory is rare enough that
this is justified (actually, it may well be justified to crash the app
before this limit, setting some sort of critical upper bound on memory
usage, and provoking the dev to reduce the footprint).

however, often NULL works just as well, because, as a general rule accessing
memory through a NULL pointer will crash the app anyways (most of the time,
address 0 being set to no-access).

this would be, assuming the issue is not handled.


better though, would be an exception handling system, namely, we crash only
if the out-of-memory exception goes unhandled. now, this is less nice in C,
which lacks a good exception system (nor the language features to really
implement one).

that is not to say, however, that a crude one could not be built around
setjmp/longjmp (library feature), or could be added, for example, as a
compiler syntax extension that rewrites exception handlers into the library
feature's form.


thus, in this case, an unhandled exception thus results in an abort (sadly,
an actual 'crash' at this point is pointless, since the context of the
exception is already lost).

of course, another approach would be threaded or recursive exception
handling (in C, however, there would be present other problems, and
semantics which would be problematic to reconcile with the unwinding
variety).

the system could, likely, be made to handle both, with some handlers set to
behave in a recursive manner (more like signal) and others in an unwinding
manner (longjmp). for semantic reasons, likely an unwinding handler would
silently ignore any previous recursive handlers (them being invoked in an
essentially meaningless context).

throw:
if last handler is signal-style, invoke callback;
otherwise, unwind to last handler.

handler:
if correct exception, do handling actions, and resume execution post
handler;
if there is another unwinding handler, unwind to that handler;
else, rethrow an unhandled exception.