When to use automatic variables and when to use malloc

[Keith Thompson]

Oh, and to make clear, I am aware that, since the program will crash
when I use too much stack, than I should not use huge variables on
stack unless I'm OK with the program crashing if there is not enough
memory. So if it would be OK to use the wrapper, it is OK to use VLA.
(from the "what happens if it crash" perspective. There are still
other perspectives.)

[...]

You're assuming that the program will "crash" if you attempt to
allocate too much memory with a VLA. The standard doesn't guarantee
this; it says only that the behavior is undefined. (Actually, I don't
think it even says that, but it's implicit in the lack of definition
of the behavior.) The consequences of undefined behavior can be
arbitrarily horrific, including having the program continue to run and
give bad results with no visible indication that anything has gone
wrong.

Now it's very likely that a "stack overflow" will be caught by the OS,
and will cause the program to do nothing more drastic than terminating
immediately. But the standard doens't guarantee this.

Of course, all this applies to fixed-size automatic variables just as
much as to VLAs. VLAs tend to be more dangerous, though, because of
the risk of computing the needed size incorrectly.

 

[user923005]

He meant that malloc has some dangers, not that VLAs do not.

The only danger malloc() has that VLA does not is that you do not have
to remember to free a VLA.
Every other flaw that malloc has, VLAs also have.
On the other hand, VLAs have fatal flaws that are far in excess of
what malloc() has.
What will your program do when it asks for a VLA but your program is
all out of automatic memory?
There are also some subtle things like alignment. The memory
allocated by malloc() must have suitable alignment for any sort of
object. I see no such guarantee on a VLA. So I see in this a subtle
opportunity for bus errors.

In short, what does a VLA buy for you?
You don't have to remember to call free (but it also means that you
will not get persistence that can last outside of a block).
You get memory from the automatic memory area instead of the free
store (which might be faster but also might have alignment only for
the array type at hand).

What't the big danger of a VLA:
When the allocation fails, the airplane goes out of control. It
enters your living room, and strikes you on the back of the head while
you are watching 'The Simpsons'. It flies out of your left nostril,
leaving your brains in a rather disheveled state.
Maybe : "Low on memory -- you'll have to play pong later" would have
been better.

 

[Jorge Peixoto de Morais Neto]

You made me think of another question: when a program crashes,
possibly due to lack of memory, do the OS normally flush its buffers
and close its files?

Answering myself: I have just tested, and it seems I can have a singe
2 MB automatic variable. It seems that when the program crashes the OS
does *not* flush its buffers.

Interestingly, it seems I can allocate several 2 MB variables. So the
problem is if I try to allocate too much space at once. Several 2 MB
variables seem to be possible.

 

[Jorge Peixoto de Morais Neto]

Answering myself: I have just tested, and it seems I can have a singe
2 MB automatic variable. It seems that when the program crashes the OS
does *not* flush its buffers.

Interestingly, it seems I can allocate several 2 MB variables. So the
problem is if I try to allocate too much space at once. Several 2 MB
variables seem to be possible.

I'm sorry, the test was broken. It seems that the program crashes
whenever the total size for the locals is greater than 2MB. I can't
allocate two 1.5 MB variables.

 

[robertwessel2]

Answering myself: I have just tested, and it seems I can have a singe
2 MB automatic variable. It seems that when the program crashes the OS
does *not* flush its buffers.

While the answer to your question is entirely OS dependant, it's quite
likely that you've misunderstood what happening in your OS. Almost
all OS's *will* flush *their* buffers when an application crashes (or
at least no discard them so that the get written to disk based on
whatever the OS's dirty write policy is). What you've forgotten is
that streams in C almost always implement their own buffering (that's
what's modified by setvbuf() and friends) on top of whatever the OS
might be doing. Those do *not* get flushed by the OS when the
application crashes, since they are not actually visible to the OS.
Retry your test with an unbuffered stream.

 

[user923005]

Answering myself: I have just tested, and it seems I can have a singe
2 MB automatic variable. It seems that when the program crashes the OS
does *not* flush its buffers.

Interestingly, it seems I can allocate several 2 MB variables. So the
problem is if I try to allocate too much space at once. Several 2 MB
variables seem to be possible.

You can't find the answers to questions like this very well by
experimentation.
A different compiler can have a different result.
A different machine can have a different result.
You can modify the programs stack, even after linking (on some
systems).
A change in compiler settings can turn off stack overflow detection
(and now you're really in trouble).
A a different compiler may not have stack overflow detection (and now
you're really in trouble).

Unfortunately, there is no elegant way to catch VLA allocation errors
and there is no way to reliably expect or detect them either.
You could do the old 'stack address subtraction' trick, but you can't
really {portably} know for sure how much stack is available.

In short, VLAs are a mixed blessing, and a mixed cursing.
Use with care and use with joy. But learn that they are not a toy.
OK, sometimes they're a toy.

 

[Jorge Peixoto de Morais Neto]

The only danger malloc() has that VLA does not is that you do not have
to remember to free a VLA.
Every other flaw that malloc has, VLAs also have.
On the other hand, VLAs have fatal flaws that are far in excess of
what malloc() has.
What will your program do when it asks for a VLA but your program is
all out of automatic memory?
There are also some subtle things like alignment. The memory
allocated by malloc() must have suitable alignment for any sort of
object. I see no such guarantee on a VLA. So I see in this a subtle
opportunity for bus errors.

In short, what does a VLA buy for you?
You don't have to remember to call free (but it also means that you
will not get persistence that can last outside of a block).
You get memory from the automatic memory area instead of the free
store (which might be faster but also might have alignment only for
the array type at hand).

What't the big danger of a VLA:
When the allocation fails, the airplane goes out of control. It
enters your living room, and strikes you on the back of the head while
you are watching 'The Simpsons'. It flies out of your left nostril,
leaving your brains in a rather disheveled state.
Maybe : "Low on memory -- you'll have to play pong later" would have
been better.

I got your point, but I don't see the problem with the alignment
issue. If I declare a VLA of type x, it is guaranteed to be aligned
for type x, is it not? From what I know, malloc is no better than VLA
in this aspect. The only difference is that malloc returns a void*, so
that it must be aligned for anything. A guy told me that malloc does 8-
byte alignment (obviously, this might - and almost certainly will -
differ between systems).

 

[Jorge Peixoto de Morais Neto]

While the answer to your question is entirely OS dependant, it's quite
likely that you've misunderstood what happening in your OS. Almost
all OS's *will* flush *their* buffers when an application crashes (or
at least no discard them so that the get written to disk based on
whatever the OS's dirty write policy is). What you've forgotten is
that streams in C almost always implement their own buffering (that's
what's modified by setvbuf() and friends) on top of whatever the OS
might be doing. Those do *not* get flushed by the OS when the
application crashes, since they are not actually visible to the OS.
Retry your test with an unbuffered stream.

When I said "its buffers", I was referring to the user-space stream
buffers. Sorry about that. Ambiguity in the English language. As an
offtopic remark, this kind of thing seems more common in English than
in Portuguese.

 

[user923005]

I got your point, but I don't see the problem with the alignment
issue. If I declare a VLA of type x, it is guaranteed to be aligned
for type x, is it not? From what I know, malloc is no better than VLA
in this aspect. The only difference is that malloc returns a void*, so
that it must be aligned for anything. A guy told me that malloc does 8-
byte alignment (obviously, this might - and almost certainly will -
differ between systems).

If you do things that evil C programmers are wont to do from time to
time, you might get yourself a bus fault.

For instance, suppose that your cheeky little VLA is being used as a
union because -- hey -- it's clever and it can hold anything smaller
than [dim] bytes. This might cause a bus error:
*myInteger = *(int *) &myVLA;
but it would not if you had used malloc() instead. I agree that it is
esoteric, but that's another difference worth keeping in mind.

 

[user923005]

On Feb 20, 7:09 pm, "Jorge Peixoto de Morais Neto"

A guy told me that malloc does 8-
byte alignment (obviously, this might - and almost certainly will -
differ between systems).

P.S.
It's not 8 bytes. It might be 8 bytes on system x, y, or z. But
there are no guarantees of that. So if you assume it, you could
easily get into trouble.

Memory returned by malloc()/calloc() is "suitably aligned for any data
type" -- and whatever that means is what it is.

 

[websnarf]

malloc returns a pointer. A variable length array, is still just
that; an array. So the common danger of using pointers versus arrays
is removed. (An array base cannot be NULL, cannot change relative to
its intended storage, and cannot be pointing to unaccessible memory.)

[...]

I fail to see how any danger is removed. An array name still decays
to a pointer in most contexts, and all the pitfalls of confusing
arrays and pointers (all of which can be alleviated by reading and
understanding section 6 of the comp.lang.c FAQ) are still there.

Do you truly have no neurons functioning outside the confines of the
imagined perfect workings of the standard at all? Danger is about
failure -- its about when people do things wrong, and are not reading
or implementing things 100% in accordance to the standard. And it
appears everywhere, because this language highly encourages that sort
of thing. Here's a test for *YOU*. Can you name *ANYTHING* dangerous
at all about the C language? This is just to establish exactly where
your credibility as an intellectual entity is.

A pointer *ITSELF* is storage, which can be modified. By losing track
of this pointer value, logic errors can easily cause you to be
pointing at the wrong thing when you derefence that pointer. This is
the source of danger. An array is actually implicitely determined at
runtime as a nonmutable pointer that is never NULL -- if its auto,
then its usually just an offset relative to a stack register. The
value cannot be modified at all, and its storage lifetime is never in
question or at issue.

 

[Martin Ambuhl]

You, of course, have never heard of viruses, heap overflows, double
freeing, memory leaking or information leaking. Probably because
these things don't appear in the standard and you haven't looked at
the wider world of computing since the 1970s.

Your abusive language accompanies your failure to provide any sort of
supporting argument. It may surprise you to learn that name-calling
does not count as a rational argument. Nor does your lack of competence
amount to "problems and dangers of malloc." Rather than suggestive
illusory solutions to imaginary problems with malloc, work on learning
how to program competently.

 

[Martin Ambuhl]

malloc returns a pointer. A variable length array, is still just
that; an array. So the common danger of using pointers versus arrays
is removed. (An array base cannot be NULL, cannot change relative to
its intended storage, and cannot be pointing to unaccessible memory.)

[...]

I fail to see how any danger is removed. An array name still decays
to a pointer in most contexts, and all the pitfalls of confusing
arrays and pointers (all of which can be alleviated by reading and
understanding section 6 of the comp.lang.c FAQ) are still there.

Do you truly have no neurons functioning outside the confines of the
imagined perfect workings of the standard at all? Danger is about
failure -- its about when people do things wrong, and are not reading
or implementing things 100% in accordance to the standard. And it
appears everywhere, because this language highly encourages that sort
of thing. Here's a test for *YOU*. Can you name *ANYTHING* dangerous
at all about the C language? This is just to establish exactly where
your credibility as an intellectual entity is.

Your abusive language accompanies your failure to provide any sort of
supporting argument. It may surprise you to learn that name-calling
does not count as a rational argument. Nor does your lack of competence
amount to "problems and dangers of malloc." Rather than suggestive
illusory solutions to imaginary problems with malloc, work on learning
how to program competently.

 

[Keith Thompson]

malloc returns a pointer. A variable length array, is still just
that; an array. So the common danger of using pointers versus arrays
is removed. (An array base cannot be NULL, cannot change relative to
its intended storage, and cannot be pointing to unaccessible memory.)

[...]

I fail to see how any danger is removed. An array name still decays
to a pointer in most contexts, and all the pitfalls of confusing
arrays and pointers (all of which can be alleviated by reading and
understanding section 6 of the comp.lang.c FAQ) are still there.

Do you truly have no neurons functioning

[snip]

That's about where I stopped reading. If you can't be civil, I'm just
not interested in what you have to say. It's too bad; there might
have been some interesting technical points in there, but I have
better things to do than wade through your crap.

 

[tphipps1]

Do you truly have no neurons functioning

[snip]

That's about where I stopped reading. If you can't be civil, I'm just
not interested in what you have to say. It's too bad; there might
have been some interesting technical points in there, but I have
better things to do than wade through your crap.

There was an interesting technical point, and I'd be quite interested
to hear you and others talk about it, so I'll reproduce websnarf's
post with the gratuitous insults removed:

A pointer *ITSELF* is storage, which can be modified. By losing track of
this pointer value, logic errors can easily cause you to be pointing at the
wrong thing when you derefence that pointer. This is the source of danger.
An array is actually implicitely determined at runtime as a nonmutable
pointer that is never NULL -- if its auto, then its usually just an offset
relative to a stack register. The value cannot be modified at all, and its
storage lifetime is never in question or at issue.

Is an array really equivalent to a non-mutable non-NULL pointer?

 

[Flash Gordon]

Do you truly have no neurons functioning

[snip]

That's about where I stopped reading. If you can't be civil, I'm just
not interested in what you have to say. It's too bad; there might
have been some interesting technical points in there, but I have
better things to do than wade through your crap.

There was an interesting technical point, and I'd be quite interested
to hear you and others talk about it, so I'll reproduce websnarf's
post with the gratuitous insults removed:

A pointer *ITSELF* is storage, which can be modified. By losing track of
this pointer value, logic errors can easily cause you to be pointing at the
wrong thing when you derefence that pointer. This is the source of danger.

Agreed, although in the situations where a VLA would be a suitable
alternative solution it is, IMHO, easy to ensure you do not make those
mistakes. A quick check that it is non-null after allocation and then
only use array notation on it.

That is just asking to confuse the less experienced since array and
pointers have very important differences.

type * const ptr = malloc(10 * sizeof *ptr);

There, not ptr cannot be modified without you going to a lot of work.

Now you just have to remember to free it which is not that difficult in
the situation where a VLA could be used.

So yes, there is more danger of programmer error, but if you are
sensible not much. However, there is more danger of a VLA causing
problems when memory runs short (or you just try for too large an array)
than with malloc because you cannot trap the error.

Is an array really equivalent to a non-mutable non-NULL pointer?

Check the result of applying sizeof to it and you will find out that it
is not. Also the type of &array is not the same as the type of &ptr

Repeat after me, "an array is not a pointer and a pointer is not an
array". Also check out section 6 of the comp.lang.c FAQ.

 

[Keith Thompson]

Do you truly have no neurons functioning

[snip]

That's about where I stopped reading. If you can't be civil, I'm just
not interested in what you have to say. It's too bad; there might
have been some interesting technical points in there, but I have
better things to do than wade through your crap.

There was an interesting technical point, and I'd be quite interested
to hear you and others talk about it, so I'll reproduce websnarf's
post with the gratuitous insults removed:

A pointer *ITSELF* is storage, which can be modified. By losing track of
this pointer value, logic errors can easily cause you to be pointing at the
wrong thing when you derefence that pointer. This is the source of danger.
An array is actually implicitely determined at runtime as a nonmutable
pointer that is never NULL -- if its auto, then its usually just an offset
relative to a stack register. The value cannot be modified at all, and its
storage lifetime is never in question or at issue.

Is an array really equivalent to a non-mutable non-NULL pointer?

It depends on what you mean by "pointer".

One common usage is to use the term "pointer" to refer to a pointer
*object*. In that sense, there is no pointer object associated with
an array object.

But if the term "pointer" refers instead to a pointer *value* (the
standard uses the term "pointer" this way), then yes, an array object
implies the existence of a pointer value which is non-mutable and
non-NULL. It is the pointer value to which the array name decays in
most contexts, the address of its first element.

Of course, arrays are not pointers and pointers are not arrays. Most
(but not all) operations on arrays in C are defined in terms of
pointer operations, but it's important to keep in mind that they're
very different things.

And if you've declared an array object, anything you do with it
outside the context in which it's declared (in the case of a VLA, this
is at most a single function) is likely to be done via pointers
anyway, with all the potential for mayhem that that implies.

As for an array object's storage lifetime being "never in question or
at issue", that's true if you know what you're doing, but we see
plenty of functions here that incorrectly attempt to return the
address of a local array object.

I'm not particularly happy with the way C sometimes conflates arrays
and pointers. But the underlying concepts are actually fairly
elegant, in an uncomfortably close-to-the-hardware kind of way. (This
is just my own personal opinion.) Reading and understanding section 6
of the comp.lang.c FAQ is a good way to cope with the confusion.

(This is not to imply that websnarf is or is not aware of all this.)

 

[CBFalconer]

Do you truly have no neurons functioning

[snip]

That's about where I stopped reading. If you can't be civil, I'm
just not interested in what you have to say. It's too bad; there
might have been some interesting technical points in there, but I
have better things to do than wade through your crap.

There was an interesting technical point, and I'd be quite
interested to hear you and others talk about it, so I'll reproduce
websnarf's post with the gratuitous insults removed:

A pointer *ITSELF* is storage, which can be modified. By losing
track of this pointer value, logic errors can easily cause you to
be pointing at the wrong thing when you derefence that pointer.
This is the source of danger. An array is actually implicitely
determined at runtime as a nonmutable pointer that is never NULL
-- if its auto, then its usually just an offset relative to a
stack register. The value cannot be modified at all, and its
storage lifetime is never in question or at issue.

Is an array really equivalent to a non-mutable non-NULL pointer?

Yes, except that that pointer is a different beast than the kind
you normally bandy about and pass hither and yon. It includes such
information as the size and type of the array, the scope, the
location, and what that location is relative to (local or global
storage, which local storage, etc.) (using the common usage for
global/local). It typically lives in the compilers internal
tables.

--
<http://www.cs.auckland.ac.nz/~pgut001/pubs/vista_cost.txt>
<http://www.securityfocus.com/columnists/423>

"A man who is right every time is not likely to do very much."
-- Francis Crick, co-discover of DNA
"There is nothing more amazing than stupidity in action."
-- Thomas Matthews

 

[Richard Heathfield]

user923005 said:

What't the big danger of a VLA:
When the allocation fails, the airplane goes out of control. It
enters your living room, and strikes you on the back of the head while
you are watching 'The Simpsons'. It flies out of your left nostril,
leaving your brains in a rather disheveled state.

Moral of the story - in future, watch 'The Simpsons' in the kitchen.

 

[Eric Sosman]

You, of course, have never heard of viruses, heap overflows, double
freeing, memory leaking or information leaking. Probably because
these things don't appear in the standard and you haven't looked at
the wider world of computing since the 1970s.

I've heard of all these things, have experienced some of
them, and do not see how they are due to malloc().

On the matter of looking at the wider world of computing,
though, you're right. I looked, I saw websnarf, and I looked
away. Who could blame me?