malloc

[Richard Heathfield]

goose said:

Don't attempt to be condescending; you aren't any good at it.

But *I* am. I can provide top-quality condescension at competitive rates.
All he has to do is call.

 

[Random832]

Don't attempt to be condescending; you aren't any good at it.

Was I wrong?

See my reply to Barry Schwarz elsethread.

Your only reply to him that I can see is the one that I, in turn replied
to. Right above this one.

goose,
It's more common than you think.

What is? A program never "releases" memory without ever using it, that'd
be silly. So basically it's gambling that program A will either
"release" some memory or die before program B attempts to write the
memory *that it just allocated* most likely in the next few seconds,
thus allowing it to thwart any code that might be in place for program
B to sanely handle the case where not as much memory as it wanted was
available.

 

[CBFalconer]

Keith Thompson said:



Yes, which is why I said, in my first reply in this thread, that
the behaviour of the program is undefined. And even if I hadn't
said that, did you really think I didn't know it?

Don't take such clarifications personally. This is a public place,
and there is good reason to correct and/or clarify statements,
whether or not the originator knows better.

 

[Random832]

Agreed. To both.


Um, I wasn't referring to the tuning of malloc options on
FreeBSD, but the tuning of the VM bits of the kernel itself.

FreeBSD famously overcommits swap.

Maybe FreeBSD does this, but it's Linux that is famous for it.

Either way, malloc does not "fail gracefully" here; a process
may be targetted for killing in the event that none of the
original 4 processes end within a certain time.

Is it a certain time, or is it just whenever the 5th process tries to
access the allocated memory?

incidentally, if it is "within a certain time", that means if program
5 attempts to access the memory before that time, it will block until
that time has passed. Which opens the door for a conforming
implementation that overcommits: just block until the memory is
available, with no such timeout.

 

[Random832]

Random832 said:


"If the expression that precedes the parenthesized argument list in a
function call consists solely of an identifier, and if no declaration is
visible for this identifier, the identifier is implicitly declared exactly
as if, in the innermost block containing the function call, the declaration

extern int identifier();

appeared." (C89: 3.3.2.2)

Right. But if identifier is ever used, that's undefined behavior, which
in turn releases the compiler from all obligations.

It's required to work if the function does in fact return int and, if
it's called, it's called with the right types.

It's _not_ required to _not_ work if this is not the case.

 

[Random832]

Keith Thompson said:



Yes, which is why I said, in my first reply in this thread, that the
behaviour of the program is undefined. And even if I hadn't said that, did
you really think I didn't know it?

Speaking of what the compiler is "obliged" to do in cases of undefined
behavior, though, is misleading and dangerous. It's only not undefined
behavior if the code is never reached.

 

[CBFalconer]

goose said:



But *I* am. I can provide top-quality condescension at competitive
rates. All he has to do is call.

I assume you have a suitable walk-in freezer and a humid household.

 

[Richard Heathfield]

CBFalconer said:

there is good reason to correct and/or clarify statements,
whether or not the originator knows better.

What was unclear about my previous contributions to this thread?

 

[Richard Heathfield]

Random832 said:

Speaking of what the compiler is "obliged" to do in cases of undefined
behavior, though, is misleading and dangerous.

Read what I said again, this time more carefully. I was explaining the cause
of the undefined behaviour in terms of what the compiler was obliged to do.

It's only not undefined
behavior if the code is never reached.

That's why I said, right at the outset, that the behaviour was undefined.

 

[Kohn Emil Dan]

Why would you expect a segmentation fault?

Speaking in a more practical way, there are pretty good chances to receive
a segmentation fault, but not in *your* program, but in other buggy
programs running at the same time on your system which won't check the
return value of malloc() and/or other system calls or library functions.
Running your program without appropriate process limits will cause it to
hog up the entire system memory. When the entire memory has been hogged
up, malloc() and other library functions which internally allocate memory
will fail. Unfortunately many programs fail to check the return values of
the library functions and ultimately crash with segmentation faults
because of dereferencing NULL pointers.

Bottom line: If all the programs running on your system correctly checked
the return values of all the library functions, you shouldn't receive a
segfault; unfortunately in real life you pretty much have a good chance to
receive one.


Emil

 

[goose]

Was I wrong?

You weren't even in the same context. It's impossible to be
wrong if you don't even understand the question.

Your only reply to him that I can see is the one that I, in turn replied
to. Right above this one.


What is? A program never "releases" memory without ever using it, that'd
be silly.

I'd advise you to read up a little on this before you go on. I've
provided
links to the threaded mailing-list discussion of this. Feel free.

So basically it's gambling that program A will either
"release" some memory or die before program B attempts to write the
memory *that it just allocated* most likely in the next few seconds,
thus allowing it to thwart any code that might be in place for program
B to sanely handle the case where not as much memory as it wanted was
available.

Yes. I think you get the picture now.

goose,
Why yes, I *am* rather tetchy today. Why do you ask?

 

[Random832]

You weren't even in the same context. It's impossible to be
wrong if you don't even understand the question.

Regardless of everything else, your use of the term "run concurrently"
was misleading - a program can be blocking/sleeping for hours, entirely
swapped out, and it _still_ ties up every single page of virtual memory
it has ever used; you seemed to suggest this was not the case. If this
wasn't the context you were talking about, your words were wrong for the
context you DID mean.

I've provided links to the threaded mailing-list discussion of this.

You have not. At least, not that I've been able to find. Perhaps
providing these links in an actual reply to one of my messages would be
more useful?

Yes. I think you get the picture now.

When making such a bet, how often does the system win? I'd guess the
odds are probably pretty slim.

 

[goose]

Regardless of everything else, your use of the term "run concurrently"
was misleading - a program can be blocking/sleeping for hours, entirely
swapped out, and it _still_ ties up every single page of virtual memory
it has ever used; you seemed to suggest this was not the case. If this
wasn't the context you were talking about, your words were wrong for the
context you DID mean.

I apologise; I stupidly assumed that everyone reading would be
aware of what "concurrently" meant in the context of overcommits.

Rereading it, I see that I was maybe a tad ambigous.

You have not. At least, not that I've been able to find. Perhaps
providing these links in an actual reply to one of my messages would be
more useful?

They were in reply to Barry earlier. Nevertheless, if you really wanted
to you could have googled for (without the quotes) "FreeBSD overcommit
malloc"
and taken the second link to read the entire discussion (a pretty big
one, IIRC).

The first link will allow one to find out exactly *how* common this
technique
is. Like I said in a previous post, this is learnt by undergraduates
all over
the world every year - it was mentioned as a technique in my
undergraduate
OS design course.

When making such a bet, how often does the system win? I'd guess the
odds are probably pretty slim.

Well, if you bet that way, you might lose your money. FreeBSD has
always
(since I remember, anyway) had this, as well as all non System V
unixes.
Since FreeBSD (and other unixes) have had quite a good reputation for
robustness under load, I suspect that this works better than intuition
would
suggest.

The reason might have something to do with the fact that processes
allocate blocks bigger than they need *right now*, and only fill it up
to capacity later, if at all. A common method of using a dynamic
array is to double the size each time you reallocate it - you usually
end
up never accessing the last two-thirds of it under this method. More
than
onje regular here has suggested this method in the past.

goose,
Enough now, this is off-topic as it is, no need to drag it out
further.

 

[Keith Thompson]

CBFalconer said:


What was unclear about my previous contributions to this thread?

Here's the article you wrote, to which I posted my (attempted)
clarification:

| Random832 said:
|
| > 2006-12-12 <[email protected]>,
| > Richard Heathfield wrote:
| >> Under that circumstance, the compiler is obliged to assume that malloc
| >> returns int,
| >
| > Under which circumstance, in turn, the compiler is not "obliged" to
| > anything in particular at all.
|
| "If the expression that precedes the parenthesized argument list in a
| function call consists solely of an identifier, and if no declaration is
| visible for this identifier, the identifier is implicitly declared exactly
| as if, in the innermost block containing the function call, the declaration
|
| extern int identifier();
|
| appeared." (C89: 3.3.2.2)

Your citation of the standard appeared to me to be a refutation of
Random832's statement that

the compiler is not "obliged" to anything in particular at all.

In other words, you seemed (based only on that one followup, without
reference to previous articles in the thread) to be asserting that,
given:

/* Note: no "#include <stdlib.h>" */
int main(void)
{
int *p = (int*)malloc(sizeof(int));
}

the compiler is obliged to treat malloc() as a function returning int
(and the int result will be converted to int*). In fact, as we both
know but others may not, the call invokes undefined behavior, and the
implementation is therefore free to do anything it likes -- including
treating malloc() as a function returning void* if the implementer is
more interested in generating correct results for bad code than in
emitting good diatnostics.

The fact that the call invokes undefined behavior is obvious to both
of us, but possibly not to others. The fact that the undefined
behavior implies that the compiler is allowed to treat malloc() as a
function returning void* is sufficiently subtle that it seemed
plausible you might have missed it -- thus my clarification.

(In fact, if a compiler is clever enough to recognize the undefined
behavior because it know about "malloc" even without the header, it
had *better* issue a decent diagnostic rather than just quietly
generating code consistent with the user's presumed intent, but that's
a quality of implementation issue.)

 

[Random832]

In other words, you seemed (based only on that one followup, without
reference to previous articles in the thread) to be asserting that,
given:

/* Note: no "#include <stdlib.h>" */
int main(void)
{
int *p = (int*)malloc(sizeof(int));
}

the compiler is obliged to treat malloc() as a function returning int
(and the int result will be converted to int*). In fact, as we both
know but others may not, the call invokes undefined behavior, and the
implementation is therefore free to do anything it likes -- including
treating malloc() as a function returning void* if the implementer is
more interested in generating correct results for bad code than in
emitting good diatnostics.

And, conversely, if the implementer is more interested in emitting good
diagnostics, the compiler can emit a diagnostic along the lines of
"warning: malloc used without including <stdlib.h> or a proper
declaration", and convert the line to a call to abort() (along with
a repetition of the diagnostic) so that if it is reached the program
terminates (unfortunately, an implementation is not conforming if it
refuses to translate this piece of crap.)

 

[Guest]

And, conversely, if the implementer is more interested in emitting good
diagnostics, the compiler can emit a diagnostic along the lines of
"warning: malloc used without including <stdlib.h> or a proper
declaration", and convert the line to a call to abort() (along with
a repetition of the diagnostic) so that if it is reached the program
terminates (unfortunately, an implementation is not conforming if it
refuses to translate this piece of crap.)

The behaviour would be undefined for each run of the program. In that
case, an implementation is allowed to reject it. That applies even to

/* Note: no "#include <stdlib.h>" */
int main(void)
{
if(0) {
int *p = (int*)malloc(sizeof(int));
}
}

since the behaviour is undefined not just when malloc is called, but
when it is declared inappropriately as well. As IIRC Richard Heathfield
once pointed out, C90's equivalent of C99's 7.1.3 "Reserved
identifiers" does not forbid incompatible declarations, but it still
has the equivalent of C99's 6.2.7p2:
"All declarations that refer to the same object or function shall have
compatible type; otherwise, the behavior is undefined."
and malloc's implicit declaration is incompatible with that inside the
standard library itself.

In C99, by the way, rejection is allowed for more obvious reasons.

 

[Keith Thompson]

goose said:



But *I* am. I can provide top-quality condescension at competitive rates.
All he has to do is call.

Yes, I'm sure you can. No doubt your cute little condescending ways
are among the very very best in the whole wide world! I wouldn't
*dream* of trying to compete with you on that score.

(But my rates are excellent.)

 

[Richard Heathfield]

Keith Thompson said:

Your citation of the standard appeared to me to be a refutation of
Random832's statement that

the compiler is not "obliged" to anything in particular at all.

In other words, you seemed (based only on that one followup, without
reference to previous articles in the thread) to be asserting that,
given:

/* Note: no "#include <stdlib.h>" */
int main(void)
{
int *p = (int*)malloc(sizeof(int));
}

the compiler is obliged to treat malloc() as a function returning int

And so it is.

(and the int result will be converted to int*).

It would be reasonable for the compiler to generate code to provide such a
conversion, yes - although the resulting program's behaviour will be
undefined.

In fact, as we both
know but others may not, the call invokes undefined behavior,

Right, but by that stage the compiler's job is done and dusted. The call
happens at runtime, not compile time.

and the
implementation is therefore free to do anything it likes -- including
treating malloc() as a function returning void* if the implementer is
more interested in generating correct results for bad code than in
emitting good diatnostics.

Very true, but only tenuously relevant, insofar as it is merely one possible
permutation in the lottery of undefined behaviour.

<snip>

 

[Richard Heathfield]

Keith Thompson said:

Yes, I'm sure you can. No doubt your cute little condescending ways
are among the very very best in the whole wide world! I wouldn't
*dream* of trying to compete with you on that score.

Don't give up. It normally takes many years of practice, but you might be
able to accelerate the process a little with an intensive course of study,
such as the one I run (for advanced students only, I'm afraid - there is a
tough entrance examination).

Incidentally, I find myself wondering whether "condescent" would be an
improvement on "condescension".

 

[Joe Wright]

I assume you have a suitable walk-in freezer and a humid household.

You're thinking of condensation.