Freeing memory - will it be available immediately

[Kelsey Bjarnason]

Again, where is the stadard forbidding free() to release memory for
further allocation()? Chapter and verse, please.

Actually, that is what the standard *defines* free as doing - releasing
memory for further allocation.

If you mean where does it define free as not being allowed to return the
memory back to the OS, one merely needs to ask what the C standard is
defining. As far as I can tell, it is defining the behaviour of a C
program, and the behaviour of the standard library upon which that
program is built.

Part of that defined behaviour is that the memory released by free is
made available for further allocation.

That guarantee, however, is made *in the context* of defining the
behaviour of a C program and the standard library on which the C program
relies.

 

[Gordon Burditt]

Tell me what prohibits this implementation:

[irrelevances snipped]

A C program may free memory and the implementation may return it to the
OS. The OS may then let a non-C program (or a C program running in
non-standard-compliant mode)

Non-conforming mode? Yes, well, you just answered your question: you're
using a non-conforming implementation, which means it's either broken, or
non-conforming by intent; if the former, file a bug report. If the
latter, consider it a QoI issue.

An implementation only has to run *ONE* standard-conforming C program
at a time, but it can run other stuff non-conforming. Anything
else it runs need not be written in C. The C standard only applies
to the (only) program written in C and running in standard-conforming
mode.

In other words, the C program running in standard-conforming mode
must be immune to the Linux OOM killer. Other programs need not be.

 

[ymuntyan]

[snips]

If two subsequent malloc() calls never succeed (both of them, that is),
is it a conforming implementation?

We cannot tell from this.

"Further allocations" are fine, it's
just the very next that doesn't work. And to break the Dan Pop's
example: two subsequent malloc() calls do not succeed if they have the
same argument. And now the "as if" rule kicks in.

void *ptra = malloc(100);
void *ptrb = malloc(100);

You're saying that if either allocation fails, there's a problem? Aside
from the simple lack of available memory, what, exactly, is the problem?

No, I am saying that such an implementation would be conforming
(given everything else is conforming of course).

No, they're not; if this occurred, the implementation cannot meet the
behaviour mandated by the definition of free, as the car may not return.
Thus the objects - the blocks of memory - are not allowed to be carried
away when free is called.

Why? They will be brought back, so they *are* available for further
allocation.

I can point out that the implementation does not know how many toasters
are at the toaster store, thus it cannot guarantee there are - or ever
will be - enough to meet the requests, therefore it has no choice, under
the wording of the standard but to store them itself.

The definition of free says *nothing* about the memory being made
available for later allocation *if available* or *if not consumed by some
other process*. There is zero - *zero* - allowance in the wording for
any wiggle room whatsoever.

As well as there is zero guarantee about very next call to
malloc(). All you know is that it may fail. I mean your example,

p = malloc(N);
free(p);
p = malloc(N);

Note, I didn't say anything about other processes.

I make no such assumption, and nothing I've said thus far requires such
an assumption. The standard says the memory is made available for
further allocation, it does not say when - it could be next Tuesday.

Yep. And a strictly conforming program can't know when
that is going to happen. All it can see is whether given
malloc() call failed or not, and the second malloc() in
your example can legally fail. You said it can't,
and that was your assumption - that after malloc();free();
pair the state of malloc is back to what it was.

The point at hand is that once a piece of memory has been allocated by
the program, free cannot release it back to the OS, because, even if the
implementation's memory management is so inefficient it cannot make the
memory available for further allocation until Tuesday, it nevertheless is
required to do so, while handing the memory back to the OS means it
cannot ensure that it meets the requirements of the standard, no matter
how inefficiently.

No, the point is that a strictly conforming program
can't possibly know whether it's still Monday or
that implementation gave the memory back to OS.
So no strictly conforming program can demonstrate
that this implementation gave memory back to OS.
And so "gave memory back to OS" simply doesn't make
sense in the context of a C program: all it knows
is that the allocation failed or didn't fail.

If an implementation accepts and correctly executes
every strictly conforming program (plus it does
what it should with incorrect programs, like issues
diagnostics), is it conforming?

Yevgen

 

[Kelsey Bjarnason]

Tell me what prohibits this implementation:

[irrelevances snipped]

A C program may free memory and the implementation may return it to
the OS. The OS may then let a non-C program (or a C program running
in non-standard-compliant mode)

Non-conforming mode? Yes, well, you just answered your question: you're
using a non-conforming implementation, which means it's either broken,
or non-conforming by intent; if the former, file a bug report. If the
latter, consider it a QoI issue.

An implementation only has to run *ONE* standard-conforming C program at
a time, but it can run other stuff non-conforming.

And this has *what* to do with the price of tea in China?

 

[Gordon Burditt]

Tell me what prohibits this implementation:

[irrelevances snipped]

A C program may free memory and the implementation may return it to
the OS. The OS may then let a non-C program (or a C program running
in non-standard-compliant mode)

Non-conforming mode? Yes, well, you just answered your question: you're

The C standard imposes no constraints on the way the OS handles non-C
programs. It only applies to programs running as standard C programs.

An implementation of C is not broken because the OS runs non-C programs
differently from the way it runs C programs.

And this has *what* to do with the price of tea in China?

free() may return memory from the C program to the OS, and the OS
may reallocate the memory to a non-C program, and the OS may forcibly
take back the memory using a Linux OOM killer (if needed), and give
it back to the original C program, all without violating any
requirement of the C standard, which applies only to the original
C program and not to other non-C programs the OS may be running at
the same time.

 

[Richard Bos]

You keep saying that, why is it you seem unable to demonstrate it?

It's been explained several times, but you refuse to accept it.

One (one last!) more time:

Your interpretation of "for further allocation" as "for futher
allocation by the current program alone" is a reasonable one.
Others' interpretation of "for further allocation" as "for futher
allocation by any program" is also a reasonable one.
Your refusal to accept the others' interpretation as reasonable is
unreasonable; your interpretation of "for further allocation" as "for
futher allocation by the current program alone, and no other
interpretation is acceptable" is simply wrong.


Richard

 

[Kelsey Bjarnason]

[snips]

free() may return memory from the C program to the OS

Only if it can guarantee the OS will somehow tag that memory as reserved
for the C program. Otherwise, it cannot. Read the definition of free;
there is no allowance for handing the memory back to the OS in the
general case, as that would risk making the memory *not* available for
further allocation.

 

[Kelsey Bjarnason]

It's been explained several times, but you refuse to accept it.

One (one last!) more time:

Your interpretation of "for further allocation" as "for futher
allocation by the current program alone" is a reasonable one.

Since the very thing whose behaviour is being defined is the C program,
it is the *only* relevant context in which to define the behaviour.

Others'
interpretation of "for further allocation" as "for futher allocation by
any program" is also a reasonable one.

Except that this ignores the very reason for the standard, which is *not*
to define behaviours of "any program".

Your refusal to accept the
others' interpretation as reasonable is unreasonable;

On the contrary, it is perfectly reasonable, as they have yet to give any
justification for this magical intrusion of the rest of the world into
the operation of free, contrary to the dictates of the standard, and
apparently alone of all the standard library functions.

The fact they cannot justify this means they cannot claim their position
*is* justified, therefore they have no argument.

So again, how about *showing* I'm wrong, rather than simply hand-waving?
Show me where, in the standard's definition of free, it allows *any* form
of implementation-defined behaviour or the like which allows free to
return the memory back to the OS in the manner you describe.

You can't. It's not there. What is there is a guarantee that the memory
is made available for further allocation, a guarantee made in defining
the behaviours and expectations of a C program.

Do, please, show me the clause which allows the behaviour you describe,
as it's not to be found in the definition of free. Nor anywhere else as
far as I can tell. You've obviously found such a clause, why can't you
quote it?

 

[Richard Tobin]

Read the definition of free;
there is no allowance for handing the memory back to the OS in the
general case, as that would risk making the memory *not* available for
further allocation.

As you surely know by now from the statements made by committee members
in the thread Micah Cowan referred us to:

http://groups.google.com/group/comp.std.c/browse_thread/thread/930b55c2892df2f1/8510e07d4d24034d

your interpretation is a misinterpretation. Instead of pretending that
your misinterpretation takes precedence over what we know to be intended,
why not submit a Defect Report asking them to clarify the wording so
that others don't make your mistake?

-- Richard

 

[Kelsey Bjarnason]

[snips]

Why? They will be brought back

No; they *may* be brought back. They may not be. No such allowance is
made; the requirement is that they *are*.

As well as there is zero guarantee about very next call to malloc(). All
you know is that it may fail.

Correct, it may. It may _not_, however, fail because the implementation
chose to hand the memory freed back to the OS. The standard forbids
this, meaning that a failure in a subsequent allocation (of the same or
lesser amount of memory) is at best a QoI issue, at worst a bug, whereas
handing the memory back to the OS such that the OS might re-use it is not
merely a QoI issue or a bug, but a non-conforming behaviour.

Yep. And a strictly conforming program can't know when that is going to
happen.

Not quite the point, though, is it? It knows the memory _will_ be
available, whereas if the memory is handed off to the OS for use outside
the scope of the program, the guarantee is violated.

No, the point is that a strictly conforming program can't possibly know
whether it's still Monday or that implementation gave the memory back to
OS. So no strictly conforming program can demonstrate that this
implementation gave memory back to OS.

It doesn't need to. Try the following:

void *ptr = malloc( 1000 ); /* this one worked, error checked omitted */
free( ptr );
ptr = NULL;

while ( ptr == NULL )
{
ptr = malloc( 100 );
}

According to the standard, this program should fall out of the loop at
some point, whether today or tomorrow or next week, as the memory has
been allocated once, freed, thus made available.

The only cases where this is not true are:

1) Where the implementation's memory management is so bad it can't
allocate a 100-byte pool from a reserved 1,000-byte arena

2) Where the implementation's memory manage is so bad that the overhead
of the subsequent allocations exhaust the pool of previously allocated
memory

3) The implementation does something really asinine, such as handing the
memory back to the OS, such that it can no longer guarantee the program
will behave in a manner defined by the standard.

The first two are QoI issues, the latter is nothing less than a non-
conforming implementation.

Assuming a decent quality memory manager in the implementation, and that
the implementer actually read the standard, the loop _must_ exit, whether
the first time or the thousandth.

If an implementation accepts and correctly executes every strictly
conforming program (plus it does what it should with incorrect programs,
like issues diagnostics), is it conforming?

If it's handing back memory to the OS when free is called, then it is not
correctly executing ever strictly conforming program, unless "strictly
conforming program" excludes those which dynamically allocate memory, in
which case the malloc failure discussed is irrelevant.

If dynamic memory allocation _is_ part of a strictly conforming program,
then the behaviour of free is strictly curtailed; the loop above *must*
terminate.

 

[Micah Cowan]

On the contrary, it is perfectly reasonable, as they have yet to give any
justification for this magical intrusion of the rest of the world into
the operation of free, contrary to the dictates of the standard, and
apparently alone of all the standard library functions.

Whether or not the interpretation opposing yours is reasonable, as I've
already pointed out elsethread (hoping, futilely apparently, that it
would finally kill this thread), the standard's _authors_ have declared
it as being the acceptable one. Therefore, whether reasonable or not,
that is how this text _must_ be interpreted.

Given that fact, please feel free to gripe that the standard's language
is completely inappropriate for that interpretation (preferably not
here, though) - I will quite agree. It doesn't change the fact, though,
that the matter has already been firmly resolved; continuing to assert
your interpretation informally (as opposed to asking the committee to
please fix the text) is therefore tantamount to being a sore loser.

 

[muntyan]

Since the very thing whose behaviour is being defined is the C program,
it is the *only* relevant context in which to define the behaviour.


Except that this ignores the very reason for the standard, which is *not*
to define behaviours of "any program".

Exactly, it defines the behavior of strictly conforming programs.
So provide an example of a strictly conforming program whose
behavior can be affected by malloc() giving memory back to OS
(and yes, failing to allocate that big block again) or stop
waving the standard like a flag. Note the "example of a strictly
conforming program" part. Everybody heard already that the standard
says there in black and white... not toasters and cars... etc.

 

[muntyan]

[snips]

Why? They will be brought back

No; they *may* be brought back. They may not be. No such allowance is
made; the requirement is that they *are*.

As well as there is zero guarantee about very next call to malloc(). All
you know is that it may fail.

Correct, it may. It may _not_, however, fail because the implementation
chose to hand the memory freed back to the OS. The standard forbids
this, meaning that a failure in a subsequent allocation (of the same or
lesser amount of memory) is at best a QoI issue, at worst a bug, whereas
handing the memory back to the OS such that the OS might re-use it is not
merely a QoI issue or a bug, but a non-conforming behaviour.

Yep. And a strictly conforming program can't know when that is going to
happen.

Not quite the point, though, is it? It knows the memory _will_ be
available, whereas if the memory is handed off to the OS for use outside
the scope of the program, the guarantee is violated.

No, the point is that a strictly conforming program can't possibly know
whether it's still Monday or that implementation gave the memory back to
OS. So no strictly conforming program can demonstrate that this
implementation gave memory back to OS.

It doesn't need to. Try the following:

void *ptr = malloc( 1000 ); /* this one worked, error checked omitted */
free( ptr );
ptr = NULL;

while ( ptr == NULL )
{
ptr = malloc( 100 );

}

According to the standard, this program should fall out of the loop at
some point, whether today or tomorrow or next week, as the memory has
been allocated once, freed, thus made available.

The only cases where this is not true are:

1) Where the implementation's memory management is so bad it can't
allocate a 100-byte pool from a reserved 1,000-byte arena

2) Where the implementation's memory manage is so bad that the overhead
of the subsequent allocations exhaust the pool of previously allocated
memory

3) The implementation does something really asinine, such as handing the
memory back to the OS, such that it can no longer guarantee the program
will behave in a manner defined by the standard.

The first two are QoI issues, the latter is nothing less than a non-
conforming implementation.

You just said yourself that your program is not strictly
conforming - 1) and 2) cases (so this is not an example
I asked for in another post).
There are other cases when malloc() could fail there,
but it's not important, we can just agree that malloc()
can legally fail for whatever reason, be it 1), 2) or 18)

[snip]

If dynamic memory allocation _is_ part of a strictly conforming program,
then the behaviour of free is strictly curtailed; the loop above *must*
terminate.

Yes, malloc() and free() can be parts of a strictly conforming
program. But if a program output depends on whether implementation
gives memory back to the OS, then it's not strictly conforming.
See 1) and 2).

Yevgen

 

[Keith Thompson]

[snips]

free() may return memory from the C program to the OS

Only if it can guarantee the OS will somehow tag that memory as reserved
for the C program. Otherwise, it cannot. Read the definition of free;
there is no allowance for handing the memory back to the OS in the
general case, as that would risk making the memory *not* available for
further allocation.

Proof by repetition?

 

[Richard Bos]

Do, please, show me

No. You cannot be shown anything while your eyes and mind are closed.

Goodbye.

Richard

 

[Ben Bacarisse]

Since the very thing whose behaviour is being defined is the C program,
it is the *only* relevant context in which to define the behaviour.

I have been sitting on my hands with this thread, but I can't any
longer. Your basis for this is that the standard only considers one
program, that there is no "OS" so how else can one interpret "being
available"?

You miss the fact that there *is* another entity here, but only by
implication. It seems reasonable to me to interpret "available for
further allocation" as returning the memory to the place it came from.
malloc does not create the memory, it "allocates" it. This implies a
sort of change of ownership from some other entity (the OS, of course)
to the program. An allocation is an assignment, from one entity to
another. Making memory "available for further allocation" could be
taken to mean the memory reverts to the state (of ownership) that it
had before.

I don't expect you to be persuaded -- I think you have decided what
the phrase means, but I don't think your argument is as watertight as
you believe it to be. I won't press this point because the thread is
long enough already and the matter is resolved for all practical
purposes.

 

[lawrence.jones]

Show me where, in the standard's definition of free, it allows *any* form
of implementation-defined behaviour or the like which allows free to
return the memory back to the OS in the manner you describe.

The committee has discussed this issue in the past and the consensus was
that returning the memory to the OS *does* make it available for future
allocation. The fact that it may not remain available for very long is
outside the scope of the standard, as is the amount of memory that is
available for allocation at any particular point in time.

-Larry Jones

I like Mom to be impressed when I fulfill the least of my obligations.
-- Calvin

 

[Herbert Rosenau]

[snips]

free() may return memory from the C program to the OS

Only if it can guarantee the OS will somehow tag that memory as reserved
for the C program. Otherwise, it cannot. Read the definition of free;
there is no allowance for handing the memory back to the OS in the
general case, as that would risk making the memory *not* available for
further allocation.

In contrary to your false assumption there is nothing in the standard
that forbids to give back borrowed memory to the OS and request it
from there again when needed.

When there were really a request on the implementation to give never
any memory back to the OS it were mostenly implossible for the
implementation to fullify many requests of different sizes after some
requests are freed.

When your implementation runs in an environment where the memory is
based on virtual addresses and a single memory block must be virtually
continous even it is really not then only the OS knows how to build a
virtual block of continous virtual memory of requested size.

--
Tschau/Bye
Herbert

Visit http://www.ecomstation.de the home of german eComStation
eComStation 1.2R Deutsch ist da!

 

[Herbert Rosenau]

[snips]

Tell me what prohibits this implementation:

[irrelevances snipped]

A C program may free memory and the implementation may return it to the
OS. The OS may then let a non-C program (or a C program running in
non-standard-compliant mode)

Non-conforming mode? Yes, well, you just answered your question: you're
using a non-conforming implementation, which means it's either broken, or
non-conforming by intent; if the former, file a bug report. If the
latter, consider it a QoI issue.

Come on, where in the standard is defined that the abstract mashine it
describes has to know of what a program is and which behavior this
program - when it exists - has to have?

Fact is that the standard says nothing about program, nothing about an
OS, nothing about real memory, nothing about files, nothing about any
real device. It defines only an abstract mashine and its behavior from
sight of C.

The standard defines explicity the interface between the C environment
and a program - when there is at least a single program (hosted) but
it lets open anything for freestanding environments. It says nothing
about memory that is on hosted environment outside the environment, it
says not even that malloc will be able to get memory from somewhere
outside the singe program. That means from your side that malloc can
never be able to request memory from an OS it runs under because it
can't exist or is always and under all circumstance unrequestable. So
you're right. What can't requested can't freed.

From your sight a program will always be unable to open a file, it
can't read or write a file, it can't remove a file too. Because the
standard tells only about streams. A stream is as the standard
describes is something abstract that gets magically data from
somewhere and delivers data magically to somewhere.

So, however an implementation can request memory for dynamic
allocation from outside the program it must be able to release it when
the program tells it that the memory is unused now. When an
implementation is unable to fullify such request because the
underlying OS can not do it, it will not even try it.

The standard simply does not distinguish between environments who are
able to borrow memory from outside the program and such who are abe to
because it needs the freedom to live this to the real environment
completely.

When you needs to know what your implementation can do to fullify a
request for dynamic allocated memory and what it will do when there is
a block of memory available to give back to the OS when it was gotten
from it then you'll have to look into the documentation of your
implementation what goes on. As the standard does not require that
this behavor has to be docomented completely there may or may not
about that documented a bit, or more or less complete.

A good designed implementation will hold a basic block of free memory
to give away through m|c|realloc for the whole runtime of the program
and borrow more from the OS so long the program needs that and give
back the borrowed memory back when possible.



--
Tschau/Bye
Herbert

Visit http://www.ecomstation.de the home of german eComStation
eComStation 1.2R Deutsch ist da!

 

[Herbert Rosenau]

Thus, for example, one need not have a specific clause which disallows
the pre-increment operator to produce a result of 19,471 from a previous
value of 3; it is sufficient to have a definition of how pre-increment is
defined to work, plus the rules for determining what to expect: the
behaviour is defined _thus_, any implementation which does otherwise is
non-conforming.

Yes, no!. Increment and decrement is well specified by the standard.
For variables are not pointers it is always 1. For pointers it is
always the number of bytes + padding bytes the object the pointer
points to.

Therefore, by defining a behaviour for free, the standard defines what
free is expected to do, how it is expected to behave.

No, it does this only partially. It does not specify
- should a single block coming back from the program
concated into the 2 neighbour blocks already free
- should a single block coming back from the program
live free for only allocated in the same or less size
- should a memory block borrowed from the OS should
be given back to the OS when this block is completely
fee again
- says nothing about that malloc can only be a wrapper
to an OS fuction that gives only memory in size of
(requested size in bytes + 1GB) / 1GB or not

It only requests explicity that a block that comes in from free should
NOT holded in use anymore but has to be given free for further
allocation for any c|m|realloc from who ever.

It does not specifiy the behavior of malloc
- should malloc request none, one, multiple blocks
of memory to fullify requests from the program
- should malloc only request memory from the OS
when the program stars up or should it do so
whenever it runs out of memory to fullify the
request of a single call when it receives one
or should it request a much larger block and
deliver this partieally?
- it does neither allow nor deny to give block of the
multiple size of the requested one out.

It does not specify the behavior of realloc completely.
- should shrinking the block result always or
noways result in moving the block around
- should expanding the block allow
- request another block from the OS or not
- compacting the list of free block to build
at least one that is big enough to hold
enough bytes

The up-front rules

about interpreting the behaviour do not need to expressly disallow the
case you mention, as the defined behaviour of free does not include such
a case *nor* does it include any allowance for implementation-defined
behaviour or the like; it behaves as documented, or the implementation
fails to meet the requirements of the standard and thus is non-conforming.

And what *is* the defined behaviour? Right - to make the memory
available for further allocation.

Sure - it does simply not allow to block the block for ever. It does
NOT forbid to give the block complete or partially back to the OS.

Ah, but wait. To *whom* does it define this behaviour? Is the C
standard suddenly the defining document for POSIX-based OSen? How about
non-POSIX-based OSen? How about process management in general? Cars?
Toaster ovens?

Nope, none of these apply. The behaviour defined is the contract between
implementation and developer, in the context of writing C programs to be
used in conjunction with a C implementation. That is *it*. No OSen, no
cars, no toaster ovens, just C, all C, nothing but C.

That it's! The block should no more blocked from allocation, it should
given away again. But not only to this program but maybe even to
another one - saying nothing about
- the block can be splitted to allow lesser requests or not
- the block can combined with neighbar ones to build one that
is bigger than the original one
It says only "don't block that block" anymore until it is given away
again.



--
Tschau/Bye
Herbert

Visit http://www.ecomstation.de the home of german eComStation
eComStation 1.2R Deutsch ist da!