does python have useless destructors?

[Roger Binns]

def f(filename):
dispose infile, outfile

infile = file(filename, "r")
outfile = file(filename+".out", "w")
outfile.write(infile.read())

While that looks nice, the real problem I have with it is that
the caller/consumer of objects has to know that they need
disposing. For example they have to know that strings don't
need to be disposed of but files do. And then things get
really complicated with toolkits. For example in a graphical
toolkit do windows, device contexts, colours, brushes etc
have to be disposed of?

The correct answer of course is that the object itself
should be aware that it needs to be disposed of and that
real world resources can leak if it isn't.

Which brings us back in a full loop. Currently objects
signify that they need disposing by having a destructor.
Why not "fix" that mechanism?

Roger

 

[Peter Hansen]

Depends on the circumstance, I'd think.

If you mean that for simple scripts, it looks fine, then
I agree. Otherwise, maybe I wouldn't. ;-)

 

[Peter Hansen]

The correct answer of course is that the object itself
should be aware that it needs to be disposed of and that
real world resources can leak if it isn't.

Which brings us back in a full loop. Currently objects
signify that they need disposing by having a destructor.
Why not "fix" that mechanism?

Very likely the answer is "Because nobody has yet proposed
a workable solution to the several conflicting requirements".

Do _you_ have a solution? If you do and it really works,
it seems to me unlikely it would be ignored...

-Peter

 

[Roger Binns]

Do _you_ have a solution? If you do and it really works,
it seems to me unlikely it would be ignored...

I would do the same thing that shutdown does. Run
the destructors, but don't delete the underlying
memory. Set all the fields to None (just as shutdown
points modules at None). Finally in a second round
actually free the memory.

Or throw an exception if a situation is encountered
where the destructor can't be run because of various
constraints. I would rather know about it and fix
my code, than just have things silently fail/leak.

Roger

 

[David Turner]

Very likely the answer is "Because nobody has yet proposed
a workable solution to the several conflicting requirements".

Do _you_ have a solution? If you do and it really works,
it seems to me unlikely it would be ignored...

How about this:

-----
1. Objects of classes that declare a __del__ method shall be referred
to as "deterministic" objects. Such objects shall have a reference
count associated with. The reference count shall be incremented
atomically each time an additional reference to the object is created.
The reference count shall be decremented each time a name referring
to the object is deleted explicitly. Except in the situation
described in (2) below, the reference count shall be decremented each
time a name referring to the object becomes inaccessible due to the
set of locals to which the name belongs becoming inaccessible.

2. Where an exception is raised from a suite which contains
deterministic locals, the stack traceback shall contain only weak
references to said locals. However, the reference counts of
deterministic locals shall not be decreased until the path of
execution leaves the "except:" block that handles the exception. When
this occurs, the deterministic locals shall be unreferenced, starting
at the deepest level of the traceback, and ending at most shallow
level of the traceback, with the most recently created objects at each
level being unreferenced first.

3. When the reference count of a deterministic object reaches zero,
the __del__ method of the object shall be called.
-----

This will be a pain for the Jython implementers. However, it is
doable. Note that I never said the objects couldn't be garbage
collected, just that __del__ had to be called at certain well-defined
times. What this will involve is the Jython compiler inserting a lot
of implicit try/finally constructs.

Can anyone see a reason why this scheme wouldn't work?

Regards
David Turner

 

[Duncan Booth]

While that looks nice, the real problem I have with it is that
the caller/consumer of objects has to know that they need
disposing. For example they have to know that strings don't
need to be disposed of but files do. And then things get
really complicated with toolkits. For example in a graphical
toolkit do windows, device contexts, colours, brushes etc
have to be disposed of?

The correct answer of course is that the object itself
should be aware that it needs to be disposed of and that
real world resources can leak if it isn't.

The object itself can know that it needs to be safely disposed of, but it
cannot tell *when* to dispose of itself. My example function might create
multiple objects some of which need disposing when the function returns,
and others have a longer lifetime. The choice between disposing at the end
of the function or when some containing object is disposed has to be one
for the caller.

Taking your graphical toolkit example, the caller shouldn't need to care
whether each of those objects *needs* disposing, but they should be able to
control when it happens. That is why Robert Brewer's function works well,
you can tell it to dispose of an object and if the object doesn't have
__dispose__ as a method it simply ignores it.

 

[Roy Smith]

1. Objects of classes that declare a __del__ method shall be referred
to as "deterministic" objects. Such objects shall have a reference
count associated with. The reference count shall be incremented
atomically each time an additional reference to the object is created.
The reference count shall be decremented each time a name referring
to the object is deleted explicitly. Except in the situation
described in (2) below, the reference count shall be decremented each
time a name referring to the object becomes inaccessible due to the
set of locals to which the name belongs becoming inaccessible.
[...]
3. When the reference count of a deterministic object reaches zero,
the __del__ method of the object shall be called.

What if you do this...

class Top:
def __init__ (self):
self.bottom = Bottom ()

class Bottom:
def __del__ (self):
do something really important

top = Top ()
del top

The problem here is that while Bottom.__del__ will get called as soon at
top releases it's reference, there's nothing to guarantee that the
reference will disappear until top is garbage collected. You could fix
that by writing Top.__del__, but that assumes Top is a class you have
control of (it might be something out of a library).

Or am I not understanding the situation right?

 

[Tim Bradshaw]

This makes me sad. But there is some hope in the following pattern,
although it can be quite awkward:

def with_file(fname, mode, oper):
f = open(fname, mode);
try:
oper()
finally:
f.close()

Of course, this can very quickly turn your code into so much
functional goo. That's not necessarily a bad thing, as Lisp
programmers can attest... But it's still not quite as powerful as
RAII.

No Lisp programmer worth their salt would tolerate such a horror. In
the specific case above there's already an idiom for doing this in
Common Lisp:

(with-open-file (s fname ...)
...)

In the general case of cleanup actions, a decent Lisp programmer would
rapidly create their own idiom (arguably CL should provide a
higher-level cleanup framework as implementing the idiom correctly is
not completely simple). Mine is called LET/FINALIZING:

(let/finalizing ((s (open fname ...)))
...)

Is the same as the above WITH-OPEN-FILE incantation, but can be
generalised to any object which may need finalizing. You simply need
to tell the system how to finalize the object, either by defining a
suitable method on a generic function, or even at the site of use if
you only plan on doing it in one place:

(let/finalizing ((s (make-lock ...)
;; Call-specific finalizer
#'(lambda (l abortp)
(when abortp (warn "Whoops"))
;; and punt to the global one
(finalize-thing l abortp))))
...)

--tim

 

[Brian van den Broek]

Aahz said unto the world upon 10/06/2004 16:56:

Depends on the circumstance, I'd think.

Hi all,

I'm still learning Python as a first language since some BASIC quite some
time ago, so my level of knowledge/understanding is not too sophisticated.
From that standpoint, I am wondering why the code that Michael P.
Soulier provided above would worry an experienced Python programmer.

I've read the rest of the thread, but the ensuing references to C++ RAII,
malloc, etc. are not yet within my grasp. (I recognize I'm not really the
target audience of the thread.) I'd google for an explanation, but I don't
have a clear sense of what to google for.

I had thought I was being careful and smart by always checking for
filepath existence and always explicitly closing files, but I am wondering
what red flag I'm overlooking.

Any pointers for stuff to read to understand the comment would be appreciated.

Thanks and best to all,

Brian vdB

 

[Aahz]

If you mean that for simple scripts, it looks fine, then
I agree. Otherwise, maybe I wouldn't. ;-)

What if it's the logging routine of the app's outer exception block?

 

[Terry Reedy]

From that standpoint, I am wondering why the code that Michael P.
Soulier provided above would worry an experienced Python programmer.

A reallybigbuffer might not fit on the hard disk, in which case writing
would raise an exception that it not caught. For an application intended
to be run by thousands of users with disk of different capacities and
fullness and who would be shocked to see a Python trackback appear (or
even worse, not see it before it disappears), this would be bad.

As the next poster says, 'depends on the circumstances'. If you are
writing private code to run on your own machine with gigabytes of free
space, hardy worth worrying about.

I've read the rest of the thread, but the ensuing references to C++ RAII

With 30yrs experience, I have no idea what this is either. It must be
something new in C++ since I ceased paying attention some years ago after
settling on Python for my needs.

malloc,

short for 'memory allocate', the C function that does just that. Without
knowing that, yes, it could be parsed as 'mal locatation' or 'mall
occasion' or whatever ;-)

Terry J. Reedy

 

[Brian van den Broek]

Terry Reedy said unto the world upon 11/06/2004 11:31:

A reallybigbuffer might not fit on the hard disk, in which case writing
would raise an exception that it not caught. For an application intended

Terry J. Reedy

Thanks Terry!

Best,

Brian vdB

 

[David Bolen]

With 30yrs experience, I have no idea what this is either. It must be
something new in C++ since I ceased paying attention some years ago after
settling on Python for my needs.

Resource Acquisition is Initialization. It's certainly not a new
idiom, but I'm not sure how far back the RAII term (or at least the
acronym) was first coined. It is used in Stroustrup's "The C++
Programming Language" at least as of the 1997 edition, but I'm not
sure about earlier editions. So you may have used it without even
realizing you were doing it

It comes from a common method in C++ of using stack-allocated objects
to own or manage resources, because of the explicit construction and
destruction semantics of C++ objects (and their members) during
automatic stack unwinding in exception handling or normal function
returns. In other words, the desired approach mentioned in this
thread, of being able to encapsulate resource ownership (files, locks,
etc...) within an object, whose destructor can reliably and
predictably take care of cleaning up the resource.

It lets you write code like:

void some_function(void) {
Lock_object lock(some_parms);

- do something needing the lock -
}

The very fact that you can create the instance of Lock_object means
you have obtained the lock (thus the "acquisition is initialization"
part). And then since it is a stack based automatic variable, you are
guaranteed that it will be destroyed however you leave scope (normally,
exception whatever) and thus the lock object's destructor can release
the lock.

The predictable objection creation and destruction mechanisms let you
cascade such operations when multiple resources are needed - for
example:

void some_function(void) {
ResourceA get_resourceA();
ResourceB get_resourceB();

- do something needing both resources -
}

In this case, you'll acquire resource A and then B in that order (up
to any number of nested resources). If one should fail, any of the
others already acquired will naturally be released. And whether a
normal execution, or exception, they'll always be released in exactly
the opposite order of acquisition due to stack unwinding and the
destruction semantics.

I'm not sure if RAII is intended to also cover non-stack based
objects, in terms handing ownership of the object reference equating
transferring ownership of the resource. Certainly the classic example
is the stack based approach.

Some google searches should probably show up a variety of descriptions
and further information.

-- David

 

[Humpdydum]

The object itself can know that it needs to be safely disposed of, but it
cannot tell *when* to dispose of itself. My example function might create
multiple objects some of which need disposing when the function returns,
and others have a longer lifetime. The choice between disposing at the end
of the function or when some containing object is disposed has to be one
for the caller.

The object actually *does* know when to dispose of itself: when its ref
count goes to zero. Naturally, if you have an object that wraps a resource
that needs cleanup, it goes without saying that that object shouldnt' be
referenced all over the place; rather, there should be only one ref to it,
in the function that instantiates it, and that's all. Then when the function
exits, the object sees ref cnt=0 and knows to cleanup after itself.

Also naturally, if you have (inadvertently) created a cyclical reference, in
which one of the members of the cycle refers to your object, its ref cnt
will not be zero on function exit. In this case such object should force a
garbage collection and test gc.garbage and see if somehow it is involved in
a cycle. If not, it's the user's fault (some other refs were stored
somewhere), otherwise, need to find a way to break the cycle. In any case
each object individually knows when to destroy itself.

Oliver.

 

[Peter Hansen]

The object actually *does* know when to dispose of itself: when its ref
count goes to zero. Naturally, if you have an object that wraps a resource
that needs cleanup, it goes without saying that that object shouldnt' be
referenced all over the place; rather, there should be only one ref to it,
in the function that instantiates it, and that's all. Then when the function
exits, the object sees ref cnt=0 and knows to cleanup after itself.

This is an unrealistic requirement. It implies that all access
to some resource must occur within a single function, which is
too great a limitation for many real applications.

-Peter

 

[Slawomir Nowaczyk]

On Fri, 11 Jun 2004 10:34:26 -0400

#> I've read the rest of the thread, but the ensuing references to C++
#> RAII, malloc, etc. are not yet within my grasp. (I recognize I'm
#> not really the target audience of the thread.) I'd google for an
#> explanation, but I don't have a clear sense of what to google for.

If you are looking for a short description of RAII (Resource
Acquisiton Is Initialization), I would suggest this link:

http://www.research.att.com/~bs/bs_faq2.html#finally

HTH

--
Best wishes,
Slawomir Nowaczyk
( (e-mail address removed) )

As I said before, I never repeat myself.

 

[David Turner]

1. Objects of classes that declare a __del__ method shall be referred
to as "deterministic" objects. Such objects shall have a reference
count associated with. The reference count shall be incremented
atomically each time an additional reference to the object is created.
The reference count shall be decremented each time a name referring
to the object is deleted explicitly. Except in the situation
described in (2) below, the reference count shall be decremented each
time a name referring to the object becomes inaccessible due to the
set of locals to which the name belongs becoming inaccessible.
[...]
3. When the reference count of a deterministic object reaches zero,
the __del__ method of the object shall be called.

What if you do this...

class Top:
def __init__ (self):
self.bottom = Bottom ()

class Bottom:
def __del__ (self):
do something really important

top = Top ()
del top

The problem here is that while Bottom.__del__ will get called as soon at
top releases it's reference, there's nothing to guarantee that the
reference will disappear until top is garbage collected. You could fix
that by writing Top.__del__, but that assumes Top is a class you have
control of (it might be something out of a library).

Or am I not understanding the situation right?

This particular example you cite is not actually problematic. The
point is that the resources are cleaned up when Top is cleaned up,
which is what we want. How and when Top is cleaned is neither here
nor there.

However, you have touched on a deep issue. When an object is cleaned
up, whether using an explicit __del__ or implicitly, any deterministic
objects it contains will of course have to be cleaned up (in the
reverse order of allocation, again). With an explicit __del__, this
is probably okay. But when a deterministic object is in the scope of
a non-deterministic object, there could be an implementation problem.
Perhaps someone with more insight into Jython than I could shed some
light on whether or not this would be an issue?

Regards
David Turner

 

[Paul Rubin]

It lets you write code like:

void some_function(void) {
Lock_object lock(some_parms);

- do something needing the lock -
}

There's a PEP for something like that in Python:

http://www.python.org/peps/pep-0310.html

I think that's greatly preferable to dictating that the GC system act
a certain way and free resources as soon as they go out of scope.
That's just not the correct semantics for GC. GC should simply create
the illusion for the application that all objects stay around forever.

 

[Roger Binns]

The object itself can know that it needs to be safely disposed of, but it
cannot tell *when* to dispose of itself. My example function might create
multiple objects some of which need disposing when the function returns,
and others have a longer lifetime. The choice between disposing at the end
of the function or when some containing object is disposed has to be one
for the caller.

You have totally confused me now. Ignoring special cases such as cycles,
Python destructs an object when there are no references left to it,
and in CPython that happens at the moment the last reference is released.
So in normal code the objects will go away as names referencing them go
out of scope.

If the Python object is a proxy for another non-Python visible object
(such as a C level filehandle or GUI Window). In those cases
the extension module author has to deal with the situation, either by
adding an extra reference (the C level object is then the remaining reference)
or handling the lifetime of the C level object independently.

But for this thread, the actual problem is that __del__ methods may
not be called on objects, and if you read the doc implies they almost
never will be called. So we have the counter-intuitive (and IMHO user
hostile) situation where someone marks a class as needing extra code
to run to destroy it, and Python addresses that by saying you will
be lucky if the object will be destroyed (ie by not destroying the
object).

The band aid is requiring the programmer/caller to call various
"close" methods (effectively manually calling a destructor equivalent)
with all the issues that entail, get real complex real quick and are
prone to errors or ommissions, and are extremely difficult to test.
And you can see the difficulties in this thread with the issues
of just dealing with a single file handle.

I certainly agree that dealing with destructors from a language
implementor's point of view is hard. Try dealing with cycles,
resurrection, exceptions etc. But handwaving and just not
running them coupled with expecting manual code by callers all
over the place is a bad solution. Garbage collection is also
hard, but all recent languages do it.

Roger

 

[David Turner]

No Lisp programmer worth their salt would tolerate such a horror. In
the specific case above there's already an idiom for doing this in
Common Lisp:

Actually I was thinking more of the open() function than with the
with_file() function when I said "functional goo" .

It's really quite funny that Lisp doesn't suffer much from this sort
of problem. You'd think we'd learn...

Regards
David Turner