does python have useless destructors?

[Michael P. Soulier]

Greetings,

I recently had a friend tell me that because the Python spec does not
guarantee that an object will necessary be destroyed when all references
to it are gone, that desctructors in Python are not reliably called, and
thus useless. As I found it difficult to believe that Python would
include destructors for apparently no reason, I thought I should ask for
clarification here.

Can anyone say yay or nay on this issue?

I believe his comments stem from a couple of references.

http://docs.python.org/ref/customization.html#l2h-174
http://docs.python.org/ref/objects.html#l2h-18

Specifically, this paragraph in the second link:

Some objects contain references to ``external'' resources such as
open files or windows. It is understood that these resources are
freed when the object is garbage-collected, but since garbage
collection is not guaranteed to happen, such objects also provide an
explicit way to release the external resource, usually a close()
method. Programs are strongly recommended to explicitly close such
objects. The `try...finally' statement provides a convenient way to
do this.

So putting a close of some external resource in a destructor would be a
bad idea, apparently. As this is the kind of thing I typically use
destructors in OO programming for, I found this quite surprising.

Regards,
Mike

 

[Donn Cave]

I recently had a friend tell me that because the Python spec does not
guarantee that an object will necessary be destroyed when all references
to it are gone, that desctructors in Python are not reliably called, and
thus useless. As I found it difficult to believe that Python would
include destructors for apparently no reason, I thought I should ask for
clarification here.

Can anyone say yay or nay on this issue?

I believe his comments stem from a couple of references.

http://docs.python.org/ref/customization.html#l2h-174
http://docs.python.org/ref/objects.html#l2h-18

Specifically, this paragraph in the second link:

Some objects contain references to ``external'' resources such as
open files or windows. It is understood that these resources are
freed when the object is garbage-collected, but since garbage
collection is not guaranteed to happen, such objects also provide an
explicit way to release the external resource, usually a close()
method. Programs are strongly recommended to explicitly close such
objects. The `try...finally' statement provides a convenient way to
do this.

So putting a close of some external resource in a destructor would be a
bad idea, apparently. As this is the kind of thing I typically use
destructors in OO programming for, I found this quite surprising.

To answer your question, no, I don't think anyone can say
yea or nay on this issue.

There are two not-guaranteed issues with finalization.

- Reference cycles can prevent immediate finalization.
If an object holds a reference to itself, however
indirectly, it won't be deleted automatically and has
to be recovered through garbage collection, and garbage
collection won't delete instances with a __del__ method,
among other things.

- The Java implementation can't implement immediate
finalization because Java doesn't have it. In general,
Python's storage model isn't fully defined and its semantics
may vary according to the host language.

On the other hand, while we can't completely resolve this
problem, I think the text you quota above errs in trying
to paint it as a non-problem. try..finally is certainly
not a convenient substitute for guaranteed finalization.
The immediate finalization normally supported in Python
is a very powerful, elegant programming feature. For just
the same reasons that automatic control of memory allocation
is powerful and elegant -- memory is one of the resources
your objects use.

So yes, you can depend on immediate finalization in many
common situations, but you have to accept the liability of
a dependence on reference non-circularity and on the C Python.

Donn Cave, (e-mail address removed)

 

[John J. Lee]

So yes, you can depend on immediate finalization in many
common situations, but you have to accept the liability of
a dependence on reference non-circularity and on the C Python.

Adding __del__ methods also, IIRC, has the disadvantage that it messes
up the GC that does reference-cycle collection. When __del__ was
introduced, Python had no automatic cycle GC.

On balance, considering all these points, I never write __del__
methods any more, and I always (well, almost always) explictly close
files and other external resources. Other people disagree, preferring
to avoid reference cycles (eg. using the weakref module) and to rule
out Jython.


John

 

[Michael P. Soulier]

On the other hand, while we can't completely resolve this
problem, I think the text you quota above errs in trying
to paint it as a non-problem. try..finally is certainly
not a convenient substitute for guaranteed finalization.
The immediate finalization normally supported in Python
is a very powerful, elegant programming feature. For just
the same reasons that automatic control of memory allocation
is powerful and elegant -- memory is one of the resources
your objects use.

So yes, you can depend on immediate finalization in many
common situations, but you have to accept the liability of
a dependence on reference non-circularity and on the C Python.

As soon as I heard about not being able to trust destructors, I was
shocked, because I do rely on guaranteed finalization, and I also
consider it to be very elegant. If the only situations where I cannot
trust it are with Jython, and circular references, then I have no issue.

However, consider the code...

myfile = open("myfilepath", "w")
myfile.write(reallybigbuffer)
myfile.close()

If the write fails due to a lack of disk space, the exception will
prevent the explicit close() from happening. Now, if myfile goes out of
scope, I would hope that the file object is destroyed and thus the file
is closed, but after reading the python documentation, it seems that
the only way to guarantee closure of the file is using the mentioned
try/finally construct...

try:
myfile = open("myfilepath", "w")
myfile.write(reallybigbuffer)
finally:
myfile.close()

Also, there is the case of Python not guaranteeing the __del__ method of
an object being called if the object is still alive when the process
exits. For files, I don't care, since the filehandles will be returned
by the OS, but for some external resources, this would be a big issue.
FIFOs and shared memory segments, for example.

Mike

 

[Peter Hansen]

Michael P. Soulier wrote:

I'll limit my response to point out that this code:

myfile = open("myfilepath", "w")
myfile.write(reallybigbuffer)
myfile.close()

.... immediately raises a warning flag in the mind of an
experienced Python programmer.

This code, on the other hand:

try:
myfile = open("myfilepath", "w")
myfile.write(reallybigbuffer)
finally:
myfile.close()

.... "feels" just right. This is how you do it when you
want to be sure the file is closed, regardless of
other considerations like garbage collection, etc.

It is simple, clean, and most of all, very explicit.

-Peter

 

[Michael Geary]

I'll limit my response to point out that this code:


... immediately raises a warning flag in the mind of an
experienced Python programmer.

This code, on the other hand:


... "feels" just right. This is how you do it when you
want to be sure the file is closed, regardless of
other considerations like garbage collection, etc.

It is simple, clean, and most of all, very explicit.

And it has a bug.

What if "myfilepath" is not writable? The open() call raises an exception,
which jumps to the finally: block, where myfile.close() fails because myfile
is undefined.

What would be the cleanest way to code this to handle that error situation?

-Mike

 

[Terry Reedy]

To me, this is useless over-dramatization ;-) Is your car useless because
there is no guarantee that the engine will never quit? or the brakes fail?

While aspects of memory management are system dependent and outside of
Python's control, programs running on standard hardware with the CPython
interpreter do what one can expect: memory is freed for reuse by the
interpreter (but not necessarily for resuse by other programs, as some
mistakenly expect).

And they are not. Under the conditions stated above, deletion works. But
Guido did not want to restrict Python to C implementation on standard
systems.

- Reference cycles can prevent immediate finalization.
If an object holds a reference to itself, however
indirectly, it won't be deleted automatically and has
to be recovered through garbage collection, and garbage
collection won't delete instances with a __del__ method,
among other things.

This is admittedly a nuisance. Howver, the original solution was reference
counting only. Programmers had to explictly break loops to reclaim
resources. This option remains available.

When cyclic gc was introduced, I believe there was no such restriction.
The result was indeterminate chaos and sometime bad behavior. After trying
several fixes, the present rule was introduced. Of course, this made
cyclic gc less useful than intended, but seg faults are a no-no. No one
has volunteered anything better since.

Terry J. Reedy

 

[Donn Cave]

Quoth "Michael P. Soulier" <[email protected]._nospam>:
....
| However, consider the code...
|
| myfile = open("myfilepath", "w")
| myfile.write(reallybigbuffer)
| myfile.close()
|
| If the write fails due to a lack of disk space, the exception will
| prevent the explicit close() from happening. Now, if myfile goes out of
| scope, I would hope that the file object is destroyed and thus the file
| is closed, but after reading the python documentation, it seems that
| the only way to guarantee closure of the file is using the mentioned
| try/finally construct...
|
| try:
| myfile = open("myfilepath", "w")
| myfile.write(reallybigbuffer)
| finally:
| myfile.close()

I think here we're talking about the problem where an exception takes
a reference to local variables. The other solution, a bit of a hack
but it could be more convenient depending on your application, is to
release the traceback (sys.exc_traceback = None) in the handler.

Donn Cave, (e-mail address removed)

 

[Hannu Kankaanp??]

This code, on the other hand:


... "feels" just right. This is how you do it when you
want to be sure the file is closed, regardless of
other considerations like garbage collection, etc.

It is simple, clean, and most of all, very explicit.

It's not that simple when you compare it to C++ RAII idiom,
and the above code is actually wrong. If open() raises an
exception, myfile hasn't yet been assigned and myfile.close()
will raise another, unwanted exception of type "NameError".
The correct idiom is:

myfile = file("myfilepath", "w")
try:
myfile.write(reallybigbuffer)
finally:
myfile.close()

I'd very much prefer something simpler that leaves no room
for errors, e.g.

with myfile = file("myfilepath", "w"):
myfile.write(reallybigbuffer)

(this, and more generic solutions with code blocks have been
suggested here many times)

 

[Michael Geary]

Hannu Kankaanp??

The correct idiom is:

myfile = file("myfilepath", "w")
try:
myfile.write(reallybigbuffer)
finally:
myfile.close()

I'd very much prefer something simpler that leaves no room
for errors, e.g.

with myfile = file("myfilepath", "w"):
myfile.write(reallybigbuffer)

(this, and more generic solutions with code blocks have been
suggested here many times)

That's just like the way you would code it in Ruby:

File.open( 'myfilepath', 'w' ) do |myfile|
myfile.write( reallybigbuffer )
end

-Mike

 

[David Turner]

Michael P. Soulier wrote:

I'll limit my response to point out that this code:


... immediately raises a warning flag in the mind of an
experienced Python programmer.

This code, on the other hand:


... "feels" just right. This is how you do it when you
want to be sure the file is closed, regardless of
other considerations like garbage collection, etc.

Keeping track of all of the possible places where an exception may
occur can be extremely difficult. Actually _catching_ all of those
exceptions often turns the code into "finally soup". Furthermore,
it's not always myfile.close() -- many objects have cleanup semantics
that are not always obvious, particularly to someone who may be using
them for the first time. Off the top of my head, an example of this
is DirectShow's filter graphs.

"Finally" can't be considered a good solution, in the light of better
solutions which are available: ruby closures, for example (with file
do...) and C++ RAII. Both solve the same problem as finally, but far
more elegantly. In fact, finally is only one step away from explicit
acquire/release cycles - the same malloc/free approach we were trying
to get away from, right?

It is simple, clean, and most of all, very explicit.

I'd challenge you on all three of those. Simple? Not to the guy who
has to remember to write finally around every resource-allocating
operation that may raise an exception. Clean? Hardly. See my
example below. Explicit? Yes, that's true, it is explicit. Much in
the same way as malloc and free are explicit. Are you sure that's
what you want?

Here are two code samples that do exactly the same thing, one in C++
and one in Python. I would be very pleased to see the day when the
Python code is at least as neat as the C++ code.

Python:
-------
self.db_lock.acquire()
try:
self.updatedb()
finally:
self.lock.release()

C++:
----
mutex::scoped_lock lock(db_lock);
updatedb();


Regards
David Turner

 

[David Turner]

So putting a close of some external resource in a destructor would be a
bad idea, apparently. As this is the kind of thing I typically use
destructors in OO programming for, I found this quite surprising.

Me too. Fortunately, in the current vanilla Python implementation,
the destructors are fairly reliable. However there is a fatal flaw:
locals get extra references when exceptions are raised. Which pretty
much means you can't use RAII wherever it would be most useful.

This makes me sad. But there is some hope in the following pattern,
although it can be quite awkward:

def with_file(fname, mode, oper):
f = open(fname, mode);
try:
oper()
finally:
f.close()

Of course, this can very quickly turn your code into so much
functional goo. That's not necessarily a bad thing, as Lisp
programmers can attest... But it's still not quite as powerful as
RAII.


Regards
David Turner

 

[Duncan Booth]

(e-mail address removed) (Hannu Kankaanp??) wrote in

I'd very much prefer something simpler that leaves no room
for errors, e.g.

with myfile = file("myfilepath", "w"):
myfile.write(reallybigbuffer)

(this, and more generic solutions with code blocks have been
suggested here many times)

One of the downsides to this form is that when you are working with several
variables that need finalising you end up nested quite deep.

Apparently the next release of Microsoft's C++ for managed environments
will handle the issue in quite a nice way. They will let you declare
managed heap based objects as though they are on the stack, and if the
object supports their dispose pattern the compiler automatically inserts
the required try..finally block. Python might need an extra keyword, but
perhaps it could benefit from something similar.

e.g.

def f(filename):
dispose infile, outfile

infile = file(filename, "r")
outfile = file(filename+".out", "w")
outfile.write(infile.read())

would be equivalent to something like:

def f(filename):
try:
infile = file(filename, "r")
outfile = file(filename+".out", "w")
outfile.write(infile.read())
finally:
_inexception = sys.exc_info()[0] is not None
for _temp in ['outfile', 'infile']:
if _temp in locals():
try:
locals()[_temp].__dispose__()
except:
pass
if not _inexception and sys.exc_info()[0] is not None:
raise

Obviously the keyword might be something other than dispose, and the method
called to dispose of the object might have a different name. Plus I'm not
sure I got the logic right on what the equivalent code should be (which I
think is an argument *for* a scheme such as this). It should be ignoring
variables which aren't yet set, and preserving the original exception if
there is one, but if any __dispose__ throws an exception that should be
propogated while not preventing all other __dispose__ methods also being
called.

 

[Stuart Bishop]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

To me, this is useless over-dramatization ;-) Is your car useless
because
there is no guarantee that the engine will never quit? or the brakes
fail?

Your metaphor is flawed - *python* is not useless. The __del__ method
is useless. Your metaphor should be 'Are your brakes useless if the
brakes fail?'.

This is admittedly a nuisance. Howver, the original solution was
reference
counting only. Programmers had to explictly break loops to reclaim
resources. This option remains available.

Only if you write everything. Is there any guarantee that cyclic
references will not be introduced into your parent class by your
co-programmer, or in the next release of Zope, Twisted, Python, etc.?
If you use __del__, you are writing fragile code in all but the most
trivial situations (where it is usually unnecessary anyway).
An argument can be made to issue warnings whenever __del__ is defined
if this cannot or will not be fixed.

- --
Stuart Bishop <[email protected]>
http://www.stuartbishop.net/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (Darwin)

iD8DBQFAyBpTAfqZj7rGN0oRAn+VAKCNRE7BViqQsLz0coRVEd57e1sALwCeKGLl
kwYQAG7QbKJiWteMdv663eQ=
=FBlh
-----END PGP SIGNATURE-----

 

[Peter Hansen]

>
And it has a bug.

What if "myfilepath" is not writable? The open() call raises an exception,
which jumps to the finally: block, where myfile.close() fails because myfile
is undefined.

What would be the cleanest way to code this to handle that error situation?

You're quite right! Sorry... I was reacting to just the
presence or absence of the exception handling code, without
even reading it. :-(

The correct approach is of course to open the file *before*
the exception block starts.

If you are concerned that the open might fail, which is a *different*
type of exception, then you probably want a try/except block around
the whole thing as well as the try/finally.

-Peter

 

[Michael P. Soulier]

I'd challenge you on all three of those. Simple? Not to the guy who
has to remember to write finally around every resource-allocating
operation that may raise an exception. Clean? Hardly. See my
example below. Explicit? Yes, that's true, it is explicit. Much in
the same way as malloc and free are explicit. Are you sure that's
what you want?

This is exactly what I've been arguing. I had one individual email me
privately and insult my opinion of elegance. Nice to know that usenet
hasn't changed much. I find the C++ code below far more elegant than the
explicit release. Exactly what is the point of encapsulation if I have
to look inside the black box?

C++:

I greatly prefer this. One of the things I hate about Perl is the fact
that I end up with "or die()" noise on every line that attempts to do
something. Python was so much cleaner in that I could just put all
suspect code in a try block. Now, it's looking like every time I acquire
something, I'll need a try/finally around it to make sure it's released.
I've traded one set of noise for another. Python programmers are always
claiming that Python is so much cleaner than most languages. Lately, I'm
not seeing that, and I think more can be done to achieve that.

Mike

 

[Hallvard B Furuseth]

(...)

The correct approach is of course to open the file *before*
the exception block starts.

If you are concerned that the open might fail, which is a *different*
type of exception, then you probably want a try/except block around
the whole thing as well as the try/finally.

In that case I usually prefer 'myfile = None' before 'try'
and 'if myfile:' before close(), to avoid an extra nesting level.

BTW, another problem with calls like close() in destructors is that
exceptions from destructors are ignored (except for a warning message).
So if close() fails because the disk is full or something, your program
will happily continue as if the file was updated successfully.

class foo:
def __del__(self): raise NotImplementedError
def bar():
x = foo()
del x
print "Still happy."
bar()
-->
Exception exceptions.NotImplementedError:
<exceptions.NotImplementedError instance at 0xf30a0>
in <bound method foo.__del__
of <__main__.foo instance at 0xf3198>> ignored
Still happy.

 

[Günter Jantzen]

If you are concerned that the open might fail, which is a *different*
type of exception, then you probably want a try/except block around
the whole thing as well as the try/finally.

Yes this would be good. The main issue is that the finally clause works
independently of what has been done before in the try clause. The programmer
should realize this and make the finally clause as independent from the try
clause as posssible

I try too avoid too much nested exception blocks because this coding style
is sometimes difficult to understand
Sometimes I initalize my resource handlers just before the try/finally
block. And I write the finally clause so, that every resource has a chance
to be closed

--------------------------------------------------
myfile, my_db42 = None, None

try:
myfile = open("myfilepath", "w")
my_db42 = database.open.(connectstring)
#
finally:
# should work independent of what happens in the try clause !!!
# Here I assume that close never throws an exception
if myfile : myfile.close()
if my_db42: my_db42.close()

 

[Humpdydum]

This makes me sad. But there is some hope in the following pattern,
although it can be quite awkward:

def with_file(fname, mode, oper):
f = open(fname, mode);
try:
oper()
finally:
f.close()

Of course, this can very quickly turn your code into so much
functional goo. That's not necessarily a bad thing, as Lisp
programmers can attest... But it's still not quite as powerful as
RAII.

I tried out a technique yesterday that shows it's possible to have reliable
'finalization' in Python, and not surprisingly it relies on try/finally (how
else...), BUT user no longer has to use this construct (it's automatic). I
posted on python-dev to get feedback about whether or not the technique
could be put right into the interpreter, which would improve performance and
robustness.

In any case, the module is currently called *scope*. The usage idiom looks
like (pythonic pseudo-code):

class YourClass(scope.NeedsFinalization):
... nothing else to change, except:
def _finalize(self):
... do the clean-up

def yourFunc(...):
# do stuff that uses one or more instances of YourClass
# that need some form of "clean-up"
...

yourFunc = scope.ScopeGuarded(yourFunc)

# do some calls to yourFunc(); whether exception or normal return,
# your instances are guaranteed to be cleaned-up before returning
yourFunc(...)

I used the term 'finalization' even though technically not rigorous.
Conceptually that's what it is because after the call to _finalize() the
object should be assumed in an unusable state.

Some advantages over try/finally:
- Coder of class makes explicit by base class that finalizaiton is
important, no need to remember to document.
- User of class knows just by looking at class def, can't overlook a
sentence saying "call this-that when done with object"
- Some error checking can be done to decrease likelyhood of user forgetting
the scope-guarding rebind or of asking for finalization before refcount 0.
- Recursive functions need no try/finally

There are a couple of limitations in the current implementation (e.g. make
it work for global scope exit,
and methods, don't know yet about thread safety) but you get the idea. The
*scope* module would make it trivial to create some modules that wrapped
some of the standard python library classes in scope.NeedsFinalization.

If you'd like to comment on the code send me an email and I'll email
scope.py back to you.

Oliver

 

[Aahz]

... immediately raises a warning flag in the mind of an
experienced Python programmer.

Depends on the circumstance, I'd think.