Destructors and exceptions

[David Turner]

Hi all

I noticed something interesting while testing some RAII concepts
ported from C++ in Python. I haven't managed to find any information
about it on the web, hence this post.

The problem is that when an exception is raised, the destruction of
locals appears to be deferred to program exit. Am I missing
something? Is this behaviour by design? If so, is there any reason
for it? The only rationale I can think of is to speed up exception
handling; but as this approach breaks many safe programming idioms, I
see it as a poor trade.

Here is the code in question:

------------------------------------------
class Foo:
def __init__(self):
print "--foo %s created" % id(self)
def __del__(self):
print "--foo %s destroyed" % id(self)

def normal_exit():
print "normal_exit starts"
x = Foo()
print "normal_exit ends"

def premature_exit():
print "premature_exit starts"
x = Foo()
return 0
print "premature_exit ends"

def exceptional_exit():
print "exceptional_exit starts"
x = Foo()
raise "oops"
print "exceptional_exit ends"

if __name__ == "__main__":
print "main block starts"
try:
normal_exit()
premature_exit()
exceptional_exit()
except:
print "exception"
print "main block ends"
------------------------------------------

The output I get is:

------------------------------------------
main block starts
normal_exit starts
--foo 141819532 created
normal_exit ends
--foo 141819532 destroyed
premature_exit starts
--foo 141819532 created
--foo 141819532 destroyed
exceptional_exit starts
--foo 141819532 created
exception
main block ends
--foo 141819532 destroyed
------------------------------------------

....which indicates to me that the destruction of the local in
exceptional_exit() only happens when the program exits. Surely it
should occur at the point at which the exception is raised?

Regards
David Turner

 

[Terry Reedy]

The problem is that when an exception is raised, the destruction of
locals appears to be deferred to program exit. Am I missing
something?

When the last reference to an object disappears, the interpreter *may* but
is *not required* to forget or destruct the object, if indeed the concept
is meaningful. What happens is implementation dependent. CPython usually
cleans up immediately, which is sooner than with Jython. I am not sure
what human interpreters do. Write once, never erase storage of objects
(keeping a complete audit trail of objects created) would also be legal.

If you want to force the issue, give your class a close method, as with
file objects, and call it explicitly. Then code in try:finally:, which is
designed for this purpose.
try:
process
finally:
cleanup

Terry J. Reedy

 

[Humpty Dumpty]

The problem is that when an exception is raised, the destruction of
locals appears to be deferred to program exit. Am I missing
something? Is this behaviour by design? If so, is there any reason
for it? The only rationale I can think of is to speed up exception
handling; but as this approach breaks many safe programming idioms, I
see it as a poor trade.

There is no Python equivalent of C++'s "destructor garanteed to be called
upon scope exit", for a couple of reasons: scope exit only destroys
references to objects, not the objects themselves; destruction of objects is
left to the garbage collector and you have no influence on it. In
particular, the gc is not required to release resources, so finalizers (the
__del__ method, closest to C++'s destructor) may not get called. This means
__del__ is pretty much useless (AFAIMC), and you can't rely on them being
called before program exit (or ever, for that matter).

I agree, it is a real pitty that Python doesn't have a way of doing what you
mention, other than try-finally, which makes code more difficult to read. A
new specifier, e.g. "scoped", would be a required addtion to Python:
interpreter would garantee that __del__ of scoped objects would be called on
scope exit, and raise an exception if attempt to alias.

Oliver

 

[David Turner]

Hi

If you want to force the issue, give your class a close method, as with
file objects, and call it explicitly. Then code in try:finally:, which is
designed for this purpose.
try:
process
finally:
cleanup

That's a great shame. "Finally" is of course susceptible to the
"forgetful programmer" syndrom. It also pollutes the code with what I
consider to be implementation details -- for example, the fact that a
file must be closed is a characteristic of the file object, and only
accidentally relevant to the _use_ of files. Python would be a much
stronger language if it could guarantee deterministic destruction of
at least those objects which have a __del__ method. Can anyone point
out any technical reasons why such a feature should not be included in
the language?

At any rate, I wonder if there is an another solution to the problem
of resource management. Perhaps something along the lines of Ruby
closures? Any suggestions? There has to be something more elegant
(and safer!) than "finally".

Regards
David Turner

 

[Dominic]

Hi




That's a great shame. "Finally" is of course susceptible to the
"forgetful programmer" syndrom. It also pollutes the code with what I
consider to be implementation details -- for example, the fact that a
file must be closed is a characteristic of the file object, and only
accidentally relevant to the _use_ of files. Python would be a much
stronger language if it could guarantee deterministic destruction of

Well, there had been some discussion before. You can
wrap the open/close actions in a closure and write
something like with_file_do(myFunction).

Ciao,
Dominic

 

[Christos TZOTZIOY Georgiou]

Hi all

I noticed something interesting while testing some RAII concepts
ported from C++ in Python. I haven't managed to find any information
about it on the web, hence this post.

The problem is that when an exception is raised, the destruction of
locals appears to be deferred to program exit. Am I missing
something? Is this behaviour by design? If so, is there any reason
for it? The only rationale I can think of is to speed up exception
handling; but as this approach breaks many safe programming idioms, I
see it as a poor trade.

[snip code]

ISTR that when an exception is raised, there is a reference to the code
frame (and therefore to its locals) kept somewhere; you can access this
info through the sys.exc_info call. Have you tried running
sys.exc_clear after the exception was raised?

Apart from that, advice from others not to rely on finalisation of
objects to clean up still applies.

 

[David Turner]

Hi

There is no Python equivalent of C++'s "destructor garanteed to be called
upon scope exit", for a couple of reasons: scope exit only destroys
references to objects, not the objects themselves; destruction of objects is
left to the garbage collector and you have no influence on it. In
particular, the gc is not required to release resources, so finalizers (the
__del__ method, closest to C++'s destructor) may not get called. This means
__del__ is pretty much useless (AFAIMC), and you can't rely on them being
called before program exit (or ever, for that matter).

In the documentation for the "gc" module, it states that objects with
a __del__ method are not subject to garbage collection; they are
collected using reference counting. Which means that one can rely on
locals (to which there is only one reference) being destroyed in a
predictable fashion -- but there's an exception. Section 3.1 of the
Python Reference Manual specifies that raising exceptions may keep
objects alive. What I take this to mean from an implementation point
of view is that the exception mechanism doesn't unwind scope.
Therefore, the destruction of locals in the presence of an exception
is deferred to global clean-up time when the program exits.

I can't think of any technical objections to having the exception
mechanism also release references to locals that are going to go out
of scope (unless one was planning to support resuming). Can you?

Regards
David Turner

 

[Christos TZOTZIOY Georgiou]

I can't think of any technical objections to having the exception
mechanism also release references to locals that are going to go out
of scope (unless one was planning to support resuming). Can you?

Debugging.

 

[Nick Jacobson]

You know, I noticed this in the Python Reference Manual, p. 13, and
have been wondering about it.

"...note that catching an exception with a 'try...except' statement
may keep objects alive..."

No explanation is given, and I don't know why that's the case either.
But at least they're aware of it...HTH

--Nick

 

[Duncan Booth]

(e-mail address removed) (Nick Jacobson) wrote in

You know, I noticed this in the Python Reference Manual, p. 13, and
have been wondering about it.

"...note that catching an exception with a 'try...except' statement
may keep objects alive..."

No explanation is given, and I don't know why that's the case either.
But at least they're aware of it...HTH

When an exception is handled, you can access the stack traceback. This
contains references to all the local variables which were in scope at the
point when the exception was thrown. Normally these variables would be
destroyed when the functions return, but if there is a traceback
referencing them they stay alive as long as the traceback is accessible
which is usually until the next exception is thrown.

If you really want to be sure your objects are destroyed, then use 'del' in
the finally suite of a try..finally. This ensures that the traceback can't
be used to reference the objects that you deleted.

 

[David Turner]

Hi

When an exception is handled, you can access the stack traceback. This
contains references to all the local variables which were in scope at the
point when the exception was thrown. Normally these variables would be
destroyed when the functions return, but if there is a traceback
referencing them they stay alive as long as the traceback is accessible
which is usually until the next exception is thrown.

Then why not unreference the traceback (and therefore destroy it and
the down-stack locals, if the exception handler hasn't made another
reference) at the end of the handling suite?

If you really want to be sure your objects are destroyed, then use 'del' in
the finally suite of a try..finally. This ensures that the traceback can't
be used to reference the objects that you deleted.

As I've pointed out elsewhere, this works but it doesn't address the
problem I'm trying to solve. The problem is that it's easier for the
user of an object (who didn't create it, and probably hasn't read the
documentation) to omit the try/finally construct. Therefore, this is
what he will tend to do. I'm looking at what I, as a designer, can do
to prevent this from happening.

Regards
David Turner

 

[Duncan Booth]

(e-mail address removed) (David Turner) wrote in

Then why not unreference the traceback (and therefore destroy it and
the down-stack locals, if the exception handler hasn't made another
reference) at the end of the handling suite?

Fine, except that isn't what Python does now, and the current behaviour
needs to be kept for compatibility. So if that is the behaviour you want,
you have to do that explicitly yourself at the end of every exception
handler.

 

[Konstantin Veretennicov]

Hi all

I noticed something interesting while testing some RAII concepts
ported from C++ in Python.

AFAIK, cpp-style RAII is mostly unportable to other languages.

I haven't managed to find any information
about it on the web, hence this post.

This discussion may be helpful:
http://mail.python.org/pipermail/python-list/2002-March/090979.html

The problem is that when an exception is raised, the destruction of
locals appears to be deferred to program exit. Am I missing
something? Is this behaviour by design? If so, is there any reason
for it? The only rationale I can think of is to speed up exception
handling; but as this approach breaks many safe programming idioms, I
see it as a poor trade.

My impression is that many (or all?) languages with GC (especially
non-refcounting) don't guarantee deterministic destruction of objects.
I guess it's hard to have both GC and DD Please correct me if I am wrong.

See also "Deterministic Destruction can be a Bug" for an example when DD
can be a bad thing:
http://www.hpl.hp.com/personal/Hans_Boehm/gc/det_destr.html

All that said, GC-enabled language still can provide some support for RAII
pattern. Consider this C# example:

using (Font f1 = new Font("Arial", 10), f2 = new Font("Arial", 12)) {
// use f1 and f2...
} // compiler will call Dispose on f1 and f2 either on exit or on exception;
// not sure about order of disposal, but you can nest "using" anyway

This saves you from trouble of coding try+finally+Dispose.
BTW, does anybody know whether MS borrowed or invented this construct?

- kv

 

[Konstantin Veretennicov]

There is no Python equivalent of C++'s "destructor garanteed to be called
upon scope exit", for a couple of reasons: scope exit only destroys
references to objects, not the objects themselves; destruction of objects is
left to the garbage collector and you have no influence on it. In
particular, the gc is not required to release resources, so finalizers (the
__del__ method, closest to C++'s destructor) may not get called. This means
__del__ is pretty much useless (AFAIMC), and you can't rely on them being
called before program exit (or ever, for that matter).

I agree, it is a real pitty that Python doesn't have a way of doing what you
mention, other than try-finally, which makes code more difficult to read. A
new specifier, e.g. "scoped", would be a required addtion to Python:
interpreter would garantee that __del__ of scoped objects would be called on
scope exit, and raise an exception if attempt to alias.

Is there a PEP or something for "scoped specifier"?

- kv

 

[Bernhard Herzog]

Is there a PEP or something for "scoped specifier"?

http://python.org/peps/pep-0310.html

Bernhard

 

[David Turner]

Fine, except that isn't what Python does now, and the current behaviour
needs to be kept for compatibility. So if that is the behaviour you want,
you have to do that explicitly yourself at the end of every exception
handler.

Are you sure the current behaviour needs to be kept? Isn't
referencing the traceback outside of an exception handler a little
dodgy in the first place? I'm sorry if I sound argumentative, but I
do want to understand the issues thoroughly .

Regards
David Turner

 

[Duncan Booth]

(e-mail address removed) (David Turner) wrote in

Are you sure the current behaviour needs to be kept? Isn't
referencing the traceback outside of an exception handler a little
dodgy in the first place? I'm sorry if I sound argumentative, but I
do want to understand the issues thoroughly .

The documentation says:

exc_info( )

This function returns a tuple of three values that give information
about the exception that is currently being handled. The information
returned is specific both to the current thread and to the current
stack frame. If the current stack frame is not handling an exception,
the information is taken from the calling stack frame, or its caller,
and so on until a stack frame is found that is handling an exception.
Here, ``handling an exception'' is defined as ``executing or having
executed an except clause.'' For any stack frame, only information
about the most recently handled exception is accessible.

So, in fact I was wrong. The exception gets cleared when the function that
handled it returns:
.... try:
.... raise RuntimeError
.... except:
.... print sys.exc_info()
.... print sys.exc_info()
....(<class exceptions.RuntimeError at 0x00864810>, <exceptions.RuntimeError
instance at 0x008EA940>, <traceback object at 0x008EA968>)
.... sys.exc_clear()
.... f()
.... print sys.exc_info()
....(<class exceptions.RuntimeError at 0x00864810>, <exceptions.RuntimeError
instance at 0x008EA990>, <traceback object at 0x008EA940>)
(<class exceptions.RuntimeError at 0x00864810>, <exceptions.RuntimeError


Evidently Python does clear the exception when you leave the exception
handler, it is just that the exception handler stretches a bit further than
you might expect. This of course matters more when you put the exception
handler inside a loop, or do more processing after the handler has caught
the exception. I'm not sure how much effect it would have to restrict the
handler to actually inside the except clause.

In fact further investigation shows that the exception context is saved and
restored across function calls:
.... try:
.... raise ValueError, 'h'
.... except:
.... print sys.exc_info()
.... f()
.... print sys.exc_info()
....(<class exceptions.ValueError at 0x00864DB0>, <exceptions.ValueError
instance at 0x008EAAA8>, <traceback object at 0x008EA940>)
(<class exceptions.RuntimeError at 0x00864810>, <exceptions.RuntimeError
instance at 0x008EA760>, <traceback object at 0x008EAAD0>)
(<class exceptions.RuntimeError at 0x00864810>, <exceptions.RuntimeError
instance at 0x008EA760>, <traceback object at 0x008EAAD0>)

I never knew it did that.

 

[David Turner]

(e-mail address removed) (David Turner) wrote in message
using (Font f1 = new Font("Arial", 10), f2 = new Font("Arial", 12)) {
// use f1 and f2...
} // compiler will call Dispose on f1 and f2 either on exit or on exception;
// not sure about order of disposal, but you can nest "using" anyway

This saves you from trouble of coding try+finally+Dispose.
BTW, does anybody know whether MS borrowed or invented this construct?

It still leaves the clean-up onus in the wrong place.

The "using" concept has been kicking around for a while. I think it's
a natural evolution of Lisp's with-foo-do idiom.

Regards
David Turner