Destructors and exceptions

Discussion in 'Python' started by David Turner, Jun 7, 2004.

  1. David Turner

    David Turner Guest

    Hi all

    I noticed something interesting while testing some RAII concepts
    ported from C++ in Python. I haven't managed to find any information
    about it on the web, hence this post.

    The problem is that when an exception is raised, the destruction of
    locals appears to be deferred to program exit. Am I missing
    something? Is this behaviour by design? If so, is there any reason
    for it? The only rationale I can think of is to speed up exception
    handling; but as this approach breaks many safe programming idioms, I
    see it as a poor trade.

    Here is the code in question:

    ------------------------------------------
    class Foo:
    def __init__(self):
    print "--foo %s created" % id(self)
    def __del__(self):
    print "--foo %s destroyed" % id(self)

    def normal_exit():
    print "normal_exit starts"
    x = Foo()
    print "normal_exit ends"

    def premature_exit():
    print "premature_exit starts"
    x = Foo()
    return 0
    print "premature_exit ends"

    def exceptional_exit():
    print "exceptional_exit starts"
    x = Foo()
    raise "oops"
    print "exceptional_exit ends"

    if __name__ == "__main__":
    print "main block starts"
    try:
    normal_exit()
    premature_exit()
    exceptional_exit()
    except:
    print "exception"
    print "main block ends"
    ------------------------------------------

    The output I get is:

    ------------------------------------------
    main block starts
    normal_exit starts
    --foo 141819532 created
    normal_exit ends
    --foo 141819532 destroyed
    premature_exit starts
    --foo 141819532 created
    --foo 141819532 destroyed
    exceptional_exit starts
    --foo 141819532 created
    exception
    main block ends
    --foo 141819532 destroyed
    ------------------------------------------

    ....which indicates to me that the destruction of the local in
    exceptional_exit() only happens when the program exits. Surely it
    should occur at the point at which the exception is raised?

    Regards
    David Turner
     
    David Turner, Jun 7, 2004
    #1
    1. Advertising

  2. David Turner

    Terry Reedy Guest

    "David Turner" <> wrote in message
    news:...
    > The problem is that when an exception is raised, the destruction of
    > locals appears to be deferred to program exit. Am I missing
    > something?


    When the last reference to an object disappears, the interpreter *may* but
    is *not required* to forget or destruct the object, if indeed the concept
    is meaningful. What happens is implementation dependent. CPython usually
    cleans up immediately, which is sooner than with Jython. I am not sure
    what human interpreters do. Write once, never erase storage of objects
    (keeping a complete audit trail of objects created) would also be legal.

    If you want to force the issue, give your class a close method, as with
    file objects, and call it explicitly. Then code in try:finally:, which is
    designed for this purpose.
    try:
    process
    finally:
    cleanup

    Terry J. Reedy
     
    Terry Reedy, Jun 7, 2004
    #2
    1. Advertising

  3. "David Turner" <> wrote in message
    news:...
    >


    > The problem is that when an exception is raised, the destruction of
    > locals appears to be deferred to program exit. Am I missing
    > something? Is this behaviour by design? If so, is there any reason
    > for it? The only rationale I can think of is to speed up exception
    > handling; but as this approach breaks many safe programming idioms, I
    > see it as a poor trade.



    There is no Python equivalent of C++'s "destructor garanteed to be called
    upon scope exit", for a couple of reasons: scope exit only destroys
    references to objects, not the objects themselves; destruction of objects is
    left to the garbage collector and you have no influence on it. In
    particular, the gc is not required to release resources, so finalizers (the
    __del__ method, closest to C++'s destructor) may not get called. This means
    __del__ is pretty much useless (AFAIMC), and you can't rely on them being
    called before program exit (or ever, for that matter).

    I agree, it is a real pitty that Python doesn't have a way of doing what you
    mention, other than try-finally, which makes code more difficult to read. A
    new specifier, e.g. "scoped", would be a required addtion to Python:
    interpreter would garantee that __del__ of scoped objects would be called on
    scope exit, and raise an exception if attempt to alias.

    Oliver
     
    Humpty Dumpty, Jun 8, 2004
    #3
  4. David Turner

    David Turner Guest

    Hi

    > If you want to force the issue, give your class a close method, as with
    > file objects, and call it explicitly. Then code in try:finally:, which is
    > designed for this purpose.
    > try:
    > process
    > finally:
    > cleanup
    >


    That's a great shame. "Finally" is of course susceptible to the
    "forgetful programmer" syndrom. It also pollutes the code with what I
    consider to be implementation details -- for example, the fact that a
    file must be closed is a characteristic of the file object, and only
    accidentally relevant to the _use_ of files. Python would be a much
    stronger language if it could guarantee deterministic destruction of
    at least those objects which have a __del__ method. Can anyone point
    out any technical reasons why such a feature should not be included in
    the language?

    At any rate, I wonder if there is an another solution to the problem
    of resource management. Perhaps something along the lines of Ruby
    closures? Any suggestions? There has to be something more elegant
    (and safer!) than "finally".

    Regards
    David Turner
     
    David Turner, Jun 8, 2004
    #4
  5. David Turner

    Dominic Guest

    David Turner wrote:
    > Hi
    >
    >
    >>If you want to force the issue, give your class a close method, as with
    >>file objects, and call it explicitly. Then code in try:finally:, which is
    >>designed for this purpose.
    >>try:
    >> process
    >>finally:
    >> cleanup
    >>

    >
    >
    > That's a great shame. "Finally" is of course susceptible to the
    > "forgetful programmer" syndrom. It also pollutes the code with what I
    > consider to be implementation details -- for example, the fact that a
    > file must be closed is a characteristic of the file object, and only
    > accidentally relevant to the _use_ of files. Python would be a much
    > stronger language if it could guarantee deterministic destruction of

    Well, there had been some discussion before. You can
    wrap the open/close actions in a closure and write
    something like with_file_do(myFunction).

    Ciao,
    Dominic

    > at least those objects which have a __del__ method. Can anyone point
    > out any technical reasons why such a feature should not be included in
    > the language?
    >
    > At any rate, I wonder if there is an another solution to the problem
    > of resource management. Perhaps something along the lines of Ruby
    > closures? Any suggestions? There has to be something more elegant
    > (and safer!) than "finally".
    >
    > Regards
    > David Turner
     
    Dominic, Jun 8, 2004
    #5
  6. On 7 Jun 2004 07:51:23 -0700, rumours say that
    (David Turner) might have written:

    >Hi all
    >
    >I noticed something interesting while testing some RAII concepts
    >ported from C++ in Python. I haven't managed to find any information
    >about it on the web, hence this post.
    >
    >The problem is that when an exception is raised, the destruction of
    >locals appears to be deferred to program exit. Am I missing
    >something? Is this behaviour by design? If so, is there any reason
    >for it? The only rationale I can think of is to speed up exception
    >handling; but as this approach breaks many safe programming idioms, I
    >see it as a poor trade.


    [snip code]

    ISTR that when an exception is raised, there is a reference to the code
    frame (and therefore to its locals) kept somewhere; you can access this
    info through the sys.exc_info call. Have you tried running
    sys.exc_clear after the exception was raised?

    Apart from that, advice from others not to rely on finalisation of
    objects to clean up still applies.
    --
    TZOTZIOY, I speak England very best,
    "I have a cunning plan, m'lord" --Sean Bean as Odysseus/Ulysses
     
    Christos TZOTZIOY Georgiou, Jun 8, 2004
    #6
  7. David Turner

    David Turner Guest

    Hi

    >
    > There is no Python equivalent of C++'s "destructor garanteed to be called
    > upon scope exit", for a couple of reasons: scope exit only destroys
    > references to objects, not the objects themselves; destruction of objects is
    > left to the garbage collector and you have no influence on it. In
    > particular, the gc is not required to release resources, so finalizers (the
    > __del__ method, closest to C++'s destructor) may not get called. This means
    > __del__ is pretty much useless (AFAIMC), and you can't rely on them being
    > called before program exit (or ever, for that matter).
    >


    In the documentation for the "gc" module, it states that objects with
    a __del__ method are not subject to garbage collection; they are
    collected using reference counting. Which means that one can rely on
    locals (to which there is only one reference) being destroyed in a
    predictable fashion -- but there's an exception. Section 3.1 of the
    Python Reference Manual specifies that raising exceptions may keep
    objects alive. What I take this to mean from an implementation point
    of view is that the exception mechanism doesn't unwind scope.
    Therefore, the destruction of locals in the presence of an exception
    is deferred to global clean-up time when the program exits.

    I can't think of any technical objections to having the exception
    mechanism also release references to locals that are going to go out
    of scope (unless one was planning to support resuming). Can you?

    Regards
    David Turner
     
    David Turner, Jun 8, 2004
    #7
  8. On 8 Jun 2004 02:44:04 -0700, rumours say that
    (David Turner) might have written:

    >I can't think of any technical objections to having the exception
    >mechanism also release references to locals that are going to go out
    >of scope (unless one was planning to support resuming). Can you?


    Debugging.
    --
    TZOTZIOY, I speak England very best,
    "I have a cunning plan, m'lord" --Sean Bean as Odysseus/Ulysses
     
    Christos TZOTZIOY Georgiou, Jun 8, 2004
    #8
  9. You know, I noticed this in the Python Reference Manual, p. 13, and
    have been wondering about it.

    "...note that catching an exception with a 'try...except' statement
    may keep objects alive..."

    No explanation is given, and I don't know why that's the case either.
    But at least they're aware of it...HTH

    --Nick

    (David Turner) wrote in message news:<>...
    > Hi all
    >
    > I noticed something interesting while testing some RAII concepts
    > ported from C++ in Python. I haven't managed to find any information
    > about it on the web, hence this post.
    >
    > The problem is that when an exception is raised, the destruction of
    > locals appears to be deferred to program exit. Am I missing
    > something? Is this behaviour by design? If so, is there any reason
    > for it? The only rationale I can think of is to speed up exception
    > handling; but as this approach breaks many safe programming idioms, I
    > see it as a poor trade.
    >
    > Here is the code in question:
    >
    > ------------------------------------------
    > class Foo:
    > def __init__(self):
    > print "--foo %s created" % id(self)
    > def __del__(self):
    > print "--foo %s destroyed" % id(self)
    >
    > def normal_exit():
    > print "normal_exit starts"
    > x = Foo()
    > print "normal_exit ends"
    >
    > def premature_exit():
    > print "premature_exit starts"
    > x = Foo()
    > return 0
    > print "premature_exit ends"
    >
    > def exceptional_exit():
    > print "exceptional_exit starts"
    > x = Foo()
    > raise "oops"
    > print "exceptional_exit ends"
    >
    > if __name__ == "__main__":
    > print "main block starts"
    > try:
    > normal_exit()
    > premature_exit()
    > exceptional_exit()
    > except:
    > print "exception"
    > print "main block ends"
    > ------------------------------------------
    >
    > The output I get is:
    >
    > ------------------------------------------
    > main block starts
    > normal_exit starts
    > --foo 141819532 created
    > normal_exit ends
    > --foo 141819532 destroyed
    > premature_exit starts
    > --foo 141819532 created
    > --foo 141819532 destroyed
    > exceptional_exit starts
    > --foo 141819532 created
    > exception
    > main block ends
    > --foo 141819532 destroyed
    > ------------------------------------------
    >
    > ...which indicates to me that the destruction of the local in
    > exceptional_exit() only happens when the program exits. Surely it
    > should occur at the point at which the exception is raised?
    >
    > Regards
    > David Turner
     
    Nick Jacobson, Jun 8, 2004
    #9
  10. David Turner

    Duncan Booth Guest

    (Nick Jacobson) wrote in
    news::

    > You know, I noticed this in the Python Reference Manual, p. 13, and
    > have been wondering about it.
    >
    > "...note that catching an exception with a 'try...except' statement
    > may keep objects alive..."
    >
    > No explanation is given, and I don't know why that's the case either.
    > But at least they're aware of it...HTH
    >


    When an exception is handled, you can access the stack traceback. This
    contains references to all the local variables which were in scope at the
    point when the exception was thrown. Normally these variables would be
    destroyed when the functions return, but if there is a traceback
    referencing them they stay alive as long as the traceback is accessible
    which is usually until the next exception is thrown.

    If you really want to be sure your objects are destroyed, then use 'del' in
    the finally suite of a try..finally. This ensures that the traceback can't
    be used to reference the objects that you deleted.
     
    Duncan Booth, Jun 8, 2004
    #10
  11. David Turner

    David Turner Guest

    Hi

    > When an exception is handled, you can access the stack traceback. This
    > contains references to all the local variables which were in scope at the
    > point when the exception was thrown. Normally these variables would be
    > destroyed when the functions return, but if there is a traceback
    > referencing them they stay alive as long as the traceback is accessible
    > which is usually until the next exception is thrown.


    Then why not unreference the traceback (and therefore destroy it and
    the down-stack locals, if the exception handler hasn't made another
    reference) at the end of the handling suite?


    > If you really want to be sure your objects are destroyed, then use 'del' in
    > the finally suite of a try..finally. This ensures that the traceback can't
    > be used to reference the objects that you deleted.


    As I've pointed out elsewhere, this works but it doesn't address the
    problem I'm trying to solve. The problem is that it's easier for the
    user of an object (who didn't create it, and probably hasn't read the
    documentation) to omit the try/finally construct. Therefore, this is
    what he will tend to do. I'm looking at what I, as a designer, can do
    to prevent this from happening.

    Regards
    David Turner
     
    David Turner, Jun 9, 2004
    #11
  12. David Turner

    Duncan Booth Guest

    (David Turner) wrote in
    news::

    >> When an exception is handled, you can access the stack traceback.
    >> This contains references to all the local variables which were in
    >> scope at the point when the exception was thrown. Normally these
    >> variables would be destroyed when the functions return, but if there
    >> is a traceback referencing them they stay alive as long as the
    >> traceback is accessible which is usually until the next exception is
    >> thrown.

    >
    > Then why not unreference the traceback (and therefore destroy it and
    > the down-stack locals, if the exception handler hasn't made another
    > reference) at the end of the handling suite?


    Fine, except that isn't what Python does now, and the current behaviour
    needs to be kept for compatibility. So if that is the behaviour you want,
    you have to do that explicitly yourself at the end of every exception
    handler.
     
    Duncan Booth, Jun 9, 2004
    #12
  13. (David Turner) wrote in message news:<>...
    > Hi all
    >
    > I noticed something interesting while testing some RAII concepts
    > ported from C++ in Python.


    AFAIK, cpp-style RAII is mostly unportable to other languages.

    > I haven't managed to find any information
    > about it on the web, hence this post.


    This discussion may be helpful:
    http://mail.python.org/pipermail/python-list/2002-March/090979.html

    >
    > The problem is that when an exception is raised, the destruction of
    > locals appears to be deferred to program exit. Am I missing
    > something? Is this behaviour by design? If so, is there any reason
    > for it? The only rationale I can think of is to speed up exception
    > handling; but as this approach breaks many safe programming idioms, I
    > see it as a poor trade.
    >


    My impression is that many (or all?) languages with GC (especially
    non-refcounting) don't guarantee deterministic destruction of objects.
    I guess it's hard to have both GC and DD :) Please correct me if I am wrong.

    See also "Deterministic Destruction can be a Bug" for an example when DD
    can be a bad thing:
    http://www.hpl.hp.com/personal/Hans_Boehm/gc/det_destr.html

    All that said, GC-enabled language still can provide some support for RAII
    pattern. Consider this C# example:

    using (Font f1 = new Font("Arial", 10), f2 = new Font("Arial", 12)) {
    // use f1 and f2...
    } // compiler will call Dispose on f1 and f2 either on exit or on exception;
    // not sure about order of disposal, but you can nest "using" anyway

    This saves you from trouble of coding try+finally+Dispose.
    BTW, does anybody know whether MS borrowed or invented this construct?

    - kv
     
    Konstantin Veretennicov, Jun 9, 2004
    #13
  14. "Humpty Dumpty" <> wrote in message news:<Ysaxc.19793$>...
    >
    > There is no Python equivalent of C++'s "destructor garanteed to be called
    > upon scope exit", for a couple of reasons: scope exit only destroys
    > references to objects, not the objects themselves; destruction of objects is
    > left to the garbage collector and you have no influence on it. In
    > particular, the gc is not required to release resources, so finalizers (the
    > __del__ method, closest to C++'s destructor) may not get called. This means
    > __del__ is pretty much useless (AFAIMC), and you can't rely on them being
    > called before program exit (or ever, for that matter).
    >
    > I agree, it is a real pitty that Python doesn't have a way of doing what you
    > mention, other than try-finally, which makes code more difficult to read. A
    > new specifier, e.g. "scoped", would be a required addtion to Python:
    > interpreter would garantee that __del__ of scoped objects would be called on
    > scope exit, and raise an exception if attempt to alias.
    >


    Is there a PEP or something for "scoped specifier"?

    - kv
     
    Konstantin Veretennicov, Jun 9, 2004
    #14
  15. (Konstantin Veretennicov) writes:

    > "Humpty Dumpty" <> wrote in message news:<Ysaxc.19793$>...
    >>
    >> I agree, it is a real pitty that Python doesn't have a way of doing what you
    >> mention, other than try-finally, which makes code more difficult to read. A
    >> new specifier, e.g. "scoped", would be a required addtion to Python:
    >> interpreter would garantee that __del__ of scoped objects would be called on
    >> scope exit, and raise an exception if attempt to alias.
    >>

    >
    > Is there a PEP or something for "scoped specifier"?


    http://python.org/peps/pep-0310.html

    Bernhard

    --
    Intevation GmbH http://intevation.de/
    Skencil http://sketch.sourceforge.net/
    Thuban http://thuban.intevation.org/
     
    Bernhard Herzog, Jun 9, 2004
    #15
  16. David Turner

    David Turner Guest

    > >
    > > Then why not unreference the traceback (and therefore destroy it and
    > > the down-stack locals, if the exception handler hasn't made another
    > > reference) at the end of the handling suite?

    >
    > Fine, except that isn't what Python does now, and the current behaviour
    > needs to be kept for compatibility. So if that is the behaviour you want,
    > you have to do that explicitly yourself at the end of every exception
    > handler.


    Are you sure the current behaviour needs to be kept? Isn't
    referencing the traceback outside of an exception handler a little
    dodgy in the first place? I'm sorry if I sound argumentative, but I
    do want to understand the issues thoroughly :).

    Regards
    David Turner
     
    David Turner, Jun 9, 2004
    #16
  17. David Turner

    Duncan Booth Guest

    (David Turner) wrote in
    news::

    >> >
    >> > Then why not unreference the traceback (and therefore destroy it
    >> > and the down-stack locals, if the exception handler hasn't made
    >> > another reference) at the end of the handling suite?

    >>
    >> Fine, except that isn't what Python does now, and the current
    >> behaviour needs to be kept for compatibility. So if that is the
    >> behaviour you want, you have to do that explicitly yourself at the
    >> end of every exception handler.

    >
    > Are you sure the current behaviour needs to be kept? Isn't
    > referencing the traceback outside of an exception handler a little
    > dodgy in the first place? I'm sorry if I sound argumentative, but I
    > do want to understand the issues thoroughly :).
    >

    The documentation says:

    > exc_info( )
    >
    > This function returns a tuple of three values that give information
    > about the exception that is currently being handled. The information
    > returned is specific both to the current thread and to the current
    > stack frame. If the current stack frame is not handling an exception,
    > the information is taken from the calling stack frame, or its caller,
    > and so on until a stack frame is found that is handling an exception.
    > Here, ``handling an exception'' is defined as ``executing or having
    > executed an except clause.'' For any stack frame, only information
    > about the most recently handled exception is accessible.


    So, in fact I was wrong. The exception gets cleared when the function that
    handled it returns:

    >>> import sys
    >>> def f():

    .... try:
    .... raise RuntimeError
    .... except:
    .... print sys.exc_info()
    .... print sys.exc_info()
    ....
    >>> f()

    (<class exceptions.RuntimeError at 0x00864810>, <exceptions.RuntimeError
    instance at 0x008EA940>, <traceback object at 0x008EA968>)
    (<class exceptions.RuntimeError at 0x00864810>, <exceptions.RuntimeError
    instance at 0x008EA940>, <traceback object at 0x008EA968>)
    >>> def g():

    .... sys.exc_clear()
    .... f()
    .... print sys.exc_info()
    ....
    >>> g()

    (<class exceptions.RuntimeError at 0x00864810>, <exceptions.RuntimeError
    instance at 0x008EA990>, <traceback object at 0x008EA940>)
    (<class exceptions.RuntimeError at 0x00864810>, <exceptions.RuntimeError
    instance at 0x008EA990>, <traceback object at 0x008EA940>)
    (None, None, None)
    >>>



    Evidently Python does clear the exception when you leave the exception
    handler, it is just that the exception handler stretches a bit further than
    you might expect. This of course matters more when you put the exception
    handler inside a loop, or do more processing after the handler has caught
    the exception. I'm not sure how much effect it would have to restrict the
    handler to actually inside the except clause.

    In fact further investigation shows that the exception context is saved and
    restored across function calls:

    >>> def h():

    .... try:
    .... raise ValueError, 'h'
    .... except:
    .... print sys.exc_info()
    .... f()
    .... print sys.exc_info()
    ....
    >>> h()

    (<class exceptions.ValueError at 0x00864DB0>, <exceptions.ValueError
    instance at 0x008EAAA8>, <traceback object at 0x008EA940>)
    (<class exceptions.RuntimeError at 0x00864810>, <exceptions.RuntimeError
    instance at 0x008EA760>, <traceback object at 0x008EAAD0>)
    (<class exceptions.RuntimeError at 0x00864810>, <exceptions.RuntimeError
    instance at 0x008EA760>, <traceback object at 0x008EAAD0>)
    (<class exceptions.ValueError at 0x00864DB0>, <exceptions.ValueError
    instance at 0x008EAAA8>, <traceback object at 0x008EA940>)
    >>>


    I never knew it did that.
     
    Duncan Booth, Jun 9, 2004
    #17
  18. David Turner

    David Turner Guest

    (Konstantin Veretennicov) wrote in message news:<>...
    > (David Turner) wrote in message news:<>...
    >
    > using (Font f1 = new Font("Arial", 10), f2 = new Font("Arial", 12)) {
    > // use f1 and f2...
    > } // compiler will call Dispose on f1 and f2 either on exit or on exception;
    > // not sure about order of disposal, but you can nest "using" anyway
    >
    > This saves you from trouble of coding try+finally+Dispose.
    > BTW, does anybody know whether MS borrowed or invented this construct?
    >


    It still leaves the clean-up onus in the wrong place.

    The "using" concept has been kicking around for a while. I think it's
    a natural evolution of Lisp's with-foo-do idiom.

    Regards
    David Turner
     
    David Turner, Jun 10, 2004
    #18
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ahmed Moustafa
    Replies:
    5
    Views:
    30,075
    Chris Smith
    Jul 14, 2004
  2. Magalie
    Replies:
    9
    Views:
    497
    jeffc
    Sep 4, 2003
  3. Paul Miller
    Replies:
    3
    Views:
    1,033
    Alex Martelli
    Nov 12, 2003
  4. Replies:
    17
    Views:
    495
    peter koch
    Nov 17, 2005
  5. Rennie deGraaf

    Exceptions in destructors

    Rennie deGraaf, Apr 30, 2007, in forum: C++
    Replies:
    5
    Views:
    369
    James Kanze
    Apr 30, 2007
Loading...

Share This Page