Messing with the GC

Discussion in 'Python' started by Jens Thoms Toerring, Jan 19, 2013.

  1. Hi,

    triggered by some problems I had with PySide I got a bit
    confused about what the GC may do in certain situations.
    Here's a small test program I cobbled together:

    import sys

    class X( object ) :
    def __init__( self, parent, cnt ) :
    print( "In constructor for {0} {1}".format( self, cnt ),
    file = sys.stderr )
    self.parent = parent
    self.cnt = cnt

    def __del__( self ) :
    print( "In destructor for {0} {1}".format( self, self.cnt ),
    file = sys.stderr )

    def foo( self ) :
    print( "Before", file = sys.stderr )
    self.parent.z = X( self.parent, 2 ) # Is this bad?
    print( "After", file = sys.stderr )

    class Y( object ) :
    def __init__( self ) :
    print( "In constructor for {0}".format( self ),
    file = sys.stderr )
    self.z = X( self, 1 )

    def __del__( self ) :
    print( "In destructor for {0}".format( self ),
    file = sys.stderr )

    Y( ).z.foo( )

    Have a look at the line with the comment. At this point the
    only reference in existence to the X class instance, of which
    a method is just being executed, goes out of scope. Thus I
    would assume that the GC could now kick any time, possibly
    even before the following call of print() or before the method
    call returns. That, in turn might result in a crash of the
    script.

    Is my assumption about this flawed and there are no potential
    dangers? Perhaps with

    Y( ).z.foo( )

    a temporary second reference is created that keeps the GC
    for removing the X instance...

    Another thing I'm puzzled about is the output of the
    script:

    In constructor for <__main__.Y object at 0x2919210>
    In constructor for <__main__.X object at 0x2919310> 1
    Before
    In constructor for <__main__.X object at 0x2919350> 2
    After
    In destructor for <__main__.X object at 0x2919310> 1

    Ok, the destrucor for the first instance of the X class is
    called only after printing out "After", so the GC didn't
    delete the object before. But then there are obviously no
    calls of the destructors of neither the second instance
    of the X class nor of the Y class instance. Shouldn't
    they be invoked before the program ends?

    Thanks and best regards, Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
    Jens Thoms Toerring, Jan 19, 2013
    #1
    1. Advertising

  2. On Sat, 19 Jan 2013 14:47:16 +0000, Jens Thoms Toerring wrote:

    > Ok, the destrucor for the first instance of the X class is called only
    > after printing out "After", so the GC didn't delete the object before.
    > But then there are obviously no calls of the destructors of neither the
    > second instance of the X class nor of the Y class instance. Shouldn't
    > they be invoked before the program ends?


    You should avoid __del__ destructors whenever not absolutely necessary.

    __del__ may not be called for objects that still exist when Python exits.

    If you have a cycle of objects, and *any* of them have a __del__ method,
    it may be impossible for Python to work out a safe order for them to be
    deleted. Consequently they will never be reclaimed by the garbage
    collector.

    http://docs.python.org/2/reference/datamodel.html#special-method-names

    http://docs.python.org/2/library/gc.html#gc.garbage



    --
    Steven
    Steven D'Aprano, Jan 19, 2013
    #2
    1. Advertising

  3. On Sat, 19 Jan 2013 16:24:37 +0000, Steven D'Aprano wrote:

    > On Sat, 19 Jan 2013 14:47:16 +0000, Jens Thoms Toerring wrote:
    >
    >> Ok, the destrucor for the first instance of the X class is called only
    >> after printing out "After", so the GC didn't delete the object before.
    >> But then there are obviously no calls of the destructors of neither the
    >> second instance of the X class nor of the Y class instance. Shouldn't
    >> they be invoked before the program ends?

    >
    > You should avoid __del__ destructors whenever not absolutely necessary.


    And here is another opinion: you should avoid cycles, rather the __del__.

    http://eli.thegreenplace.net/2009/06/12/safely-using-destructors-in-python/





    --
    Steven
    Steven D'Aprano, Jan 19, 2013
    #3
  4. Jens Thoms Toerring

    Terry Reedy Guest

    On 1/19/2013 9:47 AM, Jens Thoms Toerring wrote:

    The code comments mostly answer your questions about what happens or
    does not happen and when. The comments after add more detail.

    > import sys
    >
    > class X( object ) :
    > def __init__( self, parent, cnt ) :
    > print( "In constructor for {0} {1}".format( self, cnt ),
    > file = sys.stderr )
    > self.parent = parent
    > self.cnt = cnt
    >
    > def __del__( self ) :
    > print( "In destructor for {0} {1}".format( self, self.cnt ),
    > file = sys.stderr )
    >
    > def foo( self ) :


    At this point, self is self.parent.z, which is to say, self and
    self.parent.z are 2 references to 1 object. The self and self.parent
    object refer to each other and therefore constitute a reference cycle.

    > print( "Before", file = sys.stderr )
    > self.parent.z = X( self.parent, 2 ) # Is this bad?


    At this point, self.parent.z is another instance of X, breaking the
    existing reference cycle *and* creating a new one. Now self has just the
    one reference 'self' (plus another non-circular, hidden one, see below#).

    > print( "After", file = sys.stderr )


    At this point, self goes out of scope, the CPython reference count goes
    to 0, and the object that was self is deleted. Other implementations
    typically wait longer to delete.

    > class Y( object ) :
    > def __init__( self ) :
    > print( "In constructor for {0}".format( self ),
    > file = sys.stderr )
    > self.z = X( self, 1 )


    At this point, self is self.z.parent, where z is an X. This is a
    circular reference: self and self.z reference each other.

    > def __del__( self ) :
    > print( "In destructor for {0}".format( self ),
    > file = sys.stderr )
    >
    > Y( ).z.foo( )


    Y() creates a reference cycle. z.foo() substitute a different instance
    of X in the cycle but there still is a cycle, so the Y and X objects
    have reference counts of 1. There are no other references to the two
    objects, making them unreachable and 'dead' to the program.

    The cyclic reference garbage collector (see the gc module doc) that is
    meant to delete such orphans does not delete them here because of the
    __del__ methods. Since gc was added, __del__ is semi-obsolete. If an
    object might possibly be put in a reference cycle (which is quite easy),
    any code that might have been put in __del__ should go in an explicitly
    called .close() method.

    > Have a look at the line with the comment. At this point the
    > only reference in existence to the X class instance, of which
    > a method is just being executed, goes out of scope.


    Nope, the remaining reference, 'self', stays in scope until after the
    function exits. That is when X1 is deleted and the deletion message
    printed.

    > Is my assumption about this flawed


    Yes

    > and there are no potential dangers?


    The orphaned two-object cycle constitutes a memory leak.
    If you called Y( ).z.foo( ) a million times,
    you would have a million useless pairs of objects.
    This is why gc was added.

    > Y( ).z.foo( )
    >
    > [perhaps] a temporary second reference is created that keeps the GC
    > for removing the X instance...


    Function calls (normally) bind argument object to parameter names in the
    function's local namespace. That binding is a temporary reference and
    objects will not disappear in the middle of the call.

    # In CPython, at least, there is another internal reference to arguments
    that also disappears when the function returns, allowing the deletion of
    arguments without other references.

    >>> x = 12343252 # random large number to make sure its a new object
    >>> sys.getrefcount(x)

    2
    >>> def f(a): print(sys.getrefcount(a))


    >>> f(x)

    4

    So print(sys.getrefcount(self)) at the top of foo prints 4 (3+1),
    rather than 3 (2+1), as one might expect. The +1 is explained in the doc

    sys.getrefcount(obj):
    Return the reference count of the object. The count returned is
    generally one higher than you might expect, because it includes the
    (temporary) reference as an argument to getrefcount().

    (Why doesn't getrefcount add 2 instead of 1, as f seems to? I don't
    know, but perhaps because it is written in C rather than Python and
    Python code objects are different from C code.)

    --
    Terry Jan Reedy
    Terry Reedy, Jan 19, 2013
    #4
  5. And further thoughts...

    On Sat, 19 Jan 2013 14:47:16 +0000, Jens Thoms Toerring wrote:

    > Hi,
    >
    > triggered by some problems I had with PySide I got a bit
    > confused about what the GC may do in certain situations. Here's a small
    > test program I cobbled together:
    >
    > import sys
    >
    > class X( object ) :
    > def __init__( self, parent, cnt ) :
    > print( "In constructor for {0} {1}".format( self, cnt ),
    > file = sys.stderr )
    > self.parent = parent
    > self.cnt = cnt
    >
    > def __del__( self ) :
    > print( "In destructor for {0} {1}".format( self, self.cnt ),
    > file = sys.stderr )
    >
    > def foo( self ) :
    > print( "Before", file = sys.stderr )
    > self.parent.z = X( self.parent, 2 ) # Is this bad?
    > print( "After", file = sys.stderr )
    >
    > class Y( object ) :
    > def __init__( self ) :
    > print( "In constructor for {0}".format( self ),
    > file = sys.stderr )
    > self.z = X( self, 1 )
    >
    > def __del__( self ) :
    > print( "In destructor for {0}".format( self ),
    > file = sys.stderr )
    >
    > Y( ).z.foo( )
    >
    > Have a look at the line with the comment.



    You mean this line?

    self.parent.z = X( self.parent, 2 ) # Is this bad?


    > At this point the only
    > reference in existence to the X class instance, of which a method is
    > just being executed, goes out of scope.


    I don't understand this, but to the extent that I do understand it, I
    think you are wrong.

    What do you mean, "the X class instance"? If you mean the class X itself,
    no, that survives until both the class itself is deleted and every one of
    it's instances. If you mean "self", no, that doesn't get deleted by that
    line at all.


    > Thus I would assume that the GC
    > could now kick any time, possibly even before the following call of
    > print() or before the method call returns. That, in turn might result in
    > a crash of the script.


    It would be a pretty crappy garbage collector that collected objects
    while they were still being used.



    > Is my assumption about this flawed and there are no potential dangers?
    > Perhaps with
    >
    > Y( ).z.foo( )
    >
    > a temporary second reference is created that keeps the GC for removing
    > the X instance...


    I'm not even sure what X instance you are referring to, or why you think
    it is going out of scope.



    --
    Steven
    Steven D'Aprano, Jan 19, 2013
    #5
  6. Hi,

    thank you for the explanations. I had overlooked the
    cyclic nature of what I had produced here and, of course,
    the GC can't be blamed for not collecting objects that are
    part of a cycle. The other question about the last refe-
    rence to an object vanishing within a method call (which,
    as I now clearly understand, can't happen and wouldn't make
    much sense) was triggered by a segmentation fault I get
    when I do something similar in PySide, so I was getting
    worried if it might be due to a GC issue. Now I know its
    got to be something different;-)

    Thanks and best regards, Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
    Jens Thoms Toerring, Jan 20, 2013
    #6
  7. Jens Thoms Toerring

    Terry Reedy Guest

    On 1/20/2013 3:09 PM, Jens Thoms Toerring wrote:

    > thank you for the explanations. I had overlooked the
    > cyclic nature of what I had produced here and, of course,
    > the GC can't be blamed for not collecting objects that are
    > part of a cycle. The other question about the last refe-
    > rence to an object vanishing within a method call (which,
    > as I now clearly understand, can't happen and wouldn't make
    > much sense) was triggered by a segmentation fault I get
    > when I do something similar in PySide, so I was getting
    > worried if it might be due to a GC issue. Now I know its
    > got to be something different;-)


    Perhaps the hardest part of writing C extensions to CPython directly in
    C (versus something like Cython) is properly balancing increfs and
    decrefs. An incref without a later decref can lead to a memory leak. A
    decref without a preceding incref (so CPython thinks the object can be
    deleted, when it should not be) can lead to segfaults. So I would report
    PySide code leading to segfaults to the PySide people.

    --
    Terry Jan Reedy
    Terry Reedy, Jan 20, 2013
    #7
  8. Terry Reedy <> wrote:
    > On 1/20/2013 3:09 PM, Jens Thoms Toerring wrote:


    > > thank you for the explanations. I had overlooked the
    > > cyclic nature of what I had produced here and, of course,
    > > the GC can't be blamed for not collecting objects that are
    > > part of a cycle. The other question about the last refe-
    > > rence to an object vanishing within a method call (which,
    > > as I now clearly understand, can't happen and wouldn't make
    > > much sense) was triggered by a segmentation fault I get
    > > when I do something similar in PySide, so I was getting
    > > worried if it might be due to a GC issue. Now I know its
    > > got to be something different;-)


    > Perhaps the hardest part of writing C extensions to CPython directly in
    > C (versus something like Cython) is properly balancing increfs and
    > decrefs. An incref without a later decref can lead to a memory leak. A
    > decref without a preceding incref (so CPython thinks the object can be
    > deleted, when it should not be) can lead to segfaults.


    Definitely - I got started with Python having to write glue
    code to get Python to work with a C++ library. And keeping
    track of which side thinks it owns an object can sometimes
    be a bit of a challenge...

    > So I would report PySide code leading to segfaults to the
    > PySide people.


    Now that I'm more sure that it's unlikely to be a Python GC
    related issue (or my not understanding what I'm doing, to be
    precise) this is on my to-do list. But first I have to distill
    things down to a very short example program still exhibiting
    the problem - and experience tells me that this will most li-
    kely result in the realization that it's not a PySide issue
    at all but some misunderstanding on my side;-)

    Best regards, Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
    Jens Thoms Toerring, Jan 21, 2013
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Anders K. Jacobsen [DK]

    How to stop HTML View from messing up HTML code

    Anders K. Jacobsen [DK], Jan 21, 2005, in forum: ASP .Net
    Replies:
    3
    Views:
    675
    Anders K. Jacobsen [DK]
    Jan 22, 2005
  2. Jensen bredal

    IE messing up with font of web pages.

    Jensen bredal, Mar 21, 2005, in forum: ASP .Net
    Replies:
    3
    Views:
    316
    Patrice
    Mar 21, 2005
  3. =?Utf-8?B?cmdyYW5kaWRpZXI=?=

    Is ViewState messing with me?

    =?Utf-8?B?cmdyYW5kaWRpZXI=?=, Jan 9, 2006, in forum: ASP .Net
    Replies:
    2
    Views:
    341
    George Ter-Saakov
    Jan 10, 2006
  4. =?Utf-8?B?RGFiYmxlcg==?=

    validators messing up layout in IE before being triggered?

    =?Utf-8?B?RGFiYmxlcg==?=, Apr 11, 2006, in forum: ASP .Net
    Replies:
    4
    Views:
    631
    Swanand Mokashi
    Apr 12, 2006
  5. Murrgon

    Java VM messing with COM

    Murrgon, Jun 25, 2004, in forum: Java
    Replies:
    4
    Views:
    638
    James D Carroll
    Jul 5, 2004
Loading...

Share This Page