Messing with the GC

[Jens Thoms Toerring]

Hi,

triggered by some problems I had with PySide I got a bit
confused about what the GC may do in certain situations.
Here's a small test program I cobbled together:

import sys

class X( object ) :
def __init__( self, parent, cnt ) :
print( "In constructor for {0} {1}".format( self, cnt ),
file = sys.stderr )
self.parent = parent
self.cnt = cnt

def __del__( self ) :
print( "In destructor for {0} {1}".format( self, self.cnt ),
file = sys.stderr )

def foo( self ) :
print( "Before", file = sys.stderr )
self.parent.z = X( self.parent, 2 ) # Is this bad?
print( "After", file = sys.stderr )

class Y( object ) :
def __init__( self ) :
print( "In constructor for {0}".format( self ),
file = sys.stderr )
self.z = X( self, 1 )

def __del__( self ) :
print( "In destructor for {0}".format( self ),
file = sys.stderr )

Y( ).z.foo( )

Have a look at the line with the comment. At this point the
only reference in existence to the X class instance, of which
a method is just being executed, goes out of scope. Thus I
would assume that the GC could now kick any time, possibly
even before the following call of print() or before the method
call returns. That, in turn might result in a crash of the
script.

Is my assumption about this flawed and there are no potential
dangers? Perhaps with

Y( ).z.foo( )

a temporary second reference is created that keeps the GC
for removing the X instance...

Another thing I'm puzzled about is the output of the
script:

In constructor for <__main__.Y object at 0x2919210>
In constructor for <__main__.X object at 0x2919310> 1
Before
In constructor for <__main__.X object at 0x2919350> 2
After
In destructor for <__main__.X object at 0x2919310> 1

Ok, the destrucor for the first instance of the X class is
called only after printing out "After", so the GC didn't
delete the object before. But then there are obviously no
calls of the destructors of neither the second instance
of the X class nor of the Y class instance. Shouldn't
they be invoked before the program ends?

Thanks and best regards, Jens

 

[Steven D'Aprano]

Ok, the destrucor for the first instance of the X class is called only
after printing out "After", so the GC didn't delete the object before.
But then there are obviously no calls of the destructors of neither the
second instance of the X class nor of the Y class instance. Shouldn't
they be invoked before the program ends?

You should avoid __del__ destructors whenever not absolutely necessary.

__del__ may not be called for objects that still exist when Python exits.

If you have a cycle of objects, and *any* of them have a __del__ method,
it may be impossible for Python to work out a safe order for them to be
deleted. Consequently they will never be reclaimed by the garbage
collector.

http://docs.python.org/2/reference/datamodel.html#special-method-names

http://docs.python.org/2/library/gc.html#gc.garbage

 

[Steven D'Aprano]

You should avoid __del__ destructors whenever not absolutely necessary.

And here is another opinion: you should avoid cycles, rather the __del__.

http://eli.thegreenplace.net/2009/06/12/safely-using-destructors-in-python/

 

[Terry Reedy]

On 1/19/2013 9:47 AM, Jens Thoms Toerring wrote:

The code comments mostly answer your questions about what happens or
does not happen and when. The comments after add more detail.

import sys

class X( object ) :
def __init__( self, parent, cnt ) :
print( "In constructor for {0} {1}".format( self, cnt ),
file = sys.stderr )
self.parent = parent
self.cnt = cnt

def __del__( self ) :
print( "In destructor for {0} {1}".format( self, self.cnt ),
file = sys.stderr )

def foo( self ) :

At this point, self is self.parent.z, which is to say, self and
self.parent.z are 2 references to 1 object. The self and self.parent
object refer to each other and therefore constitute a reference cycle.

print( "Before", file = sys.stderr )
self.parent.z = X( self.parent, 2 ) # Is this bad?

At this point, self.parent.z is another instance of X, breaking the
existing reference cycle *and* creating a new one. Now self has just the
one reference 'self' (plus another non-circular, hidden one, see below#).

print( "After", file = sys.stderr )

At this point, self goes out of scope, the CPython reference count goes
to 0, and the object that was self is deleted. Other implementations
typically wait longer to delete.

class Y( object ) :
def __init__( self ) :
print( "In constructor for {0}".format( self ),
file = sys.stderr )
self.z = X( self, 1 )

At this point, self is self.z.parent, where z is an X. This is a
circular reference: self and self.z reference each other.

def __del__( self ) :
print( "In destructor for {0}".format( self ),
file = sys.stderr )

Y( ).z.foo( )

Y() creates a reference cycle. z.foo() substitute a different instance
of X in the cycle but there still is a cycle, so the Y and X objects
have reference counts of 1. There are no other references to the two
objects, making them unreachable and 'dead' to the program.

The cyclic reference garbage collector (see the gc module doc) that is
meant to delete such orphans does not delete them here because of the
__del__ methods. Since gc was added, __del__ is semi-obsolete. If an
object might possibly be put in a reference cycle (which is quite easy),
any code that might have been put in __del__ should go in an explicitly
called .close() method.

Have a look at the line with the comment. At this point the
only reference in existence to the X class instance, of which
a method is just being executed, goes out of scope.

Nope, the remaining reference, 'self', stays in scope until after the
function exits. That is when X1 is deleted and the deletion message
printed.

Is my assumption about this flawed
Yes

and there are no potential dangers?

The orphaned two-object cycle constitutes a memory leak.
If you called Y( ).z.foo( ) a million times,
you would have a million useless pairs of objects.
This is why gc was added.

Y( ).z.foo( )

[perhaps] a temporary second reference is created that keeps the GC
for removing the X instance...

Function calls (normally) bind argument object to parameter names in the
function's local namespace. That binding is a temporary reference and
objects will not disappear in the middle of the call.

# In CPython, at least, there is another internal reference to arguments
that also disappears when the function returns, allowing the deletion of
arguments without other references.
4

So print(sys.getrefcount(self)) at the top of foo prints 4 (3+1),
rather than 3 (2+1), as one might expect. The +1 is explained in the doc

sys.getrefcount(obj):
Return the reference count of the object. The count returned is
generally one higher than you might expect, because it includes the
(temporary) reference as an argument to getrefcount().

(Why doesn't getrefcount add 2 instead of 1, as f seems to? I don't
know, but perhaps because it is written in C rather than Python and
Python code objects are different from C code.)

 

[Steven D'Aprano]

And further thoughts...

Hi,

triggered by some problems I had with PySide I got a bit
confused about what the GC may do in certain situations. Here's a small
test program I cobbled together:

import sys

class X( object ) :
def __init__( self, parent, cnt ) :
print( "In constructor for {0} {1}".format( self, cnt ),
file = sys.stderr )
self.parent = parent
self.cnt = cnt

def __del__( self ) :
print( "In destructor for {0} {1}".format( self, self.cnt ),
file = sys.stderr )

def foo( self ) :
print( "Before", file = sys.stderr )
self.parent.z = X( self.parent, 2 ) # Is this bad?
print( "After", file = sys.stderr )

class Y( object ) :
def __init__( self ) :
print( "In constructor for {0}".format( self ),
file = sys.stderr )
self.z = X( self, 1 )

def __del__( self ) :
print( "In destructor for {0}".format( self ),
file = sys.stderr )

Y( ).z.foo( )

Have a look at the line with the comment.

You mean this line?

self.parent.z = X( self.parent, 2 ) # Is this bad?

At this point the only
reference in existence to the X class instance, of which a method is
just being executed, goes out of scope.

I don't understand this, but to the extent that I do understand it, I
think you are wrong.

What do you mean, "the X class instance"? If you mean the class X itself,
no, that survives until both the class itself is deleted and every one of
it's instances. If you mean "self", no, that doesn't get deleted by that
line at all.

Thus I would assume that the GC
could now kick any time, possibly even before the following call of
print() or before the method call returns. That, in turn might result in
a crash of the script.

It would be a pretty crappy garbage collector that collected objects
while they were still being used.

Is my assumption about this flawed and there are no potential dangers?
Perhaps with

Y( ).z.foo( )

a temporary second reference is created that keeps the GC for removing
the X instance...

I'm not even sure what X instance you are referring to, or why you think
it is going out of scope.

 

[Jens Thoms Toerring]

Hi,

thank you for the explanations. I had overlooked the
cyclic nature of what I had produced here and, of course,
the GC can't be blamed for not collecting objects that are
part of a cycle. The other question about the last refe-
rence to an object vanishing within a method call (which,
as I now clearly understand, can't happen and wouldn't make
much sense) was triggered by a segmentation fault I get
when I do something similar in PySide, so I was getting
worried if it might be due to a GC issue. Now I know its
got to be something different;-)

Thanks and best regards, Jens

 

[Terry Reedy]

thank you for the explanations. I had overlooked the
cyclic nature of what I had produced here and, of course,
the GC can't be blamed for not collecting objects that are
part of a cycle. The other question about the last refe-
rence to an object vanishing within a method call (which,
as I now clearly understand, can't happen and wouldn't make
much sense) was triggered by a segmentation fault I get
when I do something similar in PySide, so I was getting
worried if it might be due to a GC issue. Now I know its
got to be something different;-)

Perhaps the hardest part of writing C extensions to CPython directly in
C (versus something like Cython) is properly balancing increfs and
decrefs. An incref without a later decref can lead to a memory leak. A
decref without a preceding incref (so CPython thinks the object can be
deleted, when it should not be) can lead to segfaults. So I would report
PySide code leading to segfaults to the PySide people.

 

[Jens Thoms Toerring]

On 1/20/2013 3:09 PM, Jens Thoms Toerring wrote:

Perhaps the hardest part of writing C extensions to CPython directly in
C (versus something like Cython) is properly balancing increfs and
decrefs. An incref without a later decref can lead to a memory leak. A
decref without a preceding incref (so CPython thinks the object can be
deleted, when it should not be) can lead to segfaults.

Definitely - I got started with Python having to write glue
code to get Python to work with a C++ library. And keeping
track of which side thinks it owns an object can sometimes
be a bit of a challenge...

So I would report PySide code leading to segfaults to the
PySide people.

Now that I'm more sure that it's unlikely to be a Python GC
related issue (or my not understanding what I'm doing, to be
precise) this is on my to-do list. But first I have to distill
things down to a very short example program still exhibiting
the problem - and experience tells me that this will most li-
kely result in the realization that it's not a PySide issue
at all but some misunderstanding on my side;-)

Best regards, Jens