weakrefs and bound methods

[Bruno Desthuilliers]

Mathias Panzenboeck a écrit :

About the lost weakref problem: in Python, methods are just tiny
wrappers around the object, class and function created at lookup time
(yes, on *each* lookup) (and WWAI, they are by the function object
itself, which implements the descriptor protocol).

When I change the class Wrapper to following, the class Foo works:

class Wrapper(object):
def __init__(self,x):
self.func_name = x.func_name
self.x = weakref.ref(x.im_self)

def __call__(self,*args,**kwargs):
x = self.x()
if x is None:
print "lost reference"
else:
return getattr(x,self.func_name)(*args,**kwargs)

Anyway, the whole thing looks quite baroque to me - and I'm known for my
immoderate appetite for brain-melting features....

If I understand correctly, you're doing all this to avoid circular
references while keeping track of a list of methods to call ?

If yes, then you're thru much pain for nothing. Mostly because there are
other much more simpler solutions. The first one is obviously to use
names and getattr. In it's most naïve implementation, this would look like:

class Foo(object):
def __init__(self):
self._methods = set()
self._methods.add('_foo')

def _foo(self):
print "_foo"

def callMethods(self):
for name in self._methods:
meth = getattr(self, name, None)
if callable(meth):
meth()




Now if _methods is supposed to have the same content class-wise (which
is the case in your exemple), no need to have it as an instance method:

class Foo(object):
_methods = set('_foo')

def _foo(self):
print "_foo"

def callMethods(self):
for name in self._methods:
meth = getattr(self, name, None)
if callable(meth):
meth()

We could go further, like using a decorator to mark methods to add to
_methods, and a metaclass to set the whole thing up, but this would
probably be overkill. Unless there are other requirements, I'd
personnaly stop here.

Oh, and yes, while we're at it : Python usually knows how to deal with
circular references....

But that's ugly.
Indeed.

Any better idea?

Yes. KISS.

 

[Mathias Panzenboeck]

Hi.

I have a problem with weak refs and bound methods. The best explanation for the
problem is a short bit of code. I have the three classes Wrapper, Foo and Bar:

import weakref

class Wrapper(object):
def __init__(self,x):
self.x = weakref.ref(x)

def __call__(self,*args,**kwargs):
x = self.x()
if x is None:
print "lost reference"
else:
return x(*args,**kwargs)

class Foo(object):
def __init__(self):
self._methods = set()
self._methods.add(Wrapper(self._foo))

def _foo(self):
print "_foo"

def callMethods(self):
for method in self._methods:
method()

def __del__(self):
print "del Foo"

class Bar(object):
def __init__(self):
self._methods = set()
self._methods.add(self._foo)

def _foo(self):
print "_foo"

def callMethods(self):
for method in self._methods:
method()

def __del__(self):
print "del Bar"



Now look what happens when I do this:

Foo looses the reference to its method but Bar on the other hand has a refloop and
never gets deleted.

Does anyone know what happens here? How can I write such classes without refloop and
without lost reference?

I'm using:
Python 2.5.1 (r251:54863, May 2 2007, 16:56:35)
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2

-panzi

 

[Mathias Panzenboeck]

When I change the class Wrapper to following, the class Foo works:

class Wrapper(object):
def __init__(self,x):
self.func_name = x.func_name
self.x = weakref.ref(x.im_self)

def __call__(self,*args,**kwargs):
x = self.x()
if x is None:
print "lost reference"
else:
return getattr(x,self.func_name)(*args,**kwargs)


But that's ugly. Any better idea?

 

[Marc 'BlackJack' Rintsch]

import weakref

class Wrapper(object):
def __init__(self,x):
self.x = weakref.ref(x)

def __call__(self,*args,**kwargs):
x = self.x()
if x is None:
print "lost reference"
else:
return x(*args,**kwargs)

class Foo(object):
def __init__(self):
self._methods = set()
self._methods.add(Wrapper(self._foo))

def _foo(self):
print "_foo"

def callMethods(self):
for method in self._methods:
method()

def __del__(self):
print "del Foo"

class Bar(object):
def __init__(self):
self._methods = set()
self._methods.add(self._foo)

def _foo(self):
print "_foo"

def callMethods(self):
for method in self._methods:
method()

def __del__(self):
print "del Bar"



Now look what happens when I do this:


Foo looses the reference to its method but Bar on the other hand has a refloop and
never gets deleted.

``del b`` just deletes the name `b`. It does not delete the object.
There's still the name `_` bound to it in the interactive interpreter.
`_` stays bound to the last non-`None` result in the interpreter.

Drop all those `__del__()` methods as they prevent the garbage collector
from collecting "cycles".

Ciao,
Marc 'BlackJack' Rintsch

 

[Michele Simionato]

Drop all those `__del__()` methods as they prevent the garbage collector
from collecting "cycles".

I fully agree and I will add that __del__ methods are always
a bad idea. Also notice that recently Raymond Hetting said in
this list that he wanted to submit a PEP to remove __del__ from
Python 3000 (I don't know if this will ever happen tough).

Michele Simionato

 

[Mathias Panzenboeck]

``del b`` just deletes the name `b`. It does not delete the object.
There's still the name `_` bound to it in the interactive interpreter.
`_` stays bound to the last non-`None` result in the interpreter.

Why is it then that f (instance of Foo) really gets deleted? (__del__ is called)

Drop all those `__del__()` methods as they prevent the garbage collector
from collecting "cycles".

Ciao,
Marc 'BlackJack' Rintsch

I only inserted them so I can see if the objects are really freed. How can I see that
without a __del__ method?


Thanks so far,
panzi

 

[Mathias Panzenboeck]

``del b`` just deletes the name `b`. It does not delete the object.
There's still the name `_` bound to it in the interactive interpreter.
`_` stays bound to the last non-`None` result in the interpreter.

Actually I have the opposite problem. The reference (to the bound method)
gets lost but it shouldn't!


-panzi

 

[Steven D'Aprano]

I fully agree and I will add that __del__ methods are always a bad idea.

Always?

I recently wrote a bit of code where I needed to check that releasing the
first object in a tree-like structure would allow Python to garbage
collect all the other objects in a tree. I thought it would, but I wanted
to be sure ("don't guess, test"), so I wrote a simple class, gave it a
__del__ method that just printed self, inserted them in the tree, and
then deleted the first one.

Worked like a charm.

Without __del__, what should I have done to test that my code was
deleting objects and not leaking memory?

What should I do when my objects need to perform some special processing
when they are freed, if I shouldn't use __del__?

 

[Steven Bethard]

Actually I have the opposite problem. The reference (to the bound method)
gets lost but it shouldn't!

Ahh, so you expected that ``Wrapper(self._foo)`` would not immediately
lose the reference? It will, because every time you write
``self._foo``, a new bound method is created::
... def foo(self):
... pass
... (14931448, 14891008)

Thus, there is only the one reference to the bound method, and by
wrapping it in a weakref, you are allowing it to disappear immediately::
None

What behavior do you want here? That is, when were you hoping that the
bound method would disappear?

STeVe

 

[Michele Simionato]

Always?

I recently wrote a bit of code where I needed to check that releasing the
first object in a tree-like structure would allow Python to garbage
collect all the other objects in a tree. I thought it would, but I wanted
to be sure ("don't guess, test"), so I wrote a simple class, gave it a
__del__ method that just printed self, inserted them in the tree, and
then deleted the first one.

Worked like a charm.

Without __del__, what should I have done to test that my code was
deleting objects and not leaking memory?

What should I do when my objects need to perform some special processing
when they are freed, if I shouldn't use __del__?

The best thing is to use explicit resource management,
for instance with a try .. finally or with the "with" statement
in Python 2.5. The next best thing is to use weak references.
I have some code for various experiments with wearefs I did
some time ago, here it is:

import itertools, weakref, sys, gc

reference_list = [] # cannot be a set, you would lose references
resource_counter = itertools.count(1)

def resource(before_closing_callback=None,
after_closing_callback=None):
private = '_resource_%s' % resource_counter.next()

def get(self):
return getattr(self, private)

def set(self, resource):
setattr(self, private, resource)
def close(ref):
if before_closing_callback:
before_closing_callback(resource)
resource.close()
if after_closing_callback:
after_closing_callback(resource)
reference_list.remove(ref)
reference_list.append(weakref.ref(self, close))

return property(get, set)

class FakeResource(object):
def __init__(self, name):
print 'opening resource %s' % name
self.name = name
def close(self):
print 'closing resource %s' % self.name
def __repr__(self):
return '<FakeResource %r>' % self.name

class Example(object):
def __init__(self):
self.resource1 = FakeResource('r1')
self.resource2 = FakeResource('r2')
def __del__(self):
print '**************'
self.resource1.close()
self.resource2.close()

def warn_before_closing(res):
sys.stdout.write('going to close %s\n' % res)

class Example2(object):

resource1 = resource(warn_before_closing)
resource2 = resource()

def __init__(self):
self.resource1 = FakeResource('r1')
self.resource2 = FakeResource('r2')


gc.set_debug(gc.DEBUG_LEAK)

#e = Example()
e = Example2()
e.e = e
del e
print reference_list

 

[Alex Martelli]

Without __del__, what should I have done to test that my code was
deleting objects and not leaking memory?

See module gc in the Python standard library.

What should I do when my objects need to perform some special processing
when they are freed, if I shouldn't use __del__?

The solid, reliable way is:

from __future__ import with_statement

and use module contextlib from the Python standard library (or handcode
an __exit__ method, but that's rarely needed), generating these special
objects that require special processing only in 'with' statements. This
"resource acquisition is initialization" (RAII) pattern is the RIGHT way
to ensure timely finalization (particularly but not exclusively in
garbage-collected languages, and particularly but not exclusively to
ease portability to different garbage collection strategies -- e.g.,
among CPython and future versions of IronPython and/or Jython that will
support the with statement).

An alternative that will work in pre-2.5 Python (and, I believe but I'm
not sure, in Jython and IronPython _today_) is to rely on the weakref
module of the standard Python library. If your finalizer, in order to
perform "special processing", requires access to some values that depend
on the just-freed object, you'll have to carefully stash those values
"elsewhere", because the finalizer gets called _after_ the object is
freed (this crucial bit of sequencing semantics is what allows weak
references to work while "strong finalizers" [aka destructors] don't
play well with garbage collection when reference-loops are possible).
E.g., weakref.ref instances are hashable, so you can keep a per-class
dict keyed by them to hold the special values that are needed for
special processing at finalization, and use accessors as needed to make
those special values still look like attributes of instances of your
class.

E.g., consider:

import weakref

class ClosingAtDel(object):
_xs = {}
def __init__(self, x):
self._r = weakref.ref(self, self._closeit)
self._xs[self._r] = x
@property
def x(self):
return self._xs[self._r]
@classmethod
def _closeit(cls, theweakref):
cls._xs[theweakref].close()
del cls._xs[theweakref]

This will ensure that .close() is called on the object 'wrapped' in the
instance of ClosingAtDel when the latter instance goes away -- even when
the "going away" is due to a reference loop getting collected by gc. If
ClosingAtDel had a __del__ method, that would interfere with the garbage
collection. For example, consider adding to that class the following
test/example code:

class Zap(object):
def close(self): print 'closed', self

c = ClosingAtDel(Zap())
d = ClosingAtDel(Zap())
print c.x, d.x
# create a reference loop
c.xx = d; d.xx = c
# garbage-collect it anyway
import gc
del c; del d; gc.collect()
print 'done!'

you'll get a pleasant, expected output:

$ python wr.py
<__main__.Zap object at 0x6b430> <__main__.Zap object at 0x6b490>
closed <__main__.Zap object at 0x6b430>
closed <__main__.Zap object at 0x6b490>
done!

Suppose that ClosingAtDel was instead miscoded with a __del__, e.g.:


class ClosingAtDel(object):
def __init__(self, x):
self.x = x
def __del__(self):
self.x.close()


Now, the same test/example code would emit a desolating...:

$ python wr.py
<__main__.Zap object at 0x6b5b0> <__main__.Zap object at 0x6b610>
done!

I.e., the assumed-to-be-crucial calls to .close() have NOT been
performed, because __del__ inhibits collection of reference-looping
garbage. _Ensuring_ you always avoid reference loops (in intricate
real-life cases) is basically unfeasible (that's why we HAVE gc in the
first place -- for non-loopy cases, reference counting suffices;-), so
the best strategy is to avoid coding __del__ methods, just as Marc
recommends.


Alex

 

[Alex Martelli]

I only inserted them so I can see if the objects are really freed. How can
I see that without a __del__ method?

You can use weakref.ref instances with finalizer functions - see the
long post I just made on this thread for a reasonably rich and complex
example.


Alex

 

[Alex Martelli]

Actually I have the opposite problem. The reference (to the bound method)
gets lost but it shouldn't!

weakrefs to bound methods require some subtlety, see
<http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81253> (or what
I believe is the better treatment of this recipe in the printed edition
of the Python Cookbook -- of course, being the latter's editor, I'm
biased;-).


Alex

 

[Mathias Panzenboeck]

weakrefs to bound methods require some subtlety, see
<http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81253> (or what
I believe is the better treatment of this recipe in the printed edition
of the Python Cookbook -- of course, being the latter's editor, I'm
biased;-).


Alex

Thank you, that really helped.


-panzi

 

[Steven D'Aprano]

The solid, reliable way is:

from __future__ import with_statement

and use module contextlib from the Python standard library (or handcode
an __exit__ method, but that's rarely needed), generating these special
objects that require special processing only in 'with' statements. This
"resource acquisition is initialization" (RAII) pattern is the RIGHT way
to ensure timely finalization (particularly but not exclusively in
garbage-collected languages, and particularly but not exclusively to
ease portability to different garbage collection strategies -- e.g.,
among CPython and future versions of IronPython and/or Jython that will
support the with statement).

Hmmm... I'm not sure I understand how a with statement is meant to
replace class.__del__ in practice. It seems to me that the two things
have different uses. with is useful when you create an object, do
something with it, dispose of it in an orderly fashion, and then
continue. How does that help when you create an unknown number of long-
lasting objects where you can't predict how and when they will be
disposed of?

Here's a minimal example. I have a module that returns parrot objects:

# Parrot module
class Parrot(object):
def __del__(self):
print "I'm now pushing up the daisies"


# Callers code:
import Parrot
p = Parrot.Parrot()
alist = [Parrot.Parrot() for i in range(1000)]
# do something with p and alist
print p, alist
# instances are deleted in arbitrary order
del p
del alist[3]
del alist[853]
del alist[701]
del alist



It seems to me that you are saying to re-write the above as something
like this:


# Parrot module
class Parrot(object):
pass

class killed(object): # context manager
def __exit__(self, *exc_info):
print "I'm now pushing up the daisies"


# Callers code:
import Parrot
with killed(Parrot.Parrot()) as p:
# ... er, what do I do with alist???
# do arbitrary stuff with p and alist
print p, alist


Now, I don't think the above is feasible. What am I missing?

An alternative that will work in pre-2.5 Python (and, I believe but I'm
not sure, in Jython and IronPython _today_) is to rely on the weakref
module of the standard Python library.

....


I'm going to need some time to play around with that, but it seems
reasonably straightforward. But the obvious thing I can see is that
there's an awful lot of boilerplate code that needs to be written for
each and every class that needs a "weak finalizer".

That is to say, Python "guarantees" that __del__ will always be called
(unless it isn't... "guarantee void on planet Earth") for your objects,
so the developer doesn't need to do anything except write the finalizer.
To do the right thing with weakrefs means that not only does the
developer have to write the finalizer, but he has to write the code to
ensure the finalizer is called. I'm not sure that's a step forward.

 

[Michele Simionato]

The solid, reliable way is:

from __future__ import with_statement

and use module contextlib from the Python standard library (or handcode
an __exit__ method, but that's rarely needed), generating these special
objects that require special processing only in 'with' statements. This
"resource acquisition is initialization" (RAII) pattern is the RIGHT way
to ensure timely finalization (particularly but not exclusively in
garbage-collected languages, and particularly but not exclusively to
ease portability to different garbage collection strategies -- e.g.,
among CPython and future versions of IronPython and/or Jython that will
support the with statement).

Hmmm... I'm not sure I understand how a with statement is meant to
replace class.__del__ in practice. It seems to me that the two things
have different uses. with is useful when you create an object, do
something with it, dispose of it in an orderly fashion, and then
continue. How does that help when you create an unknown number of long-
lasting objects where you can't predict how and when they will be
disposed of?

Here's a minimal example. I have a module that returns parrot objects:

# Parrot module
class Parrot(object):
def __del__(self):
print "I'm now pushing up the daisies"

# Callers code:
import Parrot
p = Parrot.Parrot()
alist = [Parrot.Parrot() for i in range(1000)]
# do something with p and alist
print p, alist
# instances are deleted in arbitrary order
del p
del alist[3]
del alist[853]
del alist[701]
del alist

It seems to me that you are saying to re-write the above as something
like this:

# Parrot module
class Parrot(object):
pass

class killed(object): # context manager
def __exit__(self, *exc_info):
print "I'm now pushing up the daisies"

# Callers code:
import Parrot
with killed(Parrot.Parrot()) as p:
# ... er, what do I do with alist???
# do arbitrary stuff with p and alist
print p, alist

Now, I don't think the above is feasible. What am I missing?

Rename your __del__ to close and call it explicitely:

p.close()
alist[3].close()
alist[853].close()
alist[701].close()

...

I'm going to need some time to play around with that, but it seems
reasonably straightforward. But the obvious thing I can see is that
there's an awful lot of boilerplate code that needs to be written for
each and every class that needs a "weak finalizer".

That is to say, Python "guarantees" that __del__ will always be called
(unless it isn't... "guarantee void on planet Earth") for your objects,
so the developer doesn't need to do anything except write the finalizer.

Yep, the problem is that __del__ does NOT guarantee anything.
So, it *looks* easy, but it is actually very fragile and in
subtle cases can betray you without telling. I, for one,
I don't like to have bombs in my code waiting to explode
at some unknown moment ...

To do the right thing with weakrefs means that not only does the
developer have to write the finalizer, but he has to write the code to
ensure the finalizer is called. I'm not sure that's a step forward.

Yep, we would need a better support for finalizers in the
standard library. For the moment, there are various recipes
in the Python cookbook. See also this post:
http://groups.google.com/group/comp...t&q=simionato+__del__&rnum=1#1ef0f64bb539cfb0

 

[Steven D'Aprano]

Hmmm... I'm not sure I understand how a with statement is meant to
replace class.__del__ in practice. It seems to me that the two things
have different uses. with is useful when you create an object, do
something with it, dispose of it in an orderly fashion, and then
continue. How does that help when you create an unknown number of
long-lasting objects where you can't predict how and when they will be
disposed of?

[snip]

Now, I don't think the above is feasible. What am I missing?

Rename your __del__ to close and call it explicitely:

p.close()
alist[3].close()
alist[853].close()
alist[701].close()

Well, that utterly sucks. If I wanted to call finalizers explicitly, I'd
be writing in C.

You say that using __del__ leads to hard-to-debug "bombs in my code
waiting to explode at some unknown moment". Maybe so -- but so does
explicitly managing resources by hand in the way you suggest, and quite
frankly, I suspect I'd have a hell of a lot fewer bugs by relying on
__del__ and the garbage collector than I would from managing resources
manually in a big project.

 

[Chris Mellon]

Hmmm... I'm not sure I understand how a with statement is meant to
replace class.__del__ in practice. It seems to me that the two things
have different uses. with is useful when you create an object, do
something with it, dispose of it in an orderly fashion, and then
continue. How does that help when you create an unknown number of
long-lasting objects where you can't predict how and when they will be
disposed of?

[snip]

Now, I don't think the above is feasible. What am I missing?

Rename your __del__ to close and call it explicitely:

p.close()
alist[3].close()
alist[853].close()
alist[701].close()

Well, that utterly sucks. If I wanted to call finalizers explicitly, I'd
be writing in C.

You say that using __del__ leads to hard-to-debug "bombs in my code
waiting to explode at some unknown moment". Maybe so -- but so does
explicitly managing resources by hand in the way you suggest, and quite
frankly, I suspect I'd have a hell of a lot fewer bugs by relying on
__del__ and the garbage collector than I would from managing resources
manually in a big project.

The point that is being made is that "relying on __del__" is stupid,
because __del__ isn't reliable. It's akin to a Java finalizer - there
is no guarantee that it will be called at all. So you can use __del__,
and have to be continually aware of the dangers of circular references
and refcounting, or you use explicit finalizers, or you can use
context managers. If you have arbitrary numbers of long-lived,
potentially self-referential objects that you can't control in any
other way, you're screwed - you're going to *have* to have a manual
destruction somewhere.

 

[Mathias Panzenboeck]

Mathias Panzenboeck a écrit :

About the lost weakref problem: in Python, methods are just tiny
wrappers around the object, class and function created at lookup time
(yes, on *each* lookup) (and WWAI, they are by the function object
itself, which implements the descriptor protocol).



Anyway, the whole thing looks quite baroque to me - and I'm known for my
immoderate appetite for brain-melting features....

If I understand correctly, you're doing all this to avoid circular
references while keeping track of a list of methods to call ?

If yes, then you're thru much pain for nothing. Mostly because there are
other much more simpler solutions. The first one is obviously to use
names and getattr. In it's most naïve implementation, this would look like:

class Foo(object):
def __init__(self):
self._methods = set()
self._methods.add('_foo')

def _foo(self):
print "_foo"

def callMethods(self):
for name in self._methods:
meth = getattr(self, name, None)
if callable(meth):
meth()




Now if _methods is supposed to have the same content class-wise (which
is the case in your exemple), no need to have it as an instance method:

class Foo(object):
_methods = set('_foo')

def _foo(self):
print "_foo"

def callMethods(self):
for name in self._methods:
meth = getattr(self, name, None)
if callable(meth):
meth()

We could go further, like using a decorator to mark methods to add to
_methods, and a metaclass to set the whole thing up, but this would
probably be overkill. Unless there are other requirements, I'd
personnaly stop here.

Oh, and yes, while we're at it : Python usually knows how to deal with
circular references....

Well yes and no. It does with the enabled gc but I like refcounting much more because
of the predictable behaviour. And when there are no refloops, refcounting makes no
problems. However, Alex Martelli already showed a great way to do what I need, see:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81253

I need it for a little UI lib. Well lib is too much. Just a small wrapper around
blenders unusable API for building GUIs for your scripts. It's ugly as hell.
I made very few classes to wrap up that crap. Every input widget in my UI lib has an
action. You can register action listeners like this:

def actionListener(value):
print "new value:",value

widget.addActionListener(actionListener)

When an widget itself want's to listen for it's action and does this in __init__:

self.addActionListener(sefl.__actionListener)

there would be a ref loop. Therefore I do this now:

self.addActionListener(WeakListener(sefl.__actionListener))

And WeakListener I implemented as follows:

class WeakListener(object):
__slots__ = ('__weakref__', '_func','_obj')

def __init__(self,listener):
self._func = listener.im_func
self._obj = weakref.ref(listener.im_self)

def __call__(self,*args,**kwargs):
obj = self._obj()

if obj is None:
print "*** lost reference to %s's self object ***" % \
self._func.func_name
else:
self._func(obj,*args,**kwargs)


Could be better/more generic, but that's all I need.

To implement the action listener by overloading a action method is IMHO a bad idea.
Because that would kill the listener of the base class. Explicitly calling the base
classes action method is ugly and error prone. And when I have to register action
listeners from "outside" the widget, too.

But thanks for your reply.

-panzi