weak reference to bound method

[Ole Streicher]

Hi group,

I am trying to use a weak reference to a bound method:

class MyClass(object):
def myfunc(self):
pass

o = MyClass()
print o.myfunc
import weakref
r = weakref.ref(o.myfunc)
print r()
This is what I do not understand. The object "o" is still alive, and
therefore the bound method "o.myfunc" shall exists.

Why does the weak reference claim that it is removed? And how can I hold
the reference to the method until the object is removed?

Is this a bug or a feature? (Python 2.6)

Best regards

Ole

 

[Thomas Lehmann]

I am trying to use a weak reference to a bound method:

class MyClass(object):
def myfunc(self):
pass

o = MyClass()
print o.myfunc

import weakref
r = weakref.ref(o.myfunc)
print r()

This is what I do not understand. The object "o" is still alive, and
therefore the bound method "o.myfunc" shall exists.

Why does the weak reference claim that it is removed? And how can I hold
the reference to the method until the object is removed?

k = o.myfunc
r = weakref.ref(k)
print r()
Don't ask me why! I have just been interested for what you are
trying...

 

[Peter Otten]

I am trying to use a weak reference to a bound method:

class MyClass(object):
def myfunc(self):
pass

o = MyClass()
print o.myfunc

import weakref
r = weakref.ref(o.myfunc)
print r()

This is what I do not understand. The object "o" is still alive, and
therefore the bound method "o.myfunc" shall exists.

No. o.myfunc is a different object, a bound method, and every time you
access o's myfunc attribute a new bound method is created:
.... def myfunc(self):
.... pass
....False

The bound method holds references to the instance and the function, not the
other way around:

Why does the weak reference claim that it is removed?

Because there are indeed no more strong references to the bound method.

And how can I hold the reference to the method until the object is

removed?

If you kept a strong reference to the bound method it would in turn keep the
MyClass instance alive. Maybe you could store the bound method in the
instance and rely on cyclic garbage collection.

Is there an actual use case?

Peter

 

[Ole Streicher]

Hi Thomas,

k = o.myfunc
r = weakref.ref(k)
print r()

Don't ask me why! I have just been interested for what you are trying...

This is clear: in your case, o.myfunc is explicitely referenced by k,
this avoids the garbage collection.

My problem is that I have a class that delegates a function call, like:

--------------------8<------------------
import weakref

class WeakDelegator(object):
def __init__(self, func):
self._func = weakref.ref(func)

def __call__(self):
func = self._func()
return func() if func else None
--------------------8<------------------

This does not work for bound methods because the weak reference to a
bound method will always point to None, even if the object still exists.

Why is that the case and how can I implement such a class properly?

Best regards

Ole

 

[Ole Streicher]

Hello Peter,

Is there an actual use case?

I discussed this in the german newsgroup. Here is the use in my class:
-----------------------------8<-----------------------
import threading
import weakref

class DoAsync(threading.Thread):
def __init__(self, func):
threading.Thread.__init__(self)
self.setDaemon(True)
self._cond = threading.Condition()
self.scheduled = False
self._func = weakref.ref(func, self._cleanup)
self.start()

def run(self):
while self._func():
with self._cond:
while not self.scheduled and self._func():
self._cond.wait()
self.scheduled = False
func = self._func()
if func:
func()

def __call__(self):
with self._cond:
self.scheduled = True
self._cond.notify()

def _cleanup(self, ref):
self()
-----------------------------8<-----------------------

The use for this callable class is to take a function call, and whenever
the DoAsync object is called, trigger a call to the stored function.

Other classes use it like:

-----------------------------8<-----------------------
class MyClass:
def __init__(self):
...
self.update = DoAsync(self._do_update)

def _do_update(self):
do_something_that_takes_long_and_shall_be_done_after_an_update()
-----------------------------8<-----------------------

Since DoAsync starts its own thread, I get a classical deadlock
situation: DoAsync needs a reference to the method to be called, and as
long as the thread is running, the MyClass object (which contains the
method) cannot be cleaned up. This would be a classic case for a weak
reference, if Python would not create it at calling time.

No. o.myfunc is a different object, a bound method, and every time you
access o's myfunc attribute a new bound method is created:

What is the reason for that behaviour? It looks quite silly to me.

And how can I get a reference to a bound method that lives as long as
the method itself?

Regards

Ole

 

[Peter Otten]

Hi Thomas,




This is clear: in your case, o.myfunc is explicitely referenced by k,
this avoids the garbage collection.

My problem is that I have a class that delegates a function call, like:

--------------------8<------------------
import weakref

class WeakDelegator(object):
def __init__(self, func):
self._func = weakref.ref(func)

def __call__(self):
func = self._func()
return func() if func else None
--------------------8<------------------

This does not work for bound methods because the weak reference to a
bound method will always point to None, even if the object still exists.

Why is that the case and how can I implement such a class properly?

Maybe:

from weakref import ref

class A(object):
def f(self): return "f"

class Method(object):
def __init__(self, obj, func=None):
if func is None:
func = obj.im_func
obj = obj.im_self
self._im_self = ref(obj)
self._im_func = ref(func)
def __call__(self):
obj = self._im_self()
func = self._im_func()
if obj is not None and func is not None:
return func.__get__(obj)()

a = A()
m = Method(a.f)
print m()
del a
print m()

It's still not clear to me why you would want to do that...

Peter

 

[Miles Kaufmann]

I am trying to use a weak reference to a bound method:

class MyClass(object):
def myfunc(self):
pass

o = MyClass()
print o.myfunc

import weakref
r = weakref.ref(o.myfunc)
print r()

This is what I do not understand. The object "o" is still alive, and
therefore the bound method "o.myfunc" shall exists.

Like Peter said, bound methods are created on demand when they are
obtained from the instance, not when the instance is created.

Why does the weak reference claim that it is removed? And how can I
hold
the reference to the method until the object is removed?

You could combine unbound methods with a weakref to the object:

r = weakref.ref(o)
MyClass.myfunc(r())

You could also create a wrapper object that holds a weak reference to
the instance and creates a bound method on demand:

class WeakMethod(object):
def __init__(self, bound_method):
self.im_func = bound_method.im_func
self.im_self = weakref.ref(bound_method.im_self)
self.im_class = bound_method.im_class

def __call__(self):
obj = self.im_self()
if obj is None: return None
return types.MethodType(self.im_func, obj, self.im_class)
# could alternately act like a callableproxy

-Miles

 

[Ole Streicher]

Hi Miles,

You could also create a wrapper object that holds a weak reference to the
instance and creates a bound method on demand:
class WeakMethod(object):
def __init__(self, bound_method):
self.im_func = bound_method.im_func
self.im_self = weakref.ref(bound_method.im_self)
self.im_class = bound_method.im_class

In this case, I can use it only for bound methods, so I would need to
handle the case of unbound methods separately.

Is there a way to find out whether a function is bound? Or do I have to
use hasattr(im_func) and hasattr(im_self) and hasattr(im_class)?

Best regards

Ole

 

[Ole Streicher]

Hi Peter,

class Method(object):
def __init__(self, obj, func=None):
if func is None:
func = obj.im_func
obj = obj.im_self

This requires that func is a bound method. What I want is to have a
universal class that "always" works: with unbound functions, with
bound function, with lambda expressions, with locally defined functions,
....

For a user of my class, there is no visible reason, why some of them
shall work, while others dont.

Viele Grüße

Ole

 

[Peter Otten]

This requires that func is a bound method. What I want is to have a
universal class that "always" works: with unbound functions, with
bound function, with lambda expressions, with locally defined functions,

That's left as an exercise to the reader

 

[Ole Streicher]

Hello Peter,

That's left as an exercise to the reader

Do you have the feeling that there exists any reader that is able to
solve this exercise?

I am a bit surprised that already such a simple problem is virtually
unsolvable in python. Do you think that my concept of having a DoAsync
class is wrong?

Best regards

Ole

 

[Peter Otten]

Hello Peter,




Do you have the feeling that there exists any reader that is able to
solve this exercise?

I was thinking of you.

I am a bit surprised that already such a simple problem is virtually
unsolvable in python. Do you think that my concept of having a DoAsync
class is wrong?

I don't understand the example you give in the other post.

If you are trying to use reference counting as a means of inter-thread
communication, then yes, I think that's a bad idea.

 

[Ole Streicher]

Hi Peter,

I was thinking of you.

I could imagine that. However, I am just a beginner in Python and I dont
know which types "callables" there exist in python and which "smart"
ideas (like re-creating them at every call) additionally occur when I
implement such a beast. For example, for locally defined functions, I
have still no idea at all on how to keep them away from the garbage
collector.

I don't understand the example you give in the other post.

Hmm. I am programming a GUI client application. The client will receive
some input data (via network, and via user input) and shall be updated
after these data.

Unfortunately, the input data (and ofcourse the user input) do not come
regularly; there may times when the data come too fast to process all
of them.

Imagine, for example, that I want to provide a 2d-gaussian fit to some
region of an image and display the result in a separate window, and
updated this whenever the mouse is moved.

The fit takes (let's say) some seconds, so I cannot just call the fit
routine within the mouse move event (this would block other gui
operations, and f.e. the display of the mouse coordinates). So I need
just to trigger the fit routine on mouse movement, and to check
afterwards whether the mouse position is still current.

This is the reason for the DoAsync class: when it is called, it shall
trigger the function that was given in the constructor, t.m.

class MyClass:
def __init__(self):
self.update_fit = DoAsync(update_the_fit)

def mouse_move(self, event):
self.set_coordinates(event.x, event.y)
self.update_fit() # trigger the update_the_fit() call
...

Thus, the mouse_move() method is fast, even if the update_the_fit()
method takes some time to process.

I want to implement it now that DoAsync will be automatically garbage
collected whenever the MyClass object is deleted. Since DoAsync starts
its own thread (which only shall finish when the MyClass object is
deleted), a reference to MyClass (or one of its functions) will keep the
MyClass object from garbage collection.

If you are trying to use reference counting as a means of inter-thread
communication, then yes, I think that's a bad idea.

No; my problem is:

- a thread started in DoAsync will keep the DoAsync object from
garbage collection
- a reference to a MyClass realted object (the bound method) in DoAsync
will thus also prevent the MyClass object from garbage collection
- Even if I dont use the MyClass object anymore, and nobody else uses
the DoAsync object, both stand in memory forever, and the thread also
never finishes.

Did you get the problem?

Best regards

Ole

 

[Peter Otten]

Hi Peter,



I could imagine that. However, I am just a beginner in Python and I dont
know which types "callables" there exist in python and which "smart"
ideas (like re-creating them at every call) additionally occur when I
implement such a beast. For example, for locally defined functions, I
have still no idea at all on how to keep them away from the garbage
collector.

Btw, have you implemented such a design in another language?

Hmm. I am programming a GUI client application. The client will receive
some input data (via network, and via user input) and shall be updated
after these data.

Unfortunately, the input data (and ofcourse the user input) do not come
regularly; there may times when the data come too fast to process all
of them.

Imagine, for example, that I want to provide a 2d-gaussian fit to some
region of an image and display the result in a separate window, and
updated this whenever the mouse is moved.

The fit takes (let's say) some seconds, so I cannot just call the fit
routine within the mouse move event (this would block other gui
operations, and f.e. the display of the mouse coordinates). So I need
just to trigger the fit routine on mouse movement, and to check
afterwards whether the mouse position is still current.

This is the reason for the DoAsync class: when it is called, it shall
trigger the function that was given in the constructor, t.m.

class MyClass:
def __init__(self):
self.update_fit = DoAsync(update_the_fit)

def mouse_move(self, event):
self.set_coordinates(event.x, event.y)
self.update_fit() # trigger the update_the_fit() call
...

Thus, the mouse_move() method is fast, even if the update_the_fit()
method takes some time to process.

I want to implement it now that DoAsync will be automatically garbage
collected whenever the MyClass object is deleted. Since DoAsync starts
its own thread (which only shall finish when the MyClass object is
deleted), a reference to MyClass (or one of its functions) will keep the
MyClass object from garbage collection.


No; my problem is:

- a thread started in DoAsync will keep the DoAsync object from
garbage collection
- a reference to a MyClass realted object (the bound method) in DoAsync
will thus also prevent the MyClass object from garbage collection
- Even if I dont use the MyClass object anymore, and nobody else uses
the DoAsync object, both stand in memory forever, and the thread also
never finishes.

I think I'd go for a simpler approach, manage the lifetime of MyClass
instances manually and add a MyClass.close() method that sets a flag which
in turn is periodically read by DoAsync() and will eventually make it stop.

Peter

 

[Ole Streicher]

Hi Peter,

Btw, have you implemented such a design in another language?

No.

I think I'd go for a simpler approach, manage the lifetime of MyClass
instances manually and add a MyClass.close() method that sets a flag which
in turn is periodically read by DoAsync() and will eventually make it stop.

Has the disadvantage that I rely on the user. We already have a garbage
collector, so why not use it? It is exactly made for what I want: delete
unused objects.

The solution I have now is still untested, but maybe the separation of
the thread will work:
------------------------------<8----------------------------
import threading
import weakref

class AsyncThread(threading.Thread):
def __init__(self, func):
threading.Thread.__init__(self)
self.setDaemon(True)
self._cond = threading.Condition()
self._func = weakref.ref(func, self.run)
self.start()

def run(self):
while self._func():
with self._cond:
while not self.scheduled:
self._cond.wait()
self.scheduled = False
self._func()
func = self._func()
if func:
func()

def __call__(self):
with self._cond:
self.scheduled = True
self._cond.notify()

class DoAsync(object):
def __init__(self, func):
self._func = func
self._thread = AsyncThread(self._func)

def __call__(self):
self._thread()
------------------------------<8----------------------------

The AsyncThread has now no reference that could prevent the referenced
from the GC. And the function is always stored in the DoAsync object and
will be only collected if this is removed (what is the case if the
parent object is being deleted)

Regards

Ole

 

[ryles]

Hi group,

I am trying to use a weak reference to a bound method:

class MyClass(object):
    def myfunc(self):
        pass

o = MyClass()
print o.myfunc


import weakref
r = weakref.ref(o.myfunc)
print r()


This is what I do not understand. The object "o" is still alive, and
therefore the bound method "o.myfunc" shall exists.

Why does the weak reference claim that it is removed? And how can I hold
the reference to the method until the object is removed?

Is this a bug or a feature? (Python 2.6)

Best regards

Ole

Have a look at: http://mindtrove.info/articles/python-weak-references