Dynamically adding and removing methods

C

Collin Winter

One caveat, as I recently discovered, to dynamically adding methods is
that it doesn't work for __foo__ methods. For example, you can't make
an object into an iterator by dynamically assigning bound methods to
obj.__iter__ and obj.next. Same thing with __getitem__, __setitem__,
etc; adding them directly to the instance doesn't get you a
subscriptable object. This is because the interpreter looks at the
methods defined by the class, rather than the instance's attributes.

Just a word of caution.

Collin Winter
 
S

Steven D'Aprano

Suppose I create a class with some methods:

py> class C:
.... def spam(self, x):
.... print "spam " * x
.... def ham(self, x):
.... print "ham * %s" % x
....
py> C().spam(3)
spam spam spamham * 3

To dynamically remove the methods, delete them from the class like you
would any other attribute:

py> del C.ham
py> C().ham(3)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: C instance has no attribute 'ham'

That's fairly straight forward -- no major surprises there.

How does one dynamically add methods to an instance or class? You might
think you can do this:

py> def eggs(self, x):
.... print "eggs * %s" % x
....
py> inst = C()
py> inst.eggs = eggs
py> inst.eggs(3)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: eggs() takes exactly 2 arguments (1 given)

Even though we've added a function to the instance, it hasn't got all the
machinery to work as a proper method. A work-around is to explicitly pass
an instance:

py> inst.eggs(inst, 3)
eggs * 3

To create a proper method, we do this:

py> import new
py> inst.eggs = new.instancemethod(eggs, None, C)
py> inst.eggs(3)
eggs * 3

You can now delete the top-level function eggs, and the method bound to
inst will not be affected. Because eggs is bound to the instance, new
instances will not understand it:

py> C().eggs(3)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: C instance has no attribute 'eggs'

Or you could put the method in the class and have all instances recognise
it:

py> C.eggs = new.instancemethod(eggs, None, C)
py> C().eggs(3)
eggs * 3

Instead of passing None as the second argument to instancemethod, you can
pass an instance. If you do that, self will automatically be set to that
instance instead of the one doing the calling:

py> def truffles(self, x):
.... print self, x
....
py> C.truffles = new.instancemethod(truffles, inst, C)
py> inst
<__main__.C instance at 0xf6d2d6cc>
py> inst.truffles(3)
<__main__.C instance at 0xf6d2d6cc> 3

No surprises there. But look what happens when we use a new instance:

py> x = C()
py> x
<__main__.C instance at 0xf6d2d6cc> 3


Hope this is useful to some folks.
 
R

Ron Adam

Steven D'Aprano wrote:

Or you could put the method in the class and have all instances recognise
it:

py> C.eggs = new.instancemethod(eggs, None, C)
py> C().eggs(3)
eggs * 3

Why not just add it to the class directly? You just have to be sure
it's a class and not an instance of a class.

.... print "beacon + %s" % x
....
>>> C.beacon = beacon
>>> dir(A) ['__doc__', '__module__', 'beacon', 'ham', 'spam']
>>> A.beacon(3) beacon + 3
>>> del beacon
>>> dir(A) ['__doc__', '__module__', 'beacon', 'ham', 'spam']
>>> A.beacon(3) beacon + 3
>>> dir(C)
['__doc__', '__module__', 'beacon', 'ham', 'spam']


Cheers,
Ron
 
S

Steven D'Aprano

Steven D'Aprano wrote:



Why not just add it to the class directly? You just have to be sure
it's a class and not an instance of a class.

Because I started off explicitly adding functions to instances directly,
and when I discovered that didn't work properly, I never even tried adding
it to the class until after I discovered that instancemethod() worked.

As far as I can see, Python's treatment of functions when you dynamically
add them to classes and instances is rather confused. See, for example:

py> class Klass:
.... pass
....
py> def eggs(self, x):
.... print "eggs * %s" % x
....
py> inst = Klass() # Create a class instance.
py> inst.eggs = eggs # Dynamically add a function/method.
py> inst.eggs(1)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: eggs() takes exactly 2 arguments (1 given)

From this, I can conclude that when you assign the function to the
instance attribute, it gets modified to take two arguments instead of one.
Test it by explicitly passing an instance:

py> inst.eggs(inst, 1)
eggs * 1

My hypothesis is confirmed.

Can we get the unmodified function back again?

py> neweggs = inst.eggs
py> neweggs(1)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: eggs() takes exactly 2 arguments (1 given)

Nope. That is a gotcha. Storing a function object as an attribute, then
retrieving it, doesn't give you back the original object again.

So while you can do this:

def printgraph(f): # print a graph of a function
parameters = get_params()
draw_graph(f, parameters)

you can't do this:

def printgraph(function): # print a graph of a function
parameters = get_params()
parameters.function = f # WARNING: f is modified here
draw_graph(parameters)



When storing the function object as an instance object, it is
half-converted to a method: even though eggs is modified to expect two
arguments, Python doesn't know enough to automatically pass the instance
object as the first argument like it does when you call a true instance
method.

Furthermore, the type of the attribute isn't changed:

py> type(eggs)
<type 'function'>
py> type(inst.eggs)
<type 'function'>

But if you assign a class attribute to a function, the type changes, and
Python knows to pass the instance object:

py> Klass.eggs = eggs
py> inst2 = Klass()
py> type(inst2.eggs)
<type 'instancemethod'>
py> inst2.eggs(1)
eggs * 1

The different behaviour between adding a function to a class and an
instance is an inconsistency. The class behaviour is useful, the instance
behaviour is broken.

... print "beacon + %s" % x
...

Did you mean bacon? *wink*
['__doc__', '__module__', 'beacon', 'ham', 'spam']

Okay, you aren't showing all your code. What is A?
 
S

Steven Bethard

Steven said:
py> class Klass:
... pass
...
py> def eggs(self, x):
... print "eggs * %s" % x
...
py> inst = Klass() # Create a class instance.
py> inst.eggs = eggs # Dynamically add a function/method.
py> inst.eggs(1)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: eggs() takes exactly 2 arguments (1 given)

From this, I can conclude that when you assign the function to the
instance attribute, it gets modified to take two arguments instead of one.

No. Look at your eggs function. It takes two arguments. So the
function is not modified at all. (Perhaps you expected it to be?)
Can we get the unmodified function back again?

py> neweggs = inst.eggs
py> neweggs(1)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: eggs() takes exactly 2 arguments (1 given)

Nope. That is a gotcha. Storing a function object as an attribute, then
retrieving it, doesn't give you back the original object again.

Again, look at your eggs function. It takes two arguments. So you got
exactly the same object back. Testing this:

py> class Klass:
.... pass
....
py> def eggs(self, x):
.... print "eggs * %s" % x
....
py> inst = Klass()
py> inst.eggs = eggs
py> neweggs = inst.eggs
py> eggs is neweggs
True

So you get back exactly what you previously assigned. Note that it's
actually with *classes*, not *instances* that you don't get back what
you set:

py> Klass.eggs = eggs
py> Klass.eggs
<unbound method Klass.eggs>
py> Klass.eggs is eggs
False
Furthermore, the type of the attribute isn't changed:

py> type(eggs)
<type 'function'>
py> type(inst.eggs)
<type 'function'>

But if you assign a class attribute to a function, the type changes, and
Python knows to pass the instance object:

py> Klass.eggs = eggs
py> inst2 = Klass()
py> type(inst2.eggs)
<type 'instancemethod'>
py> inst2.eggs(1)
eggs * 1

The different behaviour between adding a function to a class and an
instance is an inconsistency. The class behaviour is useful, the instance
behaviour is broken.

With classes, the descriptor machinery is invoked:

py> Klass.eggs
<unbound method Klass.eggs>
py> Klass.eggs.__get__(None, Klass)
<unbound method Klass.eggs>
py> Klass.eggs.__get__(Klass(), Klass)
<bound method Klass.eggs of <__main__.Klass instance at 0x01290BC0>>

Because instances do not invoke the descriptor machinery, you get a
different result:

py> inst.eggs
<function eggs at 0x0126EBB0>

However, you can manually invoke the descriptor machinery if that's what
you really want:

py> inst.eggs.__get__(None, Klass)
<unbound method Klass.eggs>
py> inst.eggs.__get__(inst, Klass)
<bound method Klass.eggs of <__main__.Klass instance at 0x012946E8>>
py> inst.eggs.__get__(inst, Klass)(1)
eggs * 1

Yes, the behavior of functions that are attributes of classes is
different from the behavior of functions that are attributes of
instances. But I'm not sure I'd say that it's broken. It's a direct
result of the fact that classes are the only things that implicitly
invoke the descriptor machinery.

Note that if instances invoked the descriptor machinery, setting a
function as an attribute of an instance would mean you'd always get
bound methods back. So code like the following would break:

py> class C(object):
.... pass
....
py> def f(x):
.... print 'f(%s)' % x
....
py> def g(obj):
.... obj.f('g')
....
py> c = C()
py> c.f = f
py> g(c)
f(g)

If instances invoked the descriptor machinery, "obj.f" would return a
bound method of the "c" instance, where "x" in the "f" function was
bound to the "c" object. Thus the call to "obj.f" would result in:

py> g(c)
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "<interactive input>", line 2, in g
TypeError: f() takes exactly 1 argument (2 given)

Not that I'm claiming I write code like this. ;) But I'd be hesitant
to call it broken.

STeVe
 
D

Dennis Lee Bieber

As far as I can see, Python's treatment of functions when you dynamically
add them to classes and instances is rather confused. See, for example:

py> class Klass:
... pass
...
py> def eggs(self, x):
... print "eggs * %s" % x
...
py> inst = Klass() # Create a class instance.
py> inst.eggs = eggs # Dynamically add a function/method.
py> inst.eggs(1)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: eggs() takes exactly 2 arguments (1 given)

From this, I can conclude that when you assign the function to the
instance attribute, it gets modified to take two arguments instead of one.
Test it by explicitly passing an instance:
No, see below -- you DEFINED the function to take two arguments.
Binding it as a method did NOT create the magic to automatically pass
the instance as the first argument, so instead of "inst.eggs(1)" turning
into "eggs(inst, 1), you are only invoking "eggs(1)" which is an
argument short.
Can we get the unmodified function back again?

py> neweggs = inst.eggs
py> neweggs(1)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: eggs() takes exactly 2 arguments (1 given)

Nope. That is a gotcha. Storing a function object as an attribute, then
retrieving it, doesn't give you back the original object again.

Actually, in my trials, it did give you the original -- you never
tried invoking the original eggs with only one argument. Look at the
id() outputs:
.... print "eggs * %s" % s
.... Traceback (most recent call last):
.... pass
.... Traceback (most recent call last):
 
R

Ron Adam

Steven said:
Because I started off explicitly adding functions to instances directly,
and when I discovered that didn't work properly, I never even tried adding
it to the class until after I discovered that instancemethod() worked.

As far as I can see, Python's treatment of functions when you dynamically
add them to classes and instances is rather confused. See, for example:

py> class Klass:
... pass
...
py> def eggs(self, x):
... print "eggs * %s" % x
...
py> inst = Klass() # Create a class instance.
py> inst.eggs = eggs # Dynamically add a function/method.
py> inst.eggs(1)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: eggs() takes exactly 2 arguments (1 given)

From this, I can conclude that when you assign the function to the
instance attribute, it gets modified to take two arguments instead of one.
Test it by explicitly passing an instance:

py> inst.eggs(inst, 1)
eggs * 1

My hypothesis is confirmed.

You defined it to take two arguements.. (self, x). If it's found
directly in the object instead of indirectly in the objects parent
objects, it calls it just as you defined it.

Did you mean bacon? *wink*

Of course... remembering arbitrary word letter sequences is probably my
worst skill. ;-) That, and I think for some reason the name Francis
Beacon was in my mind at the time.
C.beacon = beacon
dir(A)
['__doc__', '__module__', 'beacon', 'ham', 'spam']


Okay, you aren't showing all your code. What is A?

'A' is an instace of 'C' which has 'ham' and 'spam' methods in it.

Define a funciton and add it directly to class 'C'.
.... print "beacon + %s" % x
....

Show that it shows up in instance 'A' and can be used.
>>> dir(A) ['__doc__', '__module__', 'beacon', 'ham', 'spam']
>>> A.beacon(3)
beacon + 3


Delete the function. And show it's usable as a method from instance 'A'.
>>> del beacon
>>> dir(A) ['__doc__', '__module__', 'beacon', 'ham', 'spam']
>>> A.beacon(3)
beacon + 3

Show it's still bound to class 'C' even thought the function was deleted.
['__doc__', '__module__', 'beacon', 'ham', 'spam']


Cheers,
Ron
 
S

Steven D'Aprano

Of course... remembering arbitrary word letter sequences is probably my
worst skill. ;-) That, and I think for some reason the name Francis
Beacon was in my mind at the time.

I think you mean Francis Bacon *wink*


Thanks to everybody for all the feedback, it was useful, but I was finding
that I couldn't keep it all straight in my head (as seen by the mistaken
conclusions I drew earlier). So I wrote a short script to systematically
test the various combinations of functions and methods. I'm satisfied with
the results. For the sake of completeness, here are my results:



In the script below, ham and spam are defined first as regular functions
outside of a class definition. ham is defined with a placeholder for
"self", spam is not. They are then assigned (in separate tests) to both an
instance and a class.

eggs is defined as a class method, and then assigned to an object outside
a class. Think of it as the control variable.

"Failure" means the call raises an exception, "Success" means it does not.

Here are the results:



####################

Comparing module function with instance method:
=====================================================
Test: func= | ham | spam | eggs |
----------------------+---------+---------+---------+
func(arg) | Failure | Success | Failure |
inst.func(arg) | Failure | Success | Success |
inst.func(self, arg) | Success | Failure | Failure |


Comparing module function with class method:
=====================================================
Test: func= | ham | spam | eggs |
----------------------+---------+---------+---------+
func(arg) | Failure | Success | Failure |
inst.func(arg) | Success | Failure | Success |
inst.func(self, arg) | Failure | Failure | Failure |


Comparing object types:
====================================
Where: | ham | spam | eggs |
--------------+------+------+------+
module level | f | f | m |
instance | f | f | m |
class | m | m | m |

Key: f = function; m = method; ? = other

####################



This clearly shows that assigning a function to a class attribute invokes
whatever machinery Python uses to turn that function into a true method,
but assigning it to an instance does not.


Here is the script for those interested:



####################

"""Testing of the dynamic addition of functions to classes and instances."""

TABLE = """
%s
=====================================================
Test: func= | ham | spam | eggs |
----------------------+---------+---------+---------+
func(arg) | %s | %s | %s |
inst.func(arg) | %s | %s | %s |
inst.func(self, arg) | %s | %s | %s |
"""

TABLE2 = """
%s
====================================
Where: | ham | spam | eggs |
--------------+------+------+------+
module level | %s | %s | %s |
instance | %s | %s | %s |
class | %s | %s | %s |

Key: f = function; m = method; ? = other
"""

# Functions and methods to be tested:

def ham(self, x):
"""Function with placeholder self argument."""
return "ham " * x

def spam(x):
"""Function without placeholder self argument."""
return "spam " * x

class Klass:
def eggs(self, x):
"""Method defined in the class."""
return "%s eggs" % x

eggs = Klass.eggs

# Testing functions:

def get_type(obj):
s = type(obj).__name__
if s == "function":
return "f"
elif s == "instancemethod":
return "m"
else:
return "?"

def single_test(func, args):
"""Calls func(*args) and returns None if it succeeds or an exception if it fails."""
try:
func(*args)
return "Success"
except Exception, obj:
return "Failure"

def multiple_test(instance, label):
"""Runs multiple tests and returns a table of results."""
L = [label]
for f in (ham, spam, eggs, instance.ham, instance.spam, instance.eggs):
L.append(single_test(f, [1]))
for f in (instance.ham, instance.spam, instance.eggs):
L.append(single_test(f, [instance, 1]))
return TABLE % tuple(L)

def type_test(inst1, inst2):
L = ["Comparing object types:"]
for obj in (ham, spam, eggs, inst1.ham, inst1.spam, inst1.eggs, \
inst2.ham, inst2.spam, inst2.eggs):
L.append(get_type(obj))
return TABLE2 % tuple(L)

def main():
inst1 = Klass()
inst1.ham = ham
inst1.spam = spam
print multiple_test(inst1, "Comparing module function with instance method:")
inst2 = Klass()
Klass.ham = ham
Klass.spam = spam
print multiple_test(inst2, "Comparing module function with class method:")
print type_test(inst1, inst2)


if __name__ == "__main__":
main()

####################


I hope this is as interesting and useful for others as it was for me.
 
R

Ron Adam

Steven said:
I think you mean Francis Bacon *wink*

Yes, I mean him, Beacon is his fathers sirname. I'm not sure if Francis
changed it or if his father did. (?)

This clearly shows that assigning a function to a class attribute invokes
whatever machinery Python uses to turn that function into a true method,
but assigning it to an instance does not.

Actually I think I'm getting more confused. At some point the function
is wrapped. Is it when it's assigned, referenced, or called?

Cheers,
Ron
 
T

Terry Reedy

Ron Adam said:
Actually I think I'm getting more confused. At some point the function
is wrapped. Is it when it's assigned, referenced, or called?

When it is referenced via the class.
If you lookup in class.__dict__, the function is still a function.
.... def meth(self): pass
....
<bound method C.meth of <__main__.C object at 0x008E4688>>

I am not sure, without looking, how much of this is language definition and
how much CPython implementation, but I think mostly the latter, as long as
the inheritance tree lookup behavior is as specified.

Terry J. Reedy
 
S

Steven Bethard

Terry said:
Actually I think I'm getting more confused. At some point the function
is wrapped. Is it when it's assigned, referenced, or called?


When it is referenced via the class.
If you lookup in class.__dict__, the function is still a function.

... def meth(self): pass
...
<bound method C.meth of <__main__.C object at 0x008E4688>>

I am not sure, without looking, how much of this is language definition and
how much CPython implementation, but I think mostly the latter

Well, being that the descriptor machinery is defined in the language
reference[1][2], I'd have to say it's entirely the former. The
descriptor machinery says basically that, for classes,
C.meth
should always be doing the equivalent of:
C.__dict__['meth'].__get__(None, C)
and for instances,
c.meth
should always be doing the equivalent of:
type(c).__dict__['meth'].__get__(c, type(c))

[1] http://docs.python.org/ref/descriptors.html
[2] http://docs.python.org/ref/descriptor-invocation.html

STeVe
 
R

Ron Adam

Terry said:
When it is referenced via the class.

Ok, that's what I suspected. Thanks for clarifying this.
If you lookup in class.__dict__, the function is still a function.

Now why did I not think of doing that? :)

... def meth(self): pass
...

<bound method C.meth of <__main__.C object at 0x008E4688>>

Ok, I got it now. Given class 'C' below, i.m(1) does....
.... def m(self, x):
.... return repr(x)
....
3 0 LOAD_GLOBAL 0 (repr)
3 LOAD_FAST 1 (x)
6 CALL_FUNCTION 1
9 RETURN_VALUE 3 0 LOAD_GLOBAL 0 (repr)
3 LOAD_FAST 1 (x)
6 CALL_FUNCTION 1
9 RETURN_VALUE
3 0 LOAD_GLOBAL 0 (repr)
3 LOAD_FAST 1 (x)
6 CALL_FUNCTION 1
9 RETURN_VALUE

Hmm... that didn't tell me anything. (?)
<function m at 0x009D6930>

Time to start digging around in the source code I guess. ;-)
I am not sure, without looking, how much of this is language definition and
how much CPython implementation, but I think mostly the latter, as long as
the inheritance tree lookup behavior is as specified.

Terry J. Reedy

Yes, it hard to tell sometimes where CPython ends and Python begins.

Cheers,
Ron
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,755
Messages
2,569,536
Members
45,007
Latest member
obedient dusk

Latest Threads

Top