is decorator the right thing to use?

D

Dmitry S. Makovey

No, really, Diez has posted the canonical Proxy form in Python, using
__getattr__ on the proxy, and then redirecting to the contained
delegate object. This code does *not* assume that only one property
('b'? where did that come from?) is being redirected - __getattr__
will intercept all attribute lookups and redirect them to the
delegate.

If you need to get fancier and support this single-proxy-to-multiple-
delegates form, then yes, you will need some kind of map that says
which method should delegate to which object. Or, if it is just a
matter of precedence (try A, then try B, then...), then use hasattr to
see if the first delegate has the given attribute, and if not, move on
to the next.

that is what I didn't like about it - I have to iterate over delegates when
I can build direct mapping once and for all and tie it to class
definition ;)
Your original question was "is decorator the right thing to use?" For
this application, the answer is "no".

yeah. seems that way. in the other fork of this thread you'll find my
conclusion which agrees with that :)
It sounds like you are trying
to force this particular to solution to your problem, but you are
probably better off giving __getattr__ intercepting another look.

__getattr__ implies constant lookups and checks (for filtering purposes) - I
want to do them once, attach generated methods as native methods and be
done with it. That is why I do not like __getattr__ in this particular
case. Otherwise - you're right.
 
A

Aaron \Castironpi\ Brady

that is what I didn't like about it - I have to iterate over delegates when
I can build direct mapping once and for all and tie it to class
definition ;)

You're right, there is a difference in performance. But it would
still be simpler to have your 'getattr' method 'learn' what functions
come from what objects, and then just route directly to them.
__getattr__ implies constant lookups and checks (for filtering purposes) - I
want to do them once, attach generated methods as native methods and be
done with it. That is why I do not like __getattr__ in this particular
case. Otherwise - you're right.

Try this (untested!):

def __getattr__( self, key ):
if key in self.knownmeths:
return self.knownmeths[ key ]
for x in self.delegs:
if hasattr( x, key ):
_= self.knownmethds[ key ]= getattr( x, key )
return _
raise name not found

self.knownmethds has desired order of precedence.
 
B

Bruno Desthuilliers

Dmitry S. Makovey a écrit :
that is what I didn't like about it - I have to iterate over delegates when
I can build direct mapping once and for all and tie it to class
definition ;)

Hem... I'm afraid you don't really take Python's dynamic nature into
account here. Do you know that even the __class__ attribute of an
instance can be rebound at runtime ? What about 'once and for all' then ?

But anyway:
yeah. seems that way. in the other fork of this thread you'll find my
conclusion which agrees with that :)


__getattr__ implies constant lookups and checks (for filtering purposes)

Unless you cache the lookups results...
- I
want to do them once, attach generated methods as native methods

What is a "native method" ? You might not be aware of the fact that
method objects are usually built anew from functions on each method call...
and be
done with it. That is why I do not like __getattr__ in this particular
case.

There's indeed an additional penalty using __getattr__, which is that
it's only called as a last resort. Now remember that premature
optimization is the root of evil... Depending on effective use (ie : how
often a same 'proxied' method is called on a given Proxy instance, on
average), using __getattr__ to retrieve the appropriate bound method on
the delegate then adding it to the proxy instance *as an instance
attribute* might be a way better (and simpler) optimization.
 
D

Dmitry S. Makovey

Bruno said:
Hem... I'm afraid you don't really take Python's dynamic nature into
account here. Do you know that even the __class__ attribute of an
instance can be rebound at runtime ? What about 'once and for all' then ?

must've been wrong wording on my part. Dynamic nature is exactly what I
wanted to use :) except that I do not expect clients to take advantage of
it while using my classes ;)
Unless you cache the lookups results...
sure


What is a "native method" ? You might not be aware of the fact that
method objects are usually built anew from functions on each method
call...

again, wrong wording on my part. by native I meant: make use as much as
possible of existing machinery and override default behavior only when it's
absolutely necessary. (hopefully my wording is not off this time ;) )
There's indeed an additional penalty using __getattr__, which is that
it's only called as a last resort. Now remember that premature
optimization is the root of evil... Depending on effective use (ie : how
often a same 'proxied' method is called on a given Proxy instance, on
average), using __getattr__ to retrieve the appropriate bound method on
the delegate then adding it to the proxy instance *as an instance
attribute* might be a way better (and simpler) optimization.

I actually ended up rewriting things (loosely based on George's suggested
code) with descriptors and not using metaclasses or decorators (so much for
my desire to use them).

With following implementation (unpolished at this stage but already
functional) I can have several instances of B objects inside of A object
and proxy certain methods to one or another object (I might be having a
case when I have A.b1 and A.b2 and passing some methods to b1 and others to
b2 having both of the same class B, maybe even multiplexing). This one
seems to be fairly light as well without need to scan instances (well,
except for one getattr, but I couldn't get around it). Maybe I didn't
account for some shoot-in-the-foot scenarios but I can't come up with any.
Last time I played with __getattr__ I shot myself in the foot quite well
BTW :)

class ProxyMethod(object):

def __init__(self,ob_name,meth):
self.ob_name=ob_name
self.meth=meth

def my_call(self,instance,*argv,**kw):
ob=getattr(instance,self.ob_name)
cls=self.meth.im_class
return self.meth.__get__(ob,cls)(*argv,**kw)

def __get__(self,instance,owner):
if not instance:
return self.my_call
ob=getattr(instance,self.ob_name)
cls=self.meth.im_class
return self.meth.__get__(ob,cls)

class B:
def __init__(self):
self.val='bval'

def bmethod(self,a):
print "B::bmethod",
print a, self.val

class A:
b=None

def __init__(self,b=None):
self.val='aval'
self.b=b
b.val='aval-b'

def mymethod(self,a):
print "A::mymethod, ",a

bmethod=ProxyMethod('b',B.bmethod)

b=B()
b.bmethod('foo')
a=A(b)
b=B()
b.val='newval'
a.mymethod('baz')
a.bmethod('bar')
A.bmethod(a,'zoom')
 
A

Aaron \Castironpi\ Brady

Dmitry S. Makovey a écrit :





Hem... I'm afraid you don't really take Python's dynamic nature into
account here. Do you know that even the __class__ attribute of an
instance can be rebound at runtime ? What about 'once and for all' then ?

But anyway:




Unless you cache the lookups results...


What is a "native method" ? You might not be aware of the fact that
method objects are usually built anew from functions on each method call....


There's indeed an additional penalty using __getattr__, which is that
it's only called as a last resort. Now remember that premature
optimization is the root of evil... Depending on effective use (ie : how
often a same 'proxied' method is called on a given Proxy instance, on
average), using __getattr__ to retrieve the appropriate bound method on
the delegate then adding it to the proxy instance *as an instance
attribute* might be a way better (and simpler) optimization.

That prohibits using a descriptor in the proxied classes, or at least
the proxied functions, since you break descriptor protocol and only
call __get__ once. Better to cache and get by name. It's only slower
by the normal amount, and technically saves space, strings vs.
instancemethod objects (except for really, really long function names).
 
D

Dmitry S. Makovey

Aaron said:
That prohibits using a descriptor in the proxied classes, or at least
the proxied functions, since you break descriptor protocol and only
call __get__ once. Better to cache and get by name. It's only slower
by the normal amount, and technically saves space, strings vs.
instancemethod objects (except for really, really long function names).

that is an interesting point since I didn't think about having descriptors
in proxied classes. my reworked code clearly breaks when descriptors are
thrown at it. It will break with methods in proxied objects that are
implemented as objects too. Now I adjusted constructor a bit to account for
that (I just can't figure out case when I'll be proxying descriptors unless
they return function but than I don't see benefit in using descriptor for
that, probably because I haven't used them much).

class ProxyMethod(object):
def __init__(self,ob_name,meth):
self.ob_name=ob_name
if not hasattr(meth,'im_class'):
if hasattr(meth,'__call__'):
self.meth=getattr(meth,'__call__')
else:
raise ValueError("Method should be either a class method or
a callable class")
else:
self.meth=meth
 
G

George Sakkis

I actually ended up rewriting things (loosely based on George's suggested
code) with descriptors and not using metaclasses or decorators (so much for
my desire to use them).

With following implementation (unpolished at this stage but already
functional) I can have several instances of B objects inside of A object
and proxy certain methods to one or another object (I might be having a
case when I have A.b1 and A.b2 and passing some methods to b1 and others to
b2 having both of the same class B, maybe even multiplexing).

You seem to enjoy pulling the rug from under our feet by changing the
requirements all the time :)
This one
seems to be fairly light as well without need to scan instances (well,
except for one getattr, but I couldn't get around it). Maybe I didn't
account for some shoot-in-the-foot scenarios but I can't come up with any..
Last time I played with __getattr__ I shot myself in the foot quite well
BTW :)

[code snipped]

class A:
    b=None
^^^^^^ you don't need this
    def __init__(self,b=None):
        self.val='aval'
        self.b=b
        b.val='aval-b'

    def mymethod(self,a):
        print "A::mymethod, ",a

    bmethod = ProxyMethod('b',B.bmethod)

Although this works, the second argument to ProxyMethod shouldn't be
necessary, it's semantically redundant; ideally you would like to
write it as "bmethod = ProxyMethod('b')". As before, I don't think
that's doable without metaclasses (or worse, stack frame hacking).
Below is the update of my original recipe; interestingly, it's
(slightly) simpler than before:

#======= usage ========================================

from proxies import Proxy

class B(object):
def __init__(self, val): self.val = val
def bmethod(self,n): print "B::bmethod", self.val, n
def bmethod2(self,n,m): print "B::bmethod2", self.val, n, m

class C(object):
def __init__(self, val): self.val = val
def cmethod(self,x): print "C::cmethod", self.val, x
def cmethod2(self,x,y): print "C::cmethod2",self.val, x, y
cattr = 4

class A(Proxy):
# DelegateMap:
# Maps each delegate method to the proxy attribute that refers to
the
# respective delegate object
DelegateMap = {
'bmethod' : 'b1',
'bmethod2': 'b2',
'cmethod' : 'c',
# do NOT delegate C.cmethod2
#'cmethod2': 'c',
}

def __init__(self, b1, b2, c):
print "init A()"
# must call Proxy.__init__
super(A,self).__init__(b1=b1, b2=b2, c=c)

def amethod(self,a):
print "A::mymethod",a

if __name__ == '__main__':
a = A(B(10), B(20), C(30))
a.amethod('foo')

print "bound proxy calls"
a.bmethod('foo')
a.bmethod2('bar','baz')
a.cmethod('foo')
try: a.cmethod2('bar','baz')
except Exception, ex: print ex

print "unbound proxy calls"
A.bmethod(a,'foo')
A.bmethod2(a,'bar','baz')
A.cmethod(a, 'foo')
try: A.cmethod2(a,'bar','baz')
except Exception, ex: print ex

#======= output ========================================

init A()
A::mymethod foo
bound proxy calls
B::bmethod 10 foo
B::bmethod2 20 bar baz
C::cmethod 30 foo
'A' object has no attribute 'cmethod2'
unbound proxy calls
B::bmethod 10 foo
B::bmethod2 20 bar baz
C::cmethod 30 foo
type object 'A' has no attribute 'cmethod2'

#====== proxies.py ======================================

class _ProxyMeta(type):
def __new__(meta, name, bases, namespace):
for methodname in namespace.get('DelegateMap', ()):
if methodname not in namespace:
namespace[methodname] = _ProxyMethod(methodname)
return super(_ProxyMeta,meta).__new__(meta, name, bases,
namespace)


class _ProxyMethod(object):
def __init__(self, name):
self._name = name

def __get__(self, proxy, proxytype):
if proxy is not None:
return proxy._get_target_attr(self._name)
else:
return self._unbound_method

def _unbound_method(self, proxy, *args, **kwds):
method = proxy._get_target_attr(self._name)
return method(*args, **kwds)


class Proxy(object):
__metaclass__ = _ProxyMeta

def __init__(self, **attr2delegate):
self.__dict__.update(attr2delegate)

def _get_target_attr(self, name):
try:
delegate = getattr(self, self.DelegateMap[name])
return getattr(delegate, name)
except (KeyError, AttributeError):
raise AttributeError('%r object has no attribute %r' %
(self.__class__.__name__, name))


HTH,
George
 
D

Dmitry S. Makovey

George said:
You seem to enjoy pulling the rug from under our feet by changing the
requirements all the time :)

but that's half the fun! ;)

Bit more seriously - I didn't know I had those requirements until now :) I'm
kind of exploring where can I get with those ideas. Initial post was based
exactly on what I had in my code with desire to make it more
automated/{typo,fool}-proof/robust, elegant and possibly faster. Hopefully
such behavior is not off-topic on this list (if it is - let me know and
I'll go exploring solo).
Although this works, the second argument to ProxyMethod shouldn't be
necessary, it's semantically redundant; ideally you would like to
write it as "bmethod = ProxyMethod('b')".

since I'm already on exploratory trail (what about that rug being pulled
from under....?) With my code I can do really dirty tricks like this (not
necessary that I'm going to):

class B_base:
def bmethod(self):
print 'B_base'

class B(B_base):
def bmethod(self):
print 'B'

class A:
bmethod=ProxyMethod('b',B_base.bmethod)
As before, I don't think
that's doable without metaclasses (or worse, stack frame hacking).
Below is the update of my original recipe; interestingly, it's
(slightly) simpler than before:

Out of curiosity (and trying to understand): why do you insist on
dictionaries with strings contents ( {'bmethod' : 'b1' } ) rather than
something more explicit ? Again, I can see that your code is working and I
can even understand what it's doing, just trying to explore alternatives :)

I guess my bias is towards more explicit declarations thus

bmethod=ProxyMethod('b',B.bmethod)

looks more attractive to me, but I stand to be corrected/educated why is
that not the right thing to do?

Another thing that turns me away from string dictionaries is that those are
the ones causing me more trouble hunting down typos. Maybe it's just "my
thing" so I'm not going to insist on it. I'm open to arguments against that
theory.

One argument I can bring in defence of more explicit declarations is IDE
parsing when autocompletion for B.bme... pops up (suggesting bmethod and
bmethod2) and with 'b':'bmethod' it never happens. However that's not good
enough reason to stick to it if it introduces other problems. What kind of
problems could those be?

P.S.
so far I find this discussion quite educating BTW. I am more of a "weekend
programmer" with Python - using basic language for most of things I need,
however I decided to explore further to educate myself and to write more
efficient code.
 
G

George Sakkis

since I'm already on exploratory trail (what about that rug being pulled
from under....?) With my code I can do really dirty tricks like this (not
necessary that I'm going to):

class B_base:
def bmethod(self):
print 'B_base'

class B(B_base):
def bmethod(self):
print 'B'

class A:
bmethod=ProxyMethod('b',B_base.bmethod)

Yes, that's indeed dirty; don't do it :)
Out of curiosity (and trying to understand): why do you insist on
dictionaries with strings contents ( {'bmethod' : 'b1' } ) rather than
something more explicit ? Again, I can see that your code is working and I
can even understand what it's doing, just trying to explore alternatives :)

I guess my bias is towards more explicit declarations thus

bmethod=ProxyMethod('b',B.bmethod)

looks more attractive to me, but I stand to be corrected/educated why is
that not the right thing to do?

I see where you're coming from and I also prefer explicit reflection
mechanisms instead of strings (e.g. avoid eval/exec as much as
possible). As I mentioned, the second argument to ProxyMethod is (for
all sane purposes) redundant, so if you could implement it in a way
that "bmethod = ProxyMethod('b')" worked, I would be all for it, but
AFAIK it's not possible without a metaclass. A dict with string keys
and values to be consumed by a metaclass is perhaps the simplest thing
that could possibly work. It contains all the relevant information for
hooking the proxy to the delegate methods and nothing more; zero
boilerplate code overhead. Also note that it's not that big of a
difference; you have to provide the attribute name as a string anyway.
Another thing that turns me away from string dictionaries is that those are
the ones causing me more trouble hunting down typos. Maybe it's just "my
thing" so I'm not going to insist on it. I'm open to arguments against that
theory.

From my experience, I am equally prone to typos for both strings and
regular attributes; I agree though that the traceback information is
often more helpful when you mistype an attribute.
One argument I can bring in defence of more explicit declarations is IDE
parsing when autocompletion for B.bme... pops up (suggesting bmethod and
bmethod2) and with 'b':'bmethod' it never happens.

I don't rely on autocompleting IDEs, at least in dynamic languages, so
it's not much of an issue for me but yes, it's another small argument
against strings.

George
 
G

George Sakkis

I see where you're coming from and I also prefer explicit reflection
mechanisms instead of strings (e.g. avoid eval/exec as much as
possible). As I mentioned, the second argument to ProxyMethod is (for
all sane purposes) redundant, so if you could implement it in a way
that "bmethod = ProxyMethod('b')" worked, I would be all for it, but
AFAIK it's not possible without a metaclass.

Just for completeness, here's a metaclass version that uses
ProxyMethod declarations instead of a dict; you'll probably like this
better:

#======= usage =========================
from proxies import Proxy, ProxyMethod

class B(object):
def __init__(self, val): self.val = val
def bmethod(self,n): print "B::bmethod", self.val, n
def bmethod2(self,n,m): print "B::bmethod2", self.val, n, m

class C(object):
def __init__(self, val): self.val = val
def cmethod(self,x): print "C::cmethod", self.val, x
def cmethod2(self,x,y): print "C::cmethod2",self.val, x, y
cattr = 4

class A(Proxy):
def __init__(self, b1, b2, c):
print "init A()"
# must call Proxy.__init__
super(A,self).__init__(b1=b1, b2=b2, c=c)

def amethod(self,a):
print "A::mymethod",a

bmethod = ProxyMethod('b1')
bmethod2 = ProxyMethod('b2')
cmethod = ProxyMethod('c')


a = A(B(10), B(20), C(30))
a.amethod('foo')

print "bound proxy calls"
a.bmethod('foo')
a.bmethod2('bar','baz')
a.cmethod('foo')
try: a.cmethod2('bar','baz')
except Exception, ex: print ex

print "unbound proxy calls"
A.bmethod(a,'foo')
A.bmethod2(a,'bar','baz')
A.cmethod(a, 'foo')
try: A.cmethod2(a,'bar','baz')
except Exception, ex: print ex

#======= output ========================================

init A()
A::mymethod foo
bound proxy calls
B::bmethod 10 foo
B::bmethod2 20 bar baz
C::cmethod 30 foo
'A' object has no attribute 'cmethod2'
unbound proxy calls
B::bmethod 10 foo
B::bmethod2 20 bar baz
C::cmethod 30 foo
type object 'A' has no attribute 'cmethod2'

#====== proxies.py ======================================

class _ProxyMeta(type):
def __new__(meta, name, bases, namespace):
for attrname,value in namespace.iteritems():
if isinstance(value, ProxyMethod) and value.name is None:
value.name = attrname
return super(_ProxyMeta,meta).__new__(meta, name, bases,
namespace)


class ProxyMethod(object):
def __init__(self, proxy_attr, name=None):
self._proxy_attr = proxy_attr
self.name = name

def __get__(self, proxy, proxytype):
if proxy is not None:
return self.__get_target_attr(proxy)
else:
return self.__unbound_method

def __unbound_method(self, proxy, *args, **kwds):
method = self.__get_target_attr(proxy)
return method(*args, **kwds)

def __get_target_attr(self, proxy):
try:
delegate = getattr(proxy, self._proxy_attr)
return getattr(delegate, self.name)
except AttributeError:
raise AttributeError('%r object has no attribute %r' %
(proxy.__class__.__name__,
self.name))


class Proxy(object):
__metaclass__ = _ProxyMeta

def __init__(self, **attr2delegate):
self.__dict__.update(attr2delegate)

#======================================================

If you want to eliminate completely specifying attributes with
strings, it's easy to modify the above so that you write instead:

class A(Proxy):
...
bmethod = ProxyMethod(lambda self: self.b1)
bmethod2 = ProxyMethod(lambda self: self.b2)

This is more verbose for the common case, but it's more flexible in
cases where the callable may be more complex than a plain getattr().
Actually you can support both, it doesn't have to be either/or; just
check whether the argument to ProxyMethod is a callable and if not,
make it:

from operator import attrgetter

class ProxyMethod(object):
def __init__(self, proxy_attr, name=None):
if not callable(proxy_attr):
proxy_attr = attrgetter(proxy_attr)
...

Remaining Implementation is left as an exercise to the reader ;)

George
 
G

George Sakkis

If you want to eliminate completely specifying attributes with
strings, it's easy to modify the above so that you write instead:

class A(Proxy):
...
bmethod = ProxyMethod(lambda self: self.b1)
bmethod2 = ProxyMethod(lambda self: self.b2)

It's funny how often you come with a better solution a few moments
after htting send! The snippet above can (ab)use the decorator syntax
so that it becomes:

class A(Proxy):

@ProxyMethod
def bmethod(self):
return self.b1

@ProxyMethod
def bmethod2(self):
return self.b2

With the observation that ProxyMethod has access both to the callable
that returns the delegate and the name of the delegated method, we can
remove the need for the metaclass and the Proxy base class altogether:

class proxymethod(object):
def __init__(self, get_delegate):
self._get_delegate = get_delegate

def __get__(self, proxy, proxytype):
if proxy is not None:
return self.__get_target_attr(proxy)
else:
return self.__unbound_method

def __unbound_method(self, proxy, *args, **kwds):
method = self.__get_target_attr(proxy)
return method(*args, **kwds)

def __get_target_attr(self, proxy):
get_delegate = self._get_delegate
try: return getattr(get_delegate(proxy),
get_delegate.__name__)
except AttributeError:
raise AttributeError('%r object has no attribute %r' %
(proxy.__class__.__name__, get_delegate.__name__))


class A(object):

def __init__(self, b1, b2):
self.b1 = b1
self.b2 = b2

def amethod(self,a):
print "A::mymethod",a

@proxymethod
def bmethod(self):
return self.b1

@proxymethod
def bmethod2(self):
return self.b2


a = A(B(10), B(20))
a.amethod('foo')

print "bound proxy calls"
a.bmethod('foo')
a.bmethod2('bar','baz')

print "unbound proxy calls"
A.bmethod(a,'foo')
A.bmethod2(a,'bar','baz')

So back to the OP's original question and after a long circle.. a
decorator might well be the right thing to use after all :)

George
 
D

Dmitry S. Makovey

George said:
It's funny how often you come with a better solution a few moments
after htting send! The snippet above can (ab)use the decorator syntax
so that it becomes:

class A(Proxy):

@ProxyMethod
def bmethod(self):
return self.b1

@ProxyMethod
def bmethod2(self):
return self.b2

That is outstanding! This code looks very clean to me (just a touch cryptic
around declarations in A, but that was unavoidable anyway). Seems like the
right way to read it would be bottom up (or is it only my mind so
perverted?). By the looks of it - it does exactly what I needed with great
number of possibilities behind it and is very lightweight and transparent.
Now I regret I haven't come up with it myself :-D

George, at this point I'm out of rugs - so no more rug pulling from under
your feet for me.

Now I'm going to apply all this knowledge to my code, see how that goes and
come back with more questions later.

Thank you (all) very much for a great discussion. This thread educated me
quite a bit on descriptors and why one would need them, and decorators -
just as subject line suggested, were not forgotten.
 
G

George Sakkis

That is outstanding!

FYI, in case you missed it the final version doesn't need a Proxy base
class, just inherit from object. Also lowercased ProxyMethod to look
similar to staticmethod/classmethod:

class A(object):

def __init__(self, b1, b2):
self.b1 = b1
self.b2 = b2

@proxymethod
def bmethod(self):
return self.b1

@proxymethod
def bmethod2(self):
return self.b2

George
 
D

Dmitry S. Makovey

George said:
FYI, in case you missed it the final version doesn't need a Proxy base
class, just inherit from object. Also lowercased ProxyMethod to look
similar to staticmethod/classmethod:

I cought that, just quoted the wrong one :)
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,780
Messages
2,569,611
Members
45,276
Latest member
Sawatmakal

Latest Threads

Top