Case study: library class inheritance with property declarations

[cdleary]

Hi all,

It's possible that I'm missing the obvious -- I've been up for over 24
hours and I'm most likely dehydrated from mass coffee intake, but I
figure many people in similar circumstances will be searching
comp.lang.python one day, so here goes!


class LibraryClass(object):

"""
A class whose implementation is out of my hands.
"""

def __init__(self):
"""
Follows good dynamic-language form and binds all instance
variables at initialization time.
"""
# Omitted: complex initialization functionality.
self.useful_attr = None


class MyInheritedClass(LibraryClass):

"""
My refinement of the functionality offered by the LibraryClass. I
now want the instance to initialize with a reference to an
external
object, and the useful_attr defined in the superclass will now
reference an attribute of that external object via fget.

Changing the attribute of the external object has undefined
behavior, so I want to omit an fset in the property declaration;
however, I have to provide some way for the superclass to
initialize useful_attr -- I can't change the superclass' code, as
it
resides in a library that is out of my hands.
"""

def __init__(self, external_obj):
LibraryClass.__init__(self)
self._external_obj = external_obj

def get_useful_attr(self):
return self._external_obj.proxy_useful_attr

useful_attr = property(fget=get_useful_attr)


def test():
class _Fake(object):
pass
external_obj = _Fake()
external_obj.proxy_useful_attr = 12
spam = MyInheritedClass(external_obj)


if __name__ == '__main__':
test()
EOF


If you're one of those people who doesn't like laboriously reading
contrived examples (elitists I'll boil it down for you: Library
class initializes some attribute, but derived class wants to eliminate
fsets for said attribute. As a result, our ideal solution

Of course, this means that the derived class will raise an error in
some circumstances where the base class wouldn't (when you're setting
that attribute), but one can assume that the inheritance is
worthwhile.

How do I come up with silly solutions to circumvent this? Let me count
the ways...

1. So-and-so: make an fset that does nothing. This ignores (what
should be) errors in code that uses MyInheritedClass in an attempt to
accommodate a useless statement in the base class -- surely non-ideal.
2. The ugly one: since you can probably view the library, copy and
paste the complex initialization functionality in the above, but leave
out the bad statement. This not only forfeits the ideals of
inheritance, but makes you totally incompatible with future library
changes.
3. Cheerleader: Pure evil. On top of the ugliness of 2, you assume
that across library revisions the indenting won't change and that the
troublesome statement will remain on the same line, and pull off one
of these babies:

def super_evil_test():
from inspect import getsourcelines
exec(''.join([line[4:] for line in
getsourcelines(LibraryClass.__init__)[0][:-1]]))
LibraryClass.__init__ = __init__
test() # Passes, but several angels no longer get their wings

Totally kidding, everybody! I hope Guido doesn't read this thread...

And this concludes the sleep deprived rambling that follows the
somewhat interesting case in point. Thoughts?

 

[Bruno Desthuilliers]

(e-mail address removed) a écrit :
(snip)

Last post -- I swear.

I failed to realize that it's all part of an extremely well defined
attribute resolution protocol, and handled via the descriptor
specification. Discussing descriptors was on the TODO list for the
type/class unification document, but there's a very fulfilling
explanation by Raymond Hettinger: http://users.rcn.com/python/download/Descriptor.htm
Also, the official doc is here: http://docs.python.org/ref/descriptors.html

Maybe the documentation for the property builtin should make reference
to the descriptor specification? If nobody thinks this is silly, I'll
submit a documentation patch in a few days.

What to say ? As an old-time pyton user (well... 7+ years now) and
meta-programming addict, it's nothing new to me, but the fact that you
didn't realize it is certainly a clear indication that documentation is
not really up to date, so yes, submitting a patch is certainly a pretty
good idea.

Sorry for the spam -- hope someone besides me learns from it!

What's to be learned is mostly that we (the whole community) should take
more care of the doc (which I never contributed FWIW, so I'm the first
one that should feel in guilt here).

 

[cdleary]

Hi all,

It's possible that I'm missing the obvious -- I've been up for over 24
hours and I'm most likely dehydrated from mass coffee intake, but I
figure many people in similar circumstances will be searching
comp.lang.python one day, so here goes!

class LibraryClass(object):

"""
A class whose implementation is out of my hands.
"""

def __init__(self):
"""
Follows good dynamic-language form and binds all instance
variables at initialization time.
"""
# Omitted: complex initialization functionality.
self.useful_attr = None

class MyInheritedClass(LibraryClass):

"""
My refinement of the functionality offered by the LibraryClass. I
now want the instance to initialize with a reference to an
external
object, and the useful_attr defined in the superclass will now
reference an attribute of that external object via fget.

Changing the attribute of the external object has undefined
behavior, so I want to omit an fset in the property declaration;
however, I have to provide some way for the superclass to
initialize useful_attr -- I can't change the superclass' code, as
it
resides in a library that is out of my hands.
"""

def __init__(self, external_obj):
LibraryClass.__init__(self)
self._external_obj = external_obj

def get_useful_attr(self):
return self._external_obj.proxy_useful_attr

useful_attr = property(fget=get_useful_attr)

def test():
class _Fake(object):
pass
external_obj = _Fake()
external_obj.proxy_useful_attr = 12
spam = MyInheritedClass(external_obj)

if __name__ == '__main__':
test()
EOF

If you're one of those people who doesn't like laboriously reading
contrived examples (elitists I'll boil it down for you: Library
class initializes some attribute, but derived class wants to eliminate
fsets for said attribute. As a result, our ideal solution

Of course, this means that the derived class will raise an error in
some circumstances where the base class wouldn't (when you're setting
that attribute), but one can assume that the inheritance is
worthwhile.

How do I come up with silly solutions to circumvent this? Let me count
the ways...

1. So-and-so: make an fset that does nothing. This ignores (what
should be) errors in code that uses MyInheritedClass in an attempt to
accommodate a useless statement in the base class -- surely non-ideal.
2. The ugly one: since you can probably view the library, copy and
paste the complex initialization functionality in the above, but leave
out the bad statement. This not only forfeits the ideals of
inheritance, but makes you totally incompatible with future library
changes.
3. Cheerleader: Pure evil. On top of the ugliness of 2, you assume
that across library revisions the indenting won't change and that the
troublesome statement will remain on the same line, and pull off one
of these babies:

def super_evil_test():
from inspect import getsourcelines
exec(''.join([line[4:] for line in
getsourcelines(LibraryClass.__init__)[0][:-1]]))
LibraryClass.__init__ = __init__
test() # Passes, but several angels no longer get their wings

Totally kidding, everybody! I hope Guido doesn't read this thread...

And this concludes the sleep deprived rambling that follows the
somewhat interesting case in point. Thoughts?

I'm sorry -- the solution was not /enough/ coffee. Got another cup and
sat down with the type/class unification doc, and found this thought-
stimulating portion:

http://www.python.org/download/releases/2.2/descrintro/#property
If you want to override the __get__ operation for properties when used
as a class attribute, you can subclass property - it is a new-style
type itself - to extend its __get__ method, or you can define a
descriptor type from scratch by creating a new-style class that
defines __get__, __set__ and __delete__ methods.
....
The get method won't be called when the property is accessed as a
class attribute (C.x) instead of as an instance attribute (C().x).


Seeing as how property is just a wrapper class, we don't need to
declare it in the class body, though it /is/ the convention and the
way it's done in all the docs I've seen. We fix our inherited class to
be the following:

[snip]
class MyInheritedClass(LibraryClass):

"""
My refinement of the functionality offered by the LibraryClass. I
now want the instance to initialize with a reference to an
external
object, and the useful_attr defined in the superclass will now
reference an attribute of that external object via fget.

Changing the attribute of the external object has undefined
behavior, so I want to omit an fset in the property declaration;
however, I have to provide some way for the superclass to
initialize useful_attr -- I can't change the superclass' code, as
it
resides in a library that is out of my hands.
"""

def __init__(self, external_obj):
LibraryClass.__init__(self)
self.useful_attr = property(fget=self.get_useful_attr)
self._external_obj = external_obj

def get_useful_attr(self):
return self._external_obj.proxy_useful_attr
[snip]

And it tests like a charm.

 

[cdleary]

Hi all,

It's possible that I'm missing the obvious -- I've been up for over 24
hours and I'm most likely dehydrated from mass coffee intake, but I
figure many people in similar circumstances will be searching
comp.lang.python one day, so here goes!

class LibraryClass(object):

"""
A class whose implementation is out of my hands.
"""

def __init__(self):
"""
Follows good dynamic-language form and binds all instance
variables at initialization time.
"""
# Omitted: complex initialization functionality.
self.useful_attr = None

class MyInheritedClass(LibraryClass):

"""
My refinement of the functionality offered by the LibraryClass. I
now want the instance to initialize with a reference to an
external
object, and the useful_attr defined in the superclass will now
reference an attribute of that external object via fget.

Changing the attribute of the external object has undefined
behavior, so I want to omit an fset in the property declaration;
however, I have to provide some way for the superclass to
initialize useful_attr -- I can't change the superclass' code, as
it
resides in a library that is out of my hands.
"""

def __init__(self, external_obj):
LibraryClass.__init__(self)
self._external_obj = external_obj

def get_useful_attr(self):
return self._external_obj.proxy_useful_attr

useful_attr = property(fget=get_useful_attr)

def test():
class _Fake(object):
pass
external_obj = _Fake()
external_obj.proxy_useful_attr = 12
spam = MyInheritedClass(external_obj)

if __name__ == '__main__':
test()
EOF

If you're one of those people who doesn't like laboriously reading
contrived examples (elitists I'll boil it down for you: Library
class initializes some attribute, but derived class wants to eliminate
fsets for said attribute. As a result, our ideal solution

Of course, this means that the derived class will raise an error in
some circumstances where the base class wouldn't (when you're setting
that attribute), but one can assume that the inheritance is
worthwhile.

How do I come up with silly solutions to circumvent this? Let me count
the ways...

1. So-and-so: make an fset that does nothing. This ignores (what
should be) errors in code that uses MyInheritedClass in an attempt to
accommodate a useless statement in the base class -- surely non-ideal.
2. The ugly one: since you can probably view the library, copy and
paste the complex initialization functionality in the above, but leave
out the bad statement. This not only forfeits the ideals of
inheritance, but makes you totally incompatible with future library
changes.
3. Cheerleader: Pure evil. On top of the ugliness of 2, you assume
that across library revisions the indenting won't change and that the
troublesome statement will remain on the same line, and pull off one
of these babies:

def super_evil_test():
from inspect import getsourcelines
exec(''.join([line[4:] for line in
getsourcelines(LibraryClass.__init__)[0][:-1]]))
LibraryClass.__init__ = __init__
test() # Passes, but several angels no longer get their wings

Totally kidding, everybody! I hope Guido doesn't read this thread...

And this concludes the sleep deprived rambling that follows the
somewhat interesting case in point. Thoughts?

I'm sorry -- the solution was not /enough/ coffee. Got another cup and
sat down with the type/class unification doc, and found this thought-
stimulating portion:

http://www.python.org/download/releases/2.2/descrintro/#property
If you want to override the __get__ operation for properties when used
as a class attribute, you can subclass property - it is a new-style
type itself - to extend its __get__ method, or you can define a
descriptor type from scratch by creating a new-style class that
defines __get__, __set__ and __delete__ methods.
...
The get method won't be called when the property is accessed as a
class attribute (C.x) instead of as an instance attribute (C().x).

Seeing as how property is just a wrapper class, we don't need to
declare it in the class body, though it /is/ the convention and the
way it's done in all the docs I've seen. We fix our inherited class to
be the following:

[snip]
class MyInheritedClass(LibraryClass):

"""
My refinement of the functionality offered by the LibraryClass. I
now want the instance to initialize with a reference to an
external
object, and the useful_attr defined in the superclass will now
reference an attribute of that external object via fget.

Changing the attribute of the external object has undefined
behavior, so I want to omit an fset in the property declaration;
however, I have to provide some way for the superclass to
initialize useful_attr -- I can't change the superclass' code, as
it
resides in a library that is out of my hands.
"""

def __init__(self, external_obj):
LibraryClass.__init__(self)
self.useful_attr = property(fget=self.get_useful_attr)
self._external_obj = external_obj

def get_useful_attr(self):
return self._external_obj.proxy_useful_attr
[snip]

And it tests like a charm.

Last post -- I swear.

I failed to realize that it's all part of an extremely well defined
attribute resolution protocol, and handled via the descriptor
specification. Discussing descriptors was on the TODO list for the
type/class unification document, but there's a very fulfilling
explanation by Raymond Hettinger: http://users.rcn.com/python/download/Descriptor.htm
Also, the official doc is here: http://docs.python.org/ref/descriptors.html

Maybe the documentation for the property builtin should make reference
to the descriptor specification? If nobody thinks this is silly, I'll
submit a documentation patch in a few days.

Sorry for the spam -- hope someone besides me learns from it!

- Chris

 

[Steve Holden]

Hi all,
It's possible that I'm missing the obvious -- I've been up for over 24
hours and I'm most likely dehydrated from mass coffee intake, but I
figure many people in similar circumstances will be searching
comp.lang.python one day, so here goes!
class LibraryClass(object):
"""
A class whose implementation is out of my hands.
"""
def __init__(self):
"""
Follows good dynamic-language form and binds all instance
variables at initialization time.
"""
# Omitted: complex initialization functionality.
self.useful_attr = None
class MyInheritedClass(LibraryClass):
"""
My refinement of the functionality offered by the LibraryClass. I
now want the instance to initialize with a reference to an
external
object, and the useful_attr defined in the superclass will now
reference an attribute of that external object via fget.
Changing the attribute of the external object has undefined
behavior, so I want to omit an fset in the property declaration;
however, I have to provide some way for the superclass to
initialize useful_attr -- I can't change the superclass' code, as
it
resides in a library that is out of my hands.
"""
def __init__(self, external_obj):
LibraryClass.__init__(self)
self._external_obj = external_obj
def get_useful_attr(self):
return self._external_obj.proxy_useful_attr
useful_attr = property(fget=get_useful_attr)
def test():
class _Fake(object):
pass
external_obj = _Fake()
external_obj.proxy_useful_attr = 12
spam = MyInheritedClass(external_obj)
if __name__ == '__main__':
test()
EOF
If you're one of those people who doesn't like laboriously reading
contrived examples (elitists I'll boil it down for you: Library
class initializes some attribute, but derived class wants to eliminate
fsets for said attribute. As a result, our ideal solution
Of course, this means that the derived class will raise an error in
some circumstances where the base class wouldn't (when you're setting
that attribute), but one can assume that the inheritance is
worthwhile.
How do I come up with silly solutions to circumvent this? Let me count
the ways...
1. So-and-so: make an fset that does nothing. This ignores (what
should be) errors in code that uses MyInheritedClass in an attempt to
accommodate a useless statement in the base class -- surely non-ideal.
2. The ugly one: since you can probably view the library, copy and
paste the complex initialization functionality in the above, but leave
out the bad statement. This not only forfeits the ideals of
inheritance, but makes you totally incompatible with future library
changes.
3. Cheerleader: Pure evil. On top of the ugliness of 2, you assume
that across library revisions the indenting won't change and that the
troublesome statement will remain on the same line, and pull off one
of these babies:
def super_evil_test():
from inspect import getsourcelines
exec(''.join([line[4:] for line in
getsourcelines(LibraryClass.__init__)[0][:-1]]))
LibraryClass.__init__ = __init__
test() # Passes, but several angels no longer get their wings
Totally kidding, everybody! I hope Guido doesn't read this thread...
And this concludes the sleep deprived rambling that follows the
somewhat interesting case in point. Thoughts?

I'm sorry -- the solution was not /enough/ coffee. Got another cup and
sat down with the type/class unification doc, and found this thought-
stimulating portion:

http://www.python.org/download/releases/2.2/descrintro/#property
If you want to override the __get__ operation for properties when used
as a class attribute, you can subclass property - it is a new-style
type itself - to extend its __get__ method, or you can define a
descriptor type from scratch by creating a new-style class that
defines __get__, __set__ and __delete__ methods.
...
The get method won't be called when the property is accessed as a
class attribute (C.x) instead of as an instance attribute (C().x).

Seeing as how property is just a wrapper class, we don't need to
declare it in the class body, though it /is/ the convention and the
way it's done in all the docs I've seen. We fix our inherited class to
be the following:

[snip]
class MyInheritedClass(LibraryClass):

"""
My refinement of the functionality offered by the LibraryClass. I
now want the instance to initialize with a reference to an
external
object, and the useful_attr defined in the superclass will now
reference an attribute of that external object via fget.

Changing the attribute of the external object has undefined
behavior, so I want to omit an fset in the property declaration;
however, I have to provide some way for the superclass to
initialize useful_attr -- I can't change the superclass' code, as
it
resides in a library that is out of my hands.
"""

def __init__(self, external_obj):
LibraryClass.__init__(self)
self.useful_attr = property(fget=self.get_useful_attr)
self._external_obj = external_obj

def get_useful_attr(self):
return self._external_obj.proxy_useful_attr
[snip]

And it tests like a charm.

Last post -- I swear.

I failed to realize that it's all part of an extremely well defined
attribute resolution protocol, and handled via the descriptor
specification. Discussing descriptors was on the TODO list for the
type/class unification document, but there's a very fulfilling
explanation by Raymond Hettinger: http://users.rcn.com/python/download/Descriptor.htm
Also, the official doc is here: http://docs.python.org/ref/descriptors.html

Maybe the documentation for the property builtin should make reference
to the descriptor specification? If nobody thinks this is silly, I'll
submit a documentation patch in a few days.

Sorry for the spam -- hope someone besides me learns from it!

And I hope you learn to get more sleep ;-)

short-on-sleep-myself-this-week-ly y'rs - steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

 

[cdleary]

(e-mail address removed) a écrit :
(snip)




What to say ? As an old-time pyton user (well... 7+ years now) and
meta-programming addict, it's nothing new to me, but the fact that you
didn't realize it is certainly a clear indication that documentation is
not really up to date, so yes, submitting a patch is certainly a pretty
good idea.


What's to be learned is mostly that we (the whole community) should take
more care of the doc (which I never contributed FWIW, so I'm the first
one that should feel in guilt here).

Actually I was wrong about this! Of course, the problem was
insufficient test coverage

The key addition: assert spam.useful_attr == 12 # FAILS!!! I failed to
realize it in the last posts...

When you don't declare a descriptor as a class attribute the
descriptor protocol isn't invoked! The descriptor protocol looks in
the /class/ attributes -- it transforms spam.useful_attr to
type(spam).__dict__['useful_attr'].__get__(spam, type(spam)), the
caveat being /type(spam)/.

From the doc: (http://users.rcn.com/python/download/Descriptor.htm)

The implementation works through a precedence chain that gives data
descriptors priority over instance variables, instance variables
priority over non-data descriptors, and assigns lowest priority to
__getattr__ if provided.

The data descriptors are only looked up in the class attributes -- the
instance attributes are returned as-is, so the above assertion fails!

The addition of 'print spam.useful_attr' produces <property object at
0x[something]>, because the portion of the descriptor protocol that
invokes __get__ is does not cover instance attributes (as declared in
__init__).

So, we're back to the original problem, with a bit more insight. What
if we declare the useful_attr property as a class attribute in the
constructor after the LibraryClass initialization function? It only
works once! Once you set a class attribute on the first instantiation
it's obviously there the next time you try to instantiate. Now our
test looks like this:

def test():
class _Fake(object):
pass
external_obj = _Fake()
external_obj.proxy_useful_attr = 12
spam = MyInheritedClass(external_obj)
assert spam.useful_attr == 12
eggs = MyInheritedClass(external_obj)
assert eggs.useful_attr == 12

Which provides coverage for the INCORRECT class-attribute-on-
instantiation solution, since eggs won't be able to instantiate. The
solution which fails this test looks like this:

class MyInheritedClass(LibraryClass):
# WRONG! Don't use this.
def __init__(self, external_obj):
LibraryClass.__init__(self)
def get_useful_attr(would_be_self): # has to be unbound
return would_be_self._external_obj.proxy_useful_attr
MyInheritedClass.useful_attr = property(get_useful_attr)
self._external_obj = external_obj

At this point it seems like the only viable option is to hack
getattribute to add special cases. To do this, we continue to use the
unbound method getter as our argument to property, but add the
getattribute hack which allows for specified instance attributes that
have the descriptor protocol invoked on them:

class MyInheritedClass(LibraryClass):
# ...
def __init__(self, external_obj):
LibraryClass.__init__(self)
def get_useful_attr(would_be_self):
return would_be_self._external_obj.proxy_useful_attr
self.useful_attr = property(fget=get_useful_attr)
self._external_obj = external_obj

def __getattribute__(self, attr_name):
"""
Hacks getattribute to allow us to return desired instance
attributes with the descriptor protocol.
"""
instance_descrs = ['useful_attr']
if attr_name in instance_descrs and attr_name in
self.__dict__:
attr = self.__dict__[attr_name]
return attr.__get__(self, type(self))
return object.__getattribute__(self, attr_name)

This passes the above test, but obviously requires a hack -- is there
a more elegant way to do it?

- Chris