customized instance dictionaries, anyone?

[wolfgang.lipp]

some time after posting my `Linkdict recipe`__ to aspn__
-- basically, a dictionary with run-time delegational
lookup, but this is not important here -- i thought gee
that would be fun to make such a customized dictionary
thingie an instance dictionary, and get some custom
namespace behavior out of that.

... __: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/465748
... __: http://aspn.activestate.com/

here is a simplified example: first, the customized
dictionary class and a test::


class CustomDict( dict ):

defaultValue = 'THIS ITEM NOT AVAILABLE'

def __getitem__( self, name ):
try:
return super( CustomDict, self ).__getitem__( name )
except KeyError:
return self.defaultValue

def __contains__( self, name ):
return True

def has_key( self, name ):
return True

print '----------------------------------------'
cd = CustomDict( foo = 'bar' )
print cd[ 'foo' ]
print cd[ 'bar' ]
print 'bar' in cd
print cd.has_key( 'bar' )

this gives us::

----------------------------------------
bar
THIS ITEM NOT AVAILABLE
True
True

so it appears to work. note that we may have failed to
implement all the conceivable ways to test for
membership (such as searching through ``keys()``) or to
retrieve a value for a given key. more on that below.
now for the class to utilize this definition::

class X( object ):

def __init__( self ):
self.__dict__ = CustomDict( foo = 'bar' )

and the test code for that::

print '----------------------------------------'
x = X()
print x.__dict__[ 'foo' ]
print x.__dict__[ 'bar' ]
print x.foo
print x.bar

which yields::

----------------------------------------
bar
THIS ITEM NOT AVAILABLE
bar

Traceback (most recent call last):
File "C:\home\projects\__svn__\sundry\#.py", line 39, in ?
print x.bar
AttributeError: 'X' object has no attribute 'bar'

ok. so the custom dict *basically* works as expected,
since it does successfully make ``x.foo`` available --
no surprise here. unfortunately, looking up ``x.bar``,
which should really return the default value string,
causes an ``AttributeError`` to be raised.

now of course given the short definition of
``CustomDict``, perhaps there is an essential lookup
method that has not been overwritten but that is
internally used for attribute lookup. however, i
carefully tested my actual class (from the recipe
mentioned above) and also compared the methods defined
there against the standard ``dict()`` interface, and
nothing of importance appeared to be missing. i also
tried to bind the dictionary to the instance earlier, in
``__new__``, to no avail. am i missing something here?

_wolfgang

 

[Bengt Richter]

some time after posting my `Linkdict recipe`__ to aspn__
-- basically, a dictionary with run-time delegational
lookup, but this is not important here -- i thought gee
that would be fun to make such a customized dictionary
thingie an instance dictionary, and get some custom
namespace behavior out of that.

.. __: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/465748
.. __: http://aspn.activestate.com/

here is a simplified example: first, the customized
dictionary class and a test::


class CustomDict( dict ):

defaultValue = 'THIS ITEM NOT AVAILABLE'

def __getitem__( self, name ):
try:
return super( CustomDict, self ).__getitem__( name )
except KeyError:
return self.defaultValue

def __contains__( self, name ):
return True

def has_key( self, name ):
return True

print '----------------------------------------'
cd = CustomDict( foo = 'bar' )
print cd[ 'foo' ]
print cd[ 'bar' ]
print 'bar' in cd
print cd.has_key( 'bar' )

this gives us::

----------------------------------------
bar
THIS ITEM NOT AVAILABLE
True
True

so it appears to work. note that we may have failed to
implement all the conceivable ways to test for
membership (such as searching through ``keys()``) or to
retrieve a value for a given key. more on that below.
now for the class to utilize this definition::

class X( object ):

def __init__( self ):
self.__dict__ = CustomDict( foo = 'bar' )

and the test code for that::

print '----------------------------------------'
x = X()
print x.__dict__[ 'foo' ]
print x.__dict__[ 'bar' ]
print x.foo
print x.bar

which yields::

----------------------------------------
bar
THIS ITEM NOT AVAILABLE
bar

Traceback (most recent call last):
File "C:\home\projects\__svn__\sundry\#.py", line 39, in ?
print x.bar
AttributeError: 'X' object has no attribute 'bar'

ok. so the custom dict *basically* works as expected,
since it does successfully make ``x.foo`` available --
no surprise here. unfortunately, looking up ``x.bar``,
which should really return the default value string,
causes an ``AttributeError`` to be raised.

now of course given the short definition of
``CustomDict``, perhaps there is an essential lookup
method that has not been overwritten but that is
internally used for attribute lookup. however, i
carefully tested my actual class (from the recipe
mentioned above) and also compared the methods defined
there against the standard ``dict()`` interface, and
nothing of importance appeared to be missing. i also
tried to bind the dictionary to the instance earlier, in
``__new__``, to no avail. am i missing something here?

Well, if you compare with the following, maybe something will fall into place?
... defaultValue = 'THIS ITEM NOT AVAILABLE'
... def __getitem__( self, name ):
... try:
... return super( CustomDict, self ).__getitem__( name )
... except KeyError:
... return self.defaultValue
... def __contains__( self, name ):
... return True
... def has_key( self, name ):
... return True
... ... __dict__ = property(lambda self:self._dict)
... def __getattr__(self, attr): return self.__dict__[attr]
... def __init__( self ):
... self._dict = CustomDict( foo = 'bar' )
...

>>> x = X()
>>> print x.__dict__['foo'] bar
>>> print x.__dict__['bar'] THIS ITEM NOT AVAILABLE
>>> print x.foo bar
>>> print x.bar

THIS ITEM NOT AVAILABLE

Additional data points:

>>> x.__dict__ {'foo': 'bar'}
>>> X.__dict__

>>> X.__dict__['__dict__']

<property object at 0x02EEF70C>

and
... def _getdict(self): print '_getdict'; return self._dict
... __dict__=property(_getdict)
... def __init__( self ):
... self._dict = CustomDict( foo = 'bar' )
... _getdict
{'foo': 'bar'} Traceback (most recent call last):

>>> def ga(self, attr): print '__getattr__(%s)'%attr; return self.__dict__[attr] ...
>>> Y.__getattr__ = ga
>>> y.foo

__getattr__(foo)
_getdict
'bar'

Regards,
Bengt Richter

 

[wolfgang.lipp]

thx! indeed, it worked -- took me some time to figure out how to
implement the setting of attributes, too. i finally managed to get that
done using super:

custom dictionary, unchanged::

class CustomDict( dict ):

defaultValue = 'THIS ITEM NOT AVAILABLE'

def __getitem__( self, name ):
#print 'CustomDict.__getitem__( %r )' % ( name, )
try:
return super( CustomDict, self ).__getitem__( name )
except KeyError:
return self.defaultValue

def __contains__( self, name ):
return True

def has_key( self, name ):
return True

dictionary user class::

class X( object ):

def __init__( self ):
#print 'X.__init__()'
self._dict = CustomDict( foo = 'bar' )

@property
def __dict__( self ):
#print 'X.__dict__ ( get() )'
return self._dict

def __getattr__( self, name ):
#print 'X.__getattr__( %r )' % ( name, )
return self.__dict__[ name ]

def __setattr__( self, name, value ):
#print 'X.__setattr__( %r, %r )' % ( name, value, )
if name == '_dict':
return super( X, self ).__setattr__( name, value )
self._dict[ name ] = value

the ``~.__setattr__()`` method tests for the special name ``_dict`` and
defers execution to the super of the ``X`` instance, ``object``. other
stuff is handled by the instance itself.
testing that with ::

x = X()
print x.__dict__[ 'foo' ]
print x.__dict__[ 'bar' ]
print x.foo
print x.bar
print x.__dict__
x.oops = 42
print x.__dict__

yields ::

bar
THIS ITEM NOT AVAILABLE
bar
THIS ITEM NOT AVAILABLE
{'foo': 'bar'}
{'foo': 'bar', 'oops': 42}

as expected.

i tried to reason *why* the usage of a property makes this particular
piece of code work, but i seemingly can't find out. anyone around who
has thoughts on that?

_wolfgang

 

[Bengt Richter]

thx! indeed, it worked -- took me some time to figure out how to
implement the setting of attributes, too. i finally managed to get that
done using super:

Seems good.

the ``~.__setattr__()`` method tests for the special name ``_dict`` and
defers execution to the super of the ``X`` instance, ``object``. other
stuff is handled by the instance itself.

seems a clean way to do it.

testing that with ::

x = X()
print x.__dict__[ 'foo' ]
print x.__dict__[ 'bar' ]
print x.foo
print x.bar
print x.__dict__
x.oops = 42
print x.__dict__

yields ::

bar
THIS ITEM NOT AVAILABLE
bar
THIS ITEM NOT AVAILABLE
{'foo': 'bar'}
{'foo': 'bar', 'oops': 42}

as expected.

i tried to reason *why* the usage of a property makes this particular
piece of code work, but i seemingly can't find out. anyone around who
has thoughts on that?

I had an inspiration and I think succeeded in doing what you were originally
trying to do (replace the actual instance dict with your custom dict), which
I tried to do but didn't realize at first that the __init__ setting
of x.__dict__ was really setting x.__dict__['__dict__'] not setting the initial
x.__dict__ itself. __dict__ is a peculiar animal, and looking for an instance's
attribute dict doesn't start at the instance. If you look for instance.__dict__,
it is just like looking for any other attribute, and it starts at type(instance).mro()[0]
looking for a descriptor (which a method also is). But the first thing "found" is
a dict proxy for looking up attributes, and when it looks up '__dict__' it returns
a descriptor, which then gets its __get__ method called with the instance whose '__dict__'
is being sought. You have to use the corresponding __set__ method to set the value of '__dict__'
(in this case the CustomDict instance). Otherwise instance.__dict__ will automatically
be set to {} and then used so you have {'__dict__':CustomDict()} instead of CustomDict() itself.

Once this is initialized correctly, then all the machinery works, except we still need to intercept
__getattr__ for some reason. I suspect this is another symptom of some optimization. One of these
days I will have to look in the source ;-) You would think it could give up on the mro method search
and get the __dict__ normally to get the attribute, but some mechanism is not finding the custom
dict, or else maybe it's bypassing the __getitem__ and going directly to the base dict method.

Someday I'll have to look in the source ;-)

customdict as before
... defaultValue = 'THIS ITEM NOT AVAILABLE'
... def __getitem__( self, name ):
... try:
... return super( CustomDict, self ).__getitem__( name )
... except KeyError:
... return self.defaultValue
... def __contains__( self, name ):
... return True
... def has_key( self, name ):
... return True
... ... def __getattr__(self, attr):
... return self.__dict__[attr]
... #return type(self).__dict__['__dict__'].__get__(self)[attr]
... def __init__( self, *args, **kw ):
... type(self).__dict__['__dict__'].__set__(self, CustomDict(*args, **kw)
...

>>> x = X(foo='bar')
>>> print x.__dict__['foo'] bar
>>> print x.__dict__['bar'] THIS ITEM NOT AVAILABLE
>>> print x.foo bar
>>> print x.bar THIS ITEM NOT AVAILABLE
>>> x.oops = 42
>>> print x.__dict__

{'foo': 'bar', 'oops': 42}

Looking at a few things of interest:

>>> vars(x) {'foo': 'bar', 'oops': 42}
>>> type(vars(x))

>>> type(x.__dict__)

>>> vars(x)['?'] 'THIS ITEM NOT AVAILABLE'
>>> type(x)

>>> type(x).__dict__

>>> type(x).__dict__['__dict__']

>>> type(x).__dict__['__dict__'].__get__

>>> type(x).__dict__['__dict__'].__get__(x) {'foo': 'bar', 'oops': 42}
>>> type(type(x).__dict__['__dict__'].__get__(x))

<class '__main__.CustomDict'>

The reason the property was needed before was really several reasons.
First was that x.__dict__ wasn't being set properly, so the internal
machinery wasn't finding the custom dict. I changed the name to _dict
and let it be an ordinary attribute, but then used property to fake
what normal stuff would do if x.__dict__ itself was set, not x.__dict__['__dict__']

Wasted a bunch of time trying to get rid of that __getattr__ ;-/

Regards,
Bengt Richter