"specialdict" module

G

Georg Brandl

Hello,

in follow-up to the recent "dictionary accumulator" thread, I wrote a
little module with several subclassed dicts.

Comments (e.g. makes it sense to use super), corrections, etc.? Is this
PEP material?

Docstrings, Documentation and test cases are to be provided later.

mfg
Georg

----------------------------------------------------------------------

class defaultdict(dict):
# _defaulttype: 0=no default, 1=defaultvalue, 2=defaultfactory
__slots__ = ['_defaulttype', '_default']

def __init__(self, *args, **kwargs):
self._defaulttype = 0

super(defaultdict, self).__init__(self, *args, **kwargs)

def setdefaultvalue(self, value):
self._defaulttype = 1
self._default = value

def setdefaultfactory(self, factory, *args, **kwargs):
if not callable(factory):
raise TypeError, 'default factory must be a callable'
self._defaulttype = 2
self._default = (factory, args, kwargs)

def cleardefault(self):
self._defaulttype = 0

def __getitem__(self, key):
try:
return super(defaultdict, self).__getitem__(key)
except KeyError:
if self._defaulttype == 0:
raise
elif self._defaulttype == 1:
return self.setdefault(key, self._default)
else:
return self.setdefault(key,
self._default[0](*self._default[1], **self._default[2]))

class keytransformdict(dict):
__slots__ = ['_transformer']

def __init__(self, *args, **kwargs):
self._transformer = lambda x: x

super(keytransformdict, self).__init__(self, *args, **kwargs)

def settransformer(self, transformer):
if not callable(transformer):
raise TypeError, 'transformer must be a callable'
self._transformer = transformer

def __setitem__(self, key, value):
super(keytransformdict, self).__setitem__(self,
self._transformer(key), value)

def __getitem__(self, key):
return super(keytransformdict, self).__getitem__(self,
self._transformer(key))

def __delitem__(self, key):
super(keytransformdict, self).__delitem__(self,
self._transformer(key))

class sorteddict(dict):
def __iter__(self):
for key in sorted(super(sorteddict, self).__iter__(self)):
yield key

def keys(self):
return list(self.iterkeys())

def items(self):
return list(self.iteritems())

def values(self):
return list(self.itervalues())

def iterkeys(self):
return iter(self)

def iteritems(self):
return ((key, self[key]) for key in self)

def itervalues(self):
return (self[key] for key in self)

if __name__ == '__main__':
x = sorteddict(a=1, b=3, c=2)

print x.keys()
print x.values()
print x.items()
 
J

Jeff Epler

The software you used to post this message wrapped some of the lines of
code. For example:
def __delitem__(self, key):
super(keytransformdict, self).__delitem__(self,
self._transformer(key))
Click to expand...

In defaultdict, I wonder whether everything should be viewed as a
factory:
def setdefaultvalue(self, value):
def factory(): return value
self.setdefaultfactory(factory)

and the "no-default" mode would either cease to exist, or
def cleardefault(self):
def factory():
raise KeyError, "key does not exist and no default defined"
self.setdefaultfactory(factory)
(too bad that the key isn't available in the factory, this degrades the
quality of the error messge)

if so, __getitem__ becomes simpler:
__slots__ = ['_default']
def __getitem__(self, key):
try:
return super(defaultdict, self).__getitem__(key)
except KeyError:
return self.setdefault(key, apply(*self._default))

I don't ever have an itch for sorted dictionaries, as far as I can
remember, and I don't immediately understand the use of
keytransformdict. Can you give an example of it?

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFCUCb2Jd01MZaTXX0RAujvAJ4mi7cu3l85k0eeTUsd42UDtGVAzACgp1um
6Iq2DhOo529q83oRXqquZ5I=
=Ytjv
-----END PGP SIGNATURE-----
 
G

Georg Brandl

Jeff said:
The software you used to post this message wrapped some of the lines of
code. For example:
Click to expand...

Somehow I feared that this would happen.
In defaultdict, I wonder whether everything should be viewed as a
factory:
def setdefaultvalue(self, value):
def factory(): return value
self.setdefaultfactory(factory)
Click to expand...

That's a reasonable approach. __init__ must be changed too, but this
shouldn't hurt too badly.
and the "no-default" mode would either cease to exist, or
def cleardefault(self):
def factory():
raise KeyError, "key does not exist and no default defined"
self.setdefaultfactory(factory)
(too bad that the key isn't available in the factory, this degrades the
quality of the error messge)
Click to expand...

That can be worked around with a solution in __getitem__, see below.
if so, __getitem__ becomes simpler:
__slots__ = ['_default']
def __getitem__(self, key):
try:
return super(defaultdict, self).__getitem__(key)
except KeyError:
return self.setdefault(key, apply(*self._default))
Click to expand...

You are peculating the kwargs. Also, apply() is on the verge of being
deprecated, so better not use it.

def __getitem__(self, key):
try:
return super(defaultdict, self).__getitem__(key)
except KeyError, err:
try:
return self.setdefault(key,
self._default[0](*self._default[1],
**self._default[2]))
except KeyError:
raise err

Although I'm not sure whether KeyError would be the right one to raise
(perhaps a custom error?).
I don't ever have an itch for sorted dictionaries, as far as I can
remember, and I don't immediately understand the use of
keytransformdict. Can you give an example of it?
Click to expand...

See the thread "Case-insensitive dict, non-destructive, fast, anyone?",
starting at 04/01/05 12:38.

mfg
Georg
 
M

Michael Spencer

Georg said:
Hello,

in follow-up to the recent "dictionary accumulator" thread, I wrote a
little module with several subclassed dicts.

Comments (e.g. makes it sense to use super), corrections, etc.? Is this
PEP material?

Docstrings, Documentation and test cases are to be provided later.

mfg
Georg
Click to expand...

Georg:

A few reactions:

1. Given that these are specializations, why not have:

class defaultvaluedict(dict):
...

class defaultfactorydict(dict):
...

rather than having to jump through hoops to make one implementation satisfy both
cases

2. I would really prefer to have the default value specified in the constructor

I realize that this is tricky due to the kw arguments of dict.__init__, but I
would favor either breaking compatibility with that interface, or adopting some
workaround to make something like d= defaultvaluedict(__default__ = 0) possible.

One worksaround would be to store the default in the dict, not as an attribute
of the dict. By default the default value would be associated with the key
"__default__", but that keyname could be changed for the (I guess very few)
cases where that key conflicted with non-default content of the dict. Then
dict.__init__ would simply take __default__ = value as a keyword argument, as it
does today, and __getitem__ for a missing key would return
dict.__getitem__(self, "__default__")

Alternatively, you could provide factory functions to construct the defaultdict.
Someone (Michele?) recently posted an implementation of this

3. Can you work in the tally and listappend methods that started this whole
thread off?

4. On super, no I don't think it's necessary or particularly desirable. These
specializations have a close association with dict. dict.method(self,...) feels
more appropriate in this case.

Michael
 
G

Georg Brandl

Michael said:
1. Given that these are specializations, why not have:

class defaultvaluedict(dict):
...

class defaultfactorydict(dict):
...

rather than having to jump through hoops to make one implementation satisfy both
cases
Click to expand...

I think I like Jeff's approach more (defaultvalues are just special
cases of default factories); there aren't many "hoops" required.
Apart from that, the names just get longer ;)
2. I would really prefer to have the default value specified in the constructor

I realize that this is tricky due to the kw arguments of dict.__init__, but I
would favor either breaking compatibility with that interface, or adopting some
workaround to make something like d= defaultvaluedict(__default__ = 0) possible.
Click to expand...

Too much specialcased for my liking.
One worksaround would be to store the default in the dict, not as an attribute
of the dict. By default the default value would be associated with the key
"__default__", but that keyname could be changed for the (I guess very few)
cases where that key conflicted with non-default content of the dict. Then
dict.__init__ would simply take __default__ = value as a keyword argument, as it
does today, and __getitem__ for a missing key would return
dict.__getitem__(self, "__default__")
Click to expand...

I thought about this too (providing a singleton instance named Default,
just like None is, and using it as a key), but you would have to
special-case the (iter)keys,values,items methods to exclude the default
- definitely too much work, and too much magic.
Alternatively, you could provide factory functions to construct the defaultdict.
Someone (Michele?) recently posted an implementation of this
Click to expand...

Yes, I think this could be reasonable.
3. Can you work in the tally and listappend methods that started this whole
thread off?
Click to expand...

They aren't necessary any longer.

Use defaultdict.setdefaultvalue(0) instead of the tally approach and
defaultdict.setdefaultfactory(list) instead of listappend.
4. On super, no I don't think it's necessary or particularly desirable. These
specializations have a close association with dict. dict.method(self,...) feels
more appropriate in this case.
Click to expand...

Any other opinions on this?

Thanks for the comments,

mfg
Georg
 
M

Michael Spencer

Georg said:
I think I like Jeff's approach more (defaultvalues are just special
cases of default factories); there aren't many "hoops" required.
Apart from that, the names just get longer ;)
Click to expand...

Yes Jeff's approach does simplify the implementation and more-or-less eliminates
my complexity objection

But why do you write:

def __getitem__(self, key):
try:
return super(defaultdict, self).__getitem__(key)
except KeyError, err:
try:
return self.setdefault(key,
self._default[0](*self._default[1],
**self._default[2]))
except KeyError:
raise err

rather than:

def __getitem__(self, key):
return self.setdefault(key,
self._default[0](*self._default[1],
**self._default[2]))

(which could catch AttributeError in the case of _default not set)
I'm sure there's a reason, but I can't see it.
....

Too much specialcased for my liking.
Click to expand...
It does set up some gotchas I concede ;-)
....


Yes, I think this could be reasonable.
Click to expand...


....though this would more naturally complement a fixed-default dictionary IMO
Your design permits - even encourages (by providing convenient setters) the
default to change over the lifetime of the dictionary. I'm not sure whether
that's good or bad, but it's a feature worth discussing.
They aren't necessary any longer.

Use defaultdict.setdefaultvalue(0) instead of the tally approach and
defaultdict.setdefaultfactory(list) instead of listappend.
Click to expand...

Oops, I see what you mean... then use += or append as required. I still prefer
the clarity of tally for its specific use-case, but it does suffer from lack of
generality.

Michael
 
G

Georg Brandl

Michael said:
Georg said:
I think I like Jeff's approach more (defaultvalues are just special
cases of default factories); there aren't many "hoops" required.
Apart from that, the names just get longer ;)
Click to expand...

Yes Jeff's approach does simplify the implementation and more-or-less eliminates
my complexity objection

But why do you write:

def __getitem__(self, key):
try:
return super(defaultdict, self).__getitem__(key)
except KeyError, err:
try:
return self.setdefault(key,
self._default[0](*self._default[1],
**self._default[2]))
except KeyError:
raise err

rather than:

def __getitem__(self, key):
return self.setdefault(key,
self._default[0](*self._default[1],
**self._default[2]))

(which could catch AttributeError in the case of _default not set)
I'm sure there's a reason, but I can't see it.
Click to expand...

In your version, the default factory is called every time a value is
retrieved, which might be a performance problem.
...though this would more naturally complement a fixed-default dictionary IMO
Your design permits - even encourages (by providing convenient setters) the
default to change over the lifetime of the dictionary. I'm not sure whether
that's good or bad, but it's a feature worth discussing.
Click to expand...

It's certainly more in the spirit of Python -- we're consenting adults,
and so we are allowed to change the default.

mfg
Georg
 
M

Michele Simionato

About not using super: you might have problems in multiple inheritance.
Suppose I want to use both your defaultdict and a thirdpartdict. A
subclass

class mydict(defaultdict, thirdpartdict):
pass

would not work if thirdpartdict requires a non-trivial __init__ , since
without super in defaultdict.__init__ you would just call dict.__init__
and not thirdpartdict.

Michele Simionato
 
G

Georg Brandl

Michele said:
About not using super: you might have problems in multiple inheritance.
Suppose I want to use both your defaultdict and a thirdpartdict. A
subclass

class mydict(defaultdict, thirdpartdict):
pass

would not work if thirdpartdict requires a non-trivial __init__ , since
without super in defaultdict.__init__ you would just call dict.__init__
and not thirdpartdict.
Click to expand...

Right. I thought about a combined defaultdict/keytransformdict,
which seems to be easy to create with the current implementation:

class defaultkeytransformdict(defaultdict, keytransformdict):
pass

At least I hope so. This is another argument against the initializing
of defaultfactory or keytransformer in __init__.

mfg
Georg

Here comes the current module (keytransformdict should be working now
for all dict methods):

# specialdict - subclasses of dict for common tasks
#

class NoDefaultGiven(Exception):
pass

class defaultdict(dict):
__slots__ = ['_default']

def __init__(self, *args, **kwargs):
self._defaulttype = 0

super(defaultdict, self).__init__(*args, **kwargs)

def setdefaultvalue(self, value):
def defaultfactory():
return value
self._default = (defaultfactory, (), {})

def setdefaultfactory(self, factory, *args, **kwargs):
if not callable(factory):
raise TypeError, 'default factory must be a callable'
self._default = (factory, args, kwargs)

def cleardefault(self):
def defaultfactory():
raise NoDefaultGiven
self._default = (defaultfactory, (), {})

def __getitem__(self, key):
try:
return super(defaultdict, self).__getitem__(key)
except KeyError, err:
try:
return self.setdefault(key, self._default[0](*self._default[1], **self._default[2]))
except NoDefaultGiven:
raise err

class keytransformdict(dict):
__slots__ = ['_transformer']

def __init__(self, *args, **kwargs):
self._transformer = lambda x: x

super(keytransformdict, self).__init__(*args, **kwargs)

def settransformer(self, transformer):
if not callable(transformer):
raise TypeError, 'transformer must be a callable'
self._transformer = transformer

def __setitem__(self, key, value):
print "setitem"
super(keytransformdict, self).__setitem__(self._transformer(key), value)

def __getitem__(self, key):
print "getitem"
return super(keytransformdict, self).__getitem__(self._transformer(key))

def __delitem__(self, key):
super(keytransformdict, self).__delitem__(self._transformer(key))

def has_key(self, key):
return super(keytransformdict, self).has_key(self._transformer(key))

def __contains__(self, key):
return self.has_key(key)

def get(self, key, default):
return super(keytransformdict, self).get(self._transformer(key), default)

def setdefault(self, key, default):
return super(keytransformdict, self).setdefault(self._transformer(key), default)

def pop(self, key, default):
return super(keytransformdict, self).pop(self._transfomer(key), default)

def update(self, other=None, **kwargs):
if other is not None:
if hasattr(other, "keys"):
super(keytransformdict, self).update((self._transformer(k), other[k]) for k in other.keys())
else:
super(keytransformdict, self).update((self._transformer(k), v) for (k, v) in other)
if kwargs:
super(keytransformdict, self).update((self._transformer(k), v) for (k, v) in kwargs.iteritems())

class sorteddict(dict):
def __iter__(self):
for key in sorted(super(sorteddict, self).__iter__()):
yield key

def keys(self):
return list(self.iterkeys())

def items(self):
return list(self.iteritems())

def values(self):
return list(self.itervalues())

def iterkeys(self):
return iter(self)

def iteritems(self):
return ((key, self[key]) for key in self)

def itervalues(self):
return (self[key] for key in self)
 
G

Georg Brandl

Georg said:
Michele said:
About not using super: you might have problems in multiple inheritance.
Suppose I want to use both your defaultdict and a thirdpartdict. A
subclass

class mydict(defaultdict, thirdpartdict):
pass

would not work if thirdpartdict requires a non-trivial __init__ , since
without super in defaultdict.__init__ you would just call dict.__init__
and not thirdpartdict.
Click to expand...

Right. I thought about a combined defaultdict/keytransformdict,
which seems to be easy to create with the current implementation:

class defaultkeytransformdict(defaultdict, keytransformdict):
pass

At least I hope so. This is another argument against the initializing
of defaultfactory or keytransformer in __init__.

mfg
Georg

Here comes the current module (keytransformdict should be working now
for all dict methods):

# specialdict - subclasses of dict for common tasks
#

class NoDefaultGiven(Exception):
pass

class defaultdict(dict):
__slots__ = ['_default']

def __init__(self, *args, **kwargs):
self._defaulttype = 0
Click to expand...
^^^^^^^^^^^^^^^^^^^^^

This must read "self.cleardefault()", of course.

mfg
Georg
 
G

Georg Brandl

Georg said:
Hello,

in follow-up to the recent "dictionary accumulator" thread, I wrote a
little module with several subclassed dicts.

Comments (e.g. makes it sense to use super), corrections, etc.? Is this
PEP material?

Docstrings, Documentation and test cases are to be provided later.
Click to expand...

So, no further interest in this? Or should I write a PEP before?

mfg
Georg
 
S

Steven Bethard

Georg said:
So, no further interest in this? Or should I write a PEP before?
Click to expand...

I'd personally like to see a PEP. It'd give people a chance to come up
with some use cases that really justify adding such specialized dicts to
the standard library. The only use cases I can come up with would be
better served by functions that take a sequence and produce a dict (e.g.
by counting items or by grouping items into lists). Once the dict is
built, I usually have no more need for the default-specialized methods.

I'm sure there are some good use cases out there for such dicts, but
before I get behind this, I wouldn't mind seeing a few of them. =)

STeVe
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Issues with writing pytest 0
which "dictionary with attribute-style access"? 8
Bunch 2.0 - a dict with a default 1
1st Sketch at at ReadOnly dict 0
what is the difference between commenting and uncommenting the__init__ method in this class? 2
Use of logging module to track TODOs 0
async fuction 1
Confused about __prepare__ 1

Members online

Total: 25 (members: 1, guests: 24)
Robots: 417

Forum statistics

Threads
473,770
Messages
2,569,584
Members
45,075
Latest member
MakersCBDBloodSupport

Latest Threads

Top