Which objects are expanded by double-star ** operator?

K

kkumer

I have to merge two dictionaries into one, and in
a "shallow" way: changing items should be possible
by operating either on two parents or on a
new dictionary. I am open to suggestions how
to do this (values are always numbers, BTW), but
I tried to do it by creating a dict-like class that just
forwards all calls to the two parent dicts, see below.

It works, but one important thing is missing. I
am not able to expand new dictionary with
double-star operator ** to use it as a
set of keyword arguments of a function.
I googled a bit, but was unable to find what
property must an object have to be correctly
treated by **.

I hope the following code is self-explanatory:

------------------------------------

import itertools

class hubDict(dict):
"""Merges two dictionaries, but not actually but just by forwarding."""

def __init__(self, da, db):
self.d1 = da
self.d2 = db

def __getitem__(self, name):
if self.d1.has_key(name):
return self.d1[name]
else:
return self.d2[name]

def __setitem__(self, name, value):
if self.d1.has_key(name):
self.d1[name] = value
else:
self.d2[name] = value

def __iter__(self):
return itertools.chain(self.d1.__iter__(), self.d2.__iter__())

def has_key(self, name):
if self.d1.has_key(name) or self.d2.has_key(name):
return True
else:
return False

def keys(self):
return self.d1.keys() + self.d2.keys()

def items(self):
return self.d1.items() + self.d2.items()

def iteritems(self):
return itertools.chain(self.d1.iteritems(), self.d2.iteritems())

def iterkeys(self):
return itertools.chain(self.d1.iterkeys(), self.d2.iterkeys())

def itervalues(self):
return itertools.chain(self.d1.itervalues(), self.d2.itervalues())

def copy(self):
print "Can't copy hubDict yet!!!"

def update(self, d):
for key in d:
self.__setitem__(key, d[key])

def popitem(self):
try:
return self.d1.popitem()
except KeyError:
return self.d2.popitem()

def __repr__(self):
return 'First: %s\nSecond: %s' % (
self.d1.__repr__(), self.d2.__repr__())


# Trying it now:

da = {'a':1}
db = {'b':2}
dh = hubDict(da, db)
def kwa(**kwargs): print kwargs

#OK
kwa(**da)

#not OK: prints empty dict
kwa(**dh)


import itertools

class hubDict(dict):
"""Merges two dictionaries, but not actually but just by forwarding."""

def __init__(self, da, db):
self.d1 = da
self.d2 = db

def __getitem__(self, name):
if self.d1.has_key(name):
return self.d1[name]
else:
return self.d2[name]

def __setitem__(self, name, value):
if self.d1.has_key(name):
self.d1[name] = value
else:
self.d2[name] = value

def __iter__(self):
return itertools.chain(self.d1.__iter__(), self.d2.__iter__())

def has_key(self, name):
if self.d1.has_key(name) or self.d2.has_key(name):
return True
else:
return False

def keys(self):
return self.d1.keys() + self.d2.keys()

def items(self):
return self.d1.items() + self.d2.items()

def iteritems(self):
return itertools.chain(self.d1.iteritems(), self.d2.iteritems())

def iterkeys(self):
return itertools.chain(self.d1.iterkeys(), self.d2.iterkeys())

def itervalues(self):
return itertools.chain(self.d1.itervalues(), self.d2.itervalues())

def copy(self):
print "Can't copy hubDict yet!!!"

def update(self, d):
for key in d:
self.__setitem__(key, d[key])

def popitem(self):
try:
return self.d1.popitem()
except KeyError:
return self.d2.popitem()

def __repr__(self):
return 'First: %s\nSecond: %s' % (
self.d1.__repr__(), self.d2.__repr__())

def __len__(self):
return self.d1.__len__() + self.d2.__len__()

# Trying it now:

da = {'a':1}
db = {'b':2}
dh = hubDict(da, db)
def kwa(**kwargs): print kwargs

#OK
kwa(**da)

#not OK: prints empty dict
kwa(**dh)
 
T

Thomas Jollans

I have to merge two dictionaries into one, and in
a "shallow" way: changing items should be possible
by operating either on two parents or on a
new dictionary. I am open to suggestions how
to do this (values are always numbers, BTW), but
I tried to do it by creating a dict-like class that just
forwards all calls to the two parent dicts, see below.

It works, but one important thing is missing. I
am not able to expand new dictionary with
double-star operator ** to use it as a
set of keyword arguments of a function.
I googled a bit, but was unable to find what
property must an object have to be correctly
treated by **.
Click to expand...
My guess would be that this only works with dicts, and uses the internal
representation of the dict, not the python-defined methods.
I don't know what you want to do, but you might be better off creating a
special "mergeabledict" type for the parents, and then allow them to
connect, actually copying all items over. Though it might be best just
to use one single dict ;-)
I hope the following code is self-explanatory:

------------------------------------

import itertools

class hubDict(dict):
"""Merges two dictionaries, but not actually but just by forwarding."""

def __init__(self, da, db):
self.d1 = da
self.d2 = db

def __getitem__(self, name):
if self.d1.has_key(name):
return self.d1[name]
else:
return self.d2[name]

def __setitem__(self, name, value):
if self.d1.has_key(name):
self.d1[name] = value
else:
self.d2[name] = value

def __iter__(self):
return itertools.chain(self.d1.__iter__(), self.d2.__iter__())

def has_key(self, name):
if self.d1.has_key(name) or self.d2.has_key(name):
return True
else:
return False

def keys(self):
return self.d1.keys() + self.d2.keys()

def items(self):
return self.d1.items() + self.d2.items()

def iteritems(self):
return itertools.chain(self.d1.iteritems(), self.d2.iteritems())

def iterkeys(self):
return itertools.chain(self.d1.iterkeys(), self.d2.iterkeys())

def itervalues(self):
return itertools.chain(self.d1.itervalues(), self.d2.itervalues())

def copy(self):
print "Can't copy hubDict yet!!!"

def update(self, d):
for key in d:
self.__setitem__(key, d[key])

def popitem(self):
try:
return self.d1.popitem()
except KeyError:
return self.d2.popitem()

def __repr__(self):
return 'First: %s\nSecond: %s' % (
self.d1.__repr__(), self.d2.__repr__())


# Trying it now:

da = {'a':1}
db = {'b':2}
dh = hubDict(da, db)
def kwa(**kwargs): print kwargs

#OK
kwa(**da)

#not OK: prints empty dict
kwa(**dh)


import itertools

class hubDict(dict):
"""Merges two dictionaries, but not actually but just by forwarding."""

def __init__(self, da, db):
self.d1 = da
self.d2 = db

def __getitem__(self, name):
if self.d1.has_key(name):
return self.d1[name]
else:
return self.d2[name]

def __setitem__(self, name, value):
if self.d1.has_key(name):
self.d1[name] = value
else:
self.d2[name] = value

def __iter__(self):
return itertools.chain(self.d1.__iter__(), self.d2.__iter__())

def has_key(self, name):
if self.d1.has_key(name) or self.d2.has_key(name):
return True
else:
return False

def keys(self):
return self.d1.keys() + self.d2.keys()

def items(self):
return self.d1.items() + self.d2.items()

def iteritems(self):
return itertools.chain(self.d1.iteritems(), self.d2.iteritems())

def iterkeys(self):
return itertools.chain(self.d1.iterkeys(), self.d2.iterkeys())

def itervalues(self):
return itertools.chain(self.d1.itervalues(), self.d2.itervalues())

def copy(self):
print "Can't copy hubDict yet!!!"

def update(self, d):
for key in d:
self.__setitem__(key, d[key])

def popitem(self):
try:
return self.d1.popitem()
except KeyError:
return self.d2.popitem()

def __repr__(self):
return 'First: %s\nSecond: %s' % (
self.d1.__repr__(), self.d2.__repr__())

def __len__(self):
return self.d1.__len__() + self.d2.__len__()

# Trying it now:

da = {'a':1}
db = {'b':2}
dh = hubDict(da, db)
def kwa(**kwargs): print kwargs

#OK
kwa(**da)

#not OK: prints empty dict
kwa(**dh)
Click to expand...
 
I

Ian Kelly

I have to merge two dictionaries into one, and in
a "shallow" way: changing items should be possible
by operating either on two parents or on a
new dictionary. I am open to suggestions how
to do this (values are always numbers, BTW), but
I tried to do it by creating a dict-like class that just
forwards all calls to the two parent dicts, see below.

It works, but one important thing is missing. I
am not able to expand new dictionary with
double-star operator ** to use it as a
set of keyword arguments of a function.
I googled a bit, but was unable to find what
property must an object have to be correctly
treated by **.
Click to expand...

I don't think that what you want to do here is possible. It appears
to be hard-coded and not affected by subclassing, as evidenced by the
following snippet:

class NonDict(dict):
for attr in dict.__dict__:
if attr not in ('__init__', '__new__',):
locals()[attr] = None

d = NonDict({'a': 1})
def kwa(**kw): print kw
kwa(**d)

Prints:
{'a': 1}

Cheers,
Ian
 
D

Dave Angel

kkumer said:
I have to merge two dictionaries into one, and in
a "shallow" way: changing items should be possible
by operating either on two parents or on a
new dictionary. I am open to suggestions how
to do this (values are always numbers, BTW), but
I tried to do it by creating a dict-like class that just
forwards all calls to the two parent dicts, see below.

It works, but one important thing is missing. I
am not able to expand new dictionary with
double-star operator ** to use it as a
set of keyword arguments of a function.
I googled a bit, but was unable to find what
property must an object have to be correctly
treated by **.

I hope the following code is self-explanatory:

------------------------------------

import itertools

class hubDict(dict):
"""Merges two dictionaries, but not actually but just by forwarding."""

def __init__(self, da, db):
self.d1 = da
self.d2 = db

def __getitem__(self, name):
if self.d1.has_key(name):
return self.d1[name]
else:
return self.d2[name]

def __setitem__(self, name, value):
if self.d1.has_key(name):
self.d1[name] = value
else:
self.d2[name] = value

def __iter__(self):
return itertools.chain(self.d1.__iter__(), self.d2.__iter__())

def has_key(self, name):
if self.d1.has_key(name) or self.d2.has_key(name):
return True
else:
return False

def keys(self):
return self.d1.keys() + self.d2.keys()

def items(self):
return self.d1.items() + self.d2.items()

def iteritems(self):
return itertools.chain(self.d1.iteritems(), self.d2.iteritems())

def iterkeys(self):
return itertools.chain(self.d1.iterkeys(), self.d2.iterkeys())

def itervalues(self):
return itertools.chain(self.d1.itervalues(), self.d2.itervalues())

def copy(self):
print "Can't copy hubDict yet!!!"

def update(self, d):
for key in d:
self.__setitem__(key, d[key])

def popitem(self):
try:
return self.d1.popitem()
except KeyError:
return self.d2.popitem()

def __repr__(self):
return 'First: %s\nSecond: %s' % (
self.d1.__repr__(), self.d2.__repr__())


# Trying it now:

da = {'a':1}
db = {'b':2}
dh = hubDict(da, db)
def kwa(**kwargs): print kwargs

#OK
kwa(**da)

#not OK: prints empty dict
kwa(**dh)


import itertools

class hubDict(dict):
"""Merges two dictionaries, but not actually but just by forwarding."""

def __init__(self, da, db):
self.d1 = da
self.d2 = db

def __getitem__(self, name):
if self.d1.has_key(name):
return self.d1[name]
else:
return self.d2[name]

def __setitem__(self, name, value):
if self.d1.has_key(name):
self.d1[name] = value
else:
self.d2[name] = value

def __iter__(self):
return itertools.chain(self.d1.__iter__(), self.d2.__iter__())

def has_key(self, name):
if self.d1.has_key(name) or self.d2.has_key(name):
return True
else:
return False

def keys(self):
return self.d1.keys() + self.d2.keys()

def items(self):
return self.d1.items() + self.d2.items()

def iteritems(self):
return itertools.chain(self.d1.iteritems(), self.d2.iteritems())

def iterkeys(self):
return itertools.chain(self.d1.iterkeys(), self.d2.iterkeys())

def itervalues(self):
return itertools.chain(self.d1.itervalues(), self.d2.itervalues())

def copy(self):
print "Can't copy hubDict yet!!!"

def update(self, d):
for key in d:
self.__setitem__(key, d[key])

def popitem(self):
try:
return self.d1.popitem()
except KeyError:
return self.d2.popitem()

def __repr__(self):
return 'First: %s\nSecond: %s' % (
self.d1.__repr__(), self.d2.__repr__())

def __len__(self):
return self.d1.__len__() + self.d2.__len__()

# Trying it now:

da = {'a':1}
db = {'b':2}
dh = hubDict(da, db)
def kwa(**kwargs): print kwargs

#OK
kwa(**da)

#not OK: prints empty dict
kwa(**dh)
Click to expand...
From the help on "Container types",
provides a DictMixin class to help create those methods from a base set
of __getitem__() <#object.__getitem__>, __setitem__()
<#object.__setitem__>, __delitem__() <#object.__delitem__>, and keys().
<<

and__contains__() <#object.__contains__> method to allow efficient use of
the in operator; for mappings, in should be equivalent of has_key(); for
sequences, it should search through the values. It is further
recommended that both mappings and sequences implement the __iter__()
<#object.__iter__> method to allow efficient iteration through the
container; for mappings, __iter__() <#object.__iter__> should be the
same as iterkeys(); for sequences, it should iterate through the values.
<<


I'd guess that the ** operator calls one or more of these special
methods directly, rather than calling the equivalent regular methods.

DaveA
 
P

Peter Otten

kkumer said:
I have to merge two dictionaries into one, and in
a "shallow" way: changing items should be possible
by operating either on two parents or on a
new dictionary. I am open to suggestions how
to do this (values are always numbers, BTW), but
I tried to do it by creating a dict-like class that just
forwards all calls to the two parent dicts, see below.

It works, but one important thing is missing. I
am not able to expand new dictionary with
double-star operator ** to use it as a
set of keyword arguments of a function.
I googled a bit, but was unable to find what
property must an object have to be correctly
treated by **.
Click to expand...

The following experiment shows that you only need to implement a keys() and
__getitem__() method.

$ cat kw.py
class A(object):
def keys(self): return list("ab")
def __getitem__(self, key):
return 42

def f(**kw):
print(kw)

f(**A())
$ python kw.py
{'a': 42, 'b': 42}

However, if you have A inherit from dict...

$ cat kwd.py
class A(dict):
def keys(self): return list("ab")
def __getitem__(self, key):
return 42

def f(**kw):
print(kw)

f(**A())
$ python kwd.py
{}

it stops working -- probably a side-effect of some optimization.
So if you change your hubDict's base class from dict to object you should
get the desired behaviour.

Peter
 
T

Terry Reedy

The answer depends on the version:
in 2.5 "If the syntax "**expression" appears in the function call,
"expression" must evaluate to a (subclass of) dictionary, "
in 3.1 "If the syntax **expression appears in the function call,
expression must evaluate to a mapping,"

ReferenceManual.Epressions.Primaries.Calls

Terry Jan Reedy
 
I

Ian Kelly

The following experiment shows that you only need to implement a keys() and
__getitem__() method.

$ cat kw.py
class A(object):
   def keys(self): return list("ab")
   def __getitem__(self, key):
       return 42

def f(**kw):
   print(kw)

f(**A())
$ python kw.py
{'a': 42, 'b': 42}
Click to expand...

This seems to require Python 2.6. With 2.5, you get:

Traceback (most recent call last):
File "kw.py", line 9, in <module>
f(**A())
TypeError: f() argument after ** must be a dictionary

Cheers,
Ian
 
T

Terry Reedy

The following experiment shows that you only need to implement a keys() and
__getitem__() method.

$ cat kw.py
class A(object):
def keys(self): return list("ab")
def __getitem__(self, key):
return 42

def f(**kw):
print(kw)

f(**A())
$ python kw.py
{'a': 42, 'b': 42}

However, if you have A inherit from dict...

$ cat kwd.py
class A(dict):
def keys(self): return list("ab")
def __getitem__(self, key):
return 42

def f(**kw):
print(kw)

f(**A())
$ python kwd.py
{}

it stops working -- probably a side-effect of some optimization.
So if you change your hubDict's base class from dict to object you should
get the desired behaviour.
Click to expand...

In 2.6, the requirement changed from '(subclass of) dictionary' to
'mapping' so this is a bit strange. It sort of looks like a bug. I will
test with 3.1 tomorrow (later today, actually).

tjr
 
K

Kresimir Kumericki

Peter Otten said:
The following experiment shows that you only need to implement a keys() and
__getitem__() method.

$ cat kw.py
class A(object):
def keys(self): return list("ab")
def __getitem__(self, key):
return 42

def f(**kw):
print(kw)

f(**A())
$ python kw.py
{'a': 42, 'b': 42}

However, if you have A inherit from dict...

$ cat kwd.py
class A(dict):
def keys(self): return list("ab")
def __getitem__(self, key):
return 42

def f(**kw):
print(kw)

f(**A())
$ python kwd.py
{}

it stops working -- probably a side-effect of some optimization.
So if you change your hubDict's base class from dict to object you should
get the desired behaviour.

Peter
Click to expand...
 
B

Bryan

Terry said:
Peter said:
kkumer wrote:
Click to expand...
The following experiment shows that you only need to implement a keys() and
__getitem__() method.
Click to expand...
$ cat kw.py
class A(object):
     def keys(self): return list("ab")
     def __getitem__(self, key):
         return 42 [...]
However, if you have A inherit from dict... [...]
it stops working -- probably a side-effect of some optimization.
So if you change your hubDict's base class from dict to object you should
get the desired behaviour.
Click to expand...

In 2.6, the requirement changed from '(subclass of) dictionary' to
'mapping' so this is a bit strange. It sort of looks like a bug. I will
test with 3.1 tomorrow (later today, actually).
Click to expand...

I get the same bug-like behavior in 3.1. I think Peter is right that
it's probably a side-effect of an optimization. kkumer seems to have
completely over-ridden the methods of dict, but if we insert into his
hubDict with the parent class's method:

dict.__setitem__(dh, 'c', 3)

Then the **dh argument passes the keyword arg c=3.
 
K

kkumer

(sorry for posting empty post by accident)

Peter Otten said:
it stops working -- probably a side-effect of some optimization.
So if you change your hubDict's base class from dict to object you should
get the desired behaviour.
Click to expand...

Yes, as already noted, this would require python >=1.6, and I am
reluctant to upgrade at this point.

What I ended up doing is to simply create a normal dictionary
and copy all items from my hubDict merged "dictionary" into it
just for the purpose of this one function call

auxdict = dict((it for it in hubDictinstance.items()))
f(**auxdict)

Since this happens only few times during runtime it's not a big deal.

Thanks to all for insights.

K.
 
K

kkumer

Bryan said:
I get the same bug-like behavior in 3.1. I think Peter is right that
it's probably a side-effect of an optimization. kkumer seems to have
completely over-ridden the methods of dict, but if we insert into his
hubDict with the parent class's method:

dict.__setitem__(dh, 'c', 3)

Then the **dh argument passes the keyword arg c=3.
Click to expand...

Yes. But problem is that this setting of item
should also affect one of the two original parent dicts,
while your proposal affects only hubDict instance.
(hubDict is never used to create new items, just to change
values of existing ones, which belong to one of
two parents).

K.
 
S

Steven D'Aprano

I don't think that what you want to do here is possible. It appears to
be hard-coded and not affected by subclassing, as evidenced by the
following snippet:

class NonDict(dict):
for attr in dict.__dict__:
if attr not in ('__init__', '__new__',):
locals()[attr] = None
Click to expand...


I'm afraid your test is invalid. As the documentation states, you CANNOT
write to locals() -- the change doesn't stick. This is nothing to do with
dicts.
.... x = 1
.... locals()['x'] = 2
.... print x
....1
 
B

Bryan

kkumer said:
Yes. But problem is that this setting of item
should also affect one of the two original parent dicts,
while  your proposal affects only hubDict instance.
(hubDict is never used to create new items, just to change
values of existing ones, which belong to one of
two parents).
Click to expand...

Right. This is helpful group, but my observation here does not solve
your immediate problem. Your work-around seems reasonable, and Peter
Otten's demo of what happens if you inherit from object rather than
dict might make **dh work as you want.
 
I

Ian Kelly

I don't think that what you want to do here is possible.  It appears to
be hard-coded and not affected by subclassing, as evidenced by the
following snippet:

class NonDict(dict):
    for attr in dict.__dict__:
        if attr not in ('__init__', '__new__',):
            locals()[attr] = None
Click to expand...


I'm afraid your test is invalid. As the documentation states, you CANNOT
write to locals() -- the change doesn't stick. This is nothing to do with
dicts.
Click to expand...

Huh, good point. But actually the docs just say that the changes
aren't guaranteed to stick, and in the case of my test I inspected the
class and the methods were indeed replaced with None. In fact, when I
first wrote it, I didn't include the if statement and got a TypeError
when I tried to construct an instance.

Cheers,
Ian
 
S

Steven D'Aprano

Huh, good point. But actually the docs just say that the changes aren't
guaranteed to stick, and in the case of my test I inspected the class
and the methods were indeed replaced with None.
Click to expand...

So they are. I've just learned something new, thanks.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

frozendict (v0.1) 16
which "dictionary with attribute-style access"? 8
Simple Python implementation of bag/multiset 1
1st Sketch at at ReadOnly dict 0
"specialdict" module 12
dictionnaries and lookup tables 6
cheetah templating: place holders needed in a template 0
caseless dict - questions 2

Members online

No members online now.
Total: 25 (members: 1, guests: 24)
Robots: 439

Forum statistics

Threads
473,770
Messages
2,569,583
Members
45,073
Latest member
DarinCeden

Latest Threads

Top