Extending the dict class

[chosechu]

Hello Pythoneers:

I need to pass a list of named arguments to a function in a given
order,
and make sure these named arguments are retrieved using keys() in the
same order they were given. Example:

keyargs={}
keyargs['one']=1
keyargs['two']=2
keyargs['three']=3

myfunc(**keyargs)
-> myfunc would retrieve key arguments with keys() in the same order
as they were set, i.e. keyargs.keys() == ['one', 'two', 'three']

To achieve that, I subclassed dict and added the required lines
in __init__(), __setitem__() and keys(). I then assigned dict to
my new class but only get the desired behaviour for dictionaries
created like:
d=dict()
but not for dictionaries created like:
d={}
or
myfunc(**keyargs)

Is it possible to force dictionary creation in these case to use
my own dict class instead of the default one?

I guess we can formulate this as a more generic question: if I
want to modify the behaviour of the dictionary class, is there
any way to do it interpreter-wide so that special dict constructors
like those mentioned above use the modified version?

Thanks for helping

 

[Duncan Booth]

Is it possible to force dictionary creation in these case to use
my own dict class instead of the default one?
No

I guess we can formulate this as a more generic question: if I
want to modify the behaviour of the dictionary class, is there
any way to do it interpreter-wide so that special dict constructors
like those mentioned above use the modified version?

Not without modifying and recompiling the interpreter.

 

[chosechu]

No

Ouch. I was expecting something like that, thanks for confirming it.

If I may: this seems inconsistent to me. I have created an augmented
version of the class with no possibility to extend 2 of the 3 legal
constructors: {} and func(**d). While I agree changing base class
behaviour is a bad idea, since we have the possibility to re-assign
dict to another class we should be able to do it consistently, i.e.
re-assing the two special constructors too.

Just my 2c on the topic... I have found workarounds anyway.

 

[Bruno Desthuilliers]

Hello Pythoneers:

I need to pass a list of named arguments to a function in a given
order,
and make sure these named arguments are retrieved using keys() in the
same order they were given. Example:

keyargs={}
keyargs['one']=1
keyargs['two']=2
keyargs['three']=3

myfunc(**keyargs)
-> myfunc would retrieve key arguments with keys() in the same order
as they were set, i.e. keyargs.keys() == ['one', 'two', 'three']

I'm not sure to understand why you want to do so - perhaps you could
tell more about your real use case ?

Anyway, and since it's not directly possible, a possible workaround
could be to pass a sequence of (key, value) tuples instead (possibly as
*args). This of course requires that you can modify the implementation
of myfunc().

 

[chosechu]

I'm not sure to understand why you want to do so - perhaps you could
tell more about your real use case ?

Without diving into too many details: I am calling SOAPpy to build
SOAP requests. When calling a proxy, named arguments are used
to build up the corresponding XML like:

proxy.call(alpha=1, beta=2, gamma=3)
would end up like:
<alpha>1</alpha> <beta>2</beta> <gamma>3</gamma>

Unfortunately, since the arguments as retrieved with **k
their order is lost in the call. The corresponding XML file
re-arranges the parameter order randomly and the final
XML request is unordered, which is not supported by a
number of braindead SOAP servers (including one produced
by a well-known software company I will not name here).

SOAPpy cannot know in advance the argument names since
they are server-dependent and SOAPpy is generic, so retrieving
named arguments with **k seems like the sensible thing to do.
Unfortunately this gets converted to a dict so looses all ordering
when being received. Extending the base dict class to support ordering
was one possible idea.

Anyway, and since it's not directly possible, a possible workaround
could be to pass a sequence of (key, value) tuples instead (possibly as
*args). This of course requires that you can modify the implementation
of myfunc().

Yes, if I could simply modify myfunc() I would have workarounds.
This would mean me modifying SOAPpy and specializing it for
my needs.

There are other ways to implement SOAP clients anyway so
no real need. Just wanted to try out some metaclass stuff.

 

[Duncan Booth]

SOAPpy cannot know in advance the argument names since
they are server-dependent and SOAPpy is generic, so retrieving
named arguments with **k seems like the sensible thing to do.
Unfortunately this gets converted to a dict so looses all ordering
when being received. Extending the base dict class to support ordering
was one possible idea.

If the order of the argument names matters, then it seems to me that should
be handled by the SOAP library, not pushed onto the end user whoch should
just be calling the function with the named arguments in the most
convenient order.

Shouldn't SOAPpy be able to get this information out of the WSDL?

 

[Fredrik Lundh]

Yes, if I could simply modify myfunc() I would have workarounds.
This would mean me modifying SOAPpy and specializing it for
my needs.

maybe you could fake it:

class fakedict(dict):
def __init__(self, *data):
self.data = list(data)
for k, v in data:
self[k] = v
def items(self):
return self.data

d = fakedict(("a", 1), ("b", 2), ("c", 3))

print d => {'a': 1, 'c': 3, 'b': 2}
print d.items() => [('a', 1), ('b', 2), ('c', 3)]
print isinstance(d, dict) => True

(the exact set of methods you need to override depends on how SOAPpy
fetches the members).

</F>

 

[chosechu]

If the order of the argument names matters, then it seems to me that should
be handled by the SOAP library, not pushed onto the end user whoch should
just be calling the function with the named arguments in the most
convenient order.

Shouldn't SOAPpy be able to get this information out of the WSDL?

Yes, SOAPpy could extract this from the WSDL specs.
SOAPpy could also find another way to pass call parameters, but I kinda
like
the named parameters (seems really Python-like). Microsoft could also
build SOAP services that parse XML without making ordering mandatory
where
nobody said it was.

.... but we are living in a different dimension, one where I can extend
the Python dict class but not touch 2 of its 3 constructors (of course
the most
useful ones).

If you really want to know, I ended up at the socket level pushing
templatized
strings. Wish me luck with future server changes.

 

[chosechu]

(the exact set of methods you need to override depends on how SOAPpy
fetches the members).

[fakedict class]
This I did. And checking out the source it seems SOAPpy retrieves named
parameters through keys(), which is the method I tried to overload.
Trouble is:
something happens to my fakedict object when getting through this:
d = fakedict(...)
SOAPmethod(**d)
I have the impression **d acts like a dict constructor, taking over my
fakedict class. I know I could find the answer by scrutinizing the
source,
call me lazy.

 

[Duncan Booth]

Yes, SOAPpy could extract this from the WSDL specs.
SOAPpy could also find another way to pass call parameters, but I
kinda like
the named parameters (seems really Python-like). Microsoft could also
build SOAP services that parse XML without making ordering mandatory
where
nobody said it was.

Indeed, the spec says that parameterOrder is a hint and may be safely
ignored.

... but we are living in a different dimension, one where I can extend
the Python dict class but not touch 2 of its 3 constructors (of course
the most
useful ones).

No, you weren't able to extend the builtin dict class nor touch any its
constructor.

All you did was to create a subclass with its own constructor and hide the
name for the builtin dictionary type. The original type was still unchanged
as you can see since anything which constructed a dictionary without using
the name you had overwritten still got the original type.

If you had looked at type(dict()) and type({}) after your subclassing, you
would see that they are different types.

 

[chosechu]

No, you weren't able to extend the builtin dict class nor touch any its
constructor.

Yes, sorry. Forgot the negation.

All you did was to create a subclass with its own constructor and hide the
name for the builtin dictionary type. The original type was still unchanged
as you can see since anything which constructed a dictionary without using
the name you had overwritten still got the original type.

If you had looked at type(dict()) and type({}) after your subclassing, you
would see that they are different types.

.... which prompted my question.

And prompts yet another one: seems like it
is not possible with Python to modify behaviour
for base classes without recompiling the
interpreter. Forgive me for asking what must
surely have been asked already, but are there
plans to implement something like that,
<teasing>like Ruby</teasing>?

I would not feel too safe navigating in a source
where base object behaviour might have been
re-defined, but it sure is a powerful way of
adding behaviour to third-party code which you
may not have possibility to modify.

 

[Tim N. van der Leeuw]

maybe you could fake it:

No you cannot fake it, because the ** form of passing arguments
constructs a new dictionary (and that dictionary is the standard
built-in type of dictionary, not your own fake-dictionary).

Just did some tests, redefining the dict-type:
.... pass
........ print type(kwargs) == native_dict
....
(NB: I also tried changing __builtins__.dict, no effect)

As the OP said, when passing keyword-arguments, a new instance of a
built-in dict is always generated and not the dict type bound to the
dict-constructor.

I do actually think that it would be useful in many circumstances to be
able to replace the default-constructor of a particular so that custom
objects are created for that type.

OTOH, there are probably many more cases where doing so would be a very
bad idea, not a good idea, and we would begin seeing an overwhelming
number of cases of mis-use of such feature.

Cheers,

--Tim

 

[Tim N. van der Leeuw]

Yes, SOAPpy could extract this from the WSDL specs.
SOAPpy could also find another way to pass call parameters, but I kinda
like
the named parameters (seems really Python-like). Microsoft could also
build SOAP services that parse XML without making ordering mandatory
where
nobody said it was.

Are you sure that the order is not defined in the WSDL still? I don't
know WSDL, but in XML Schema for instance, order is defined when using
'xsd:sequence' yet many parsers don't care for receiving elements in a
different order from what's defined in the XSD which they're supposed
to validate against (working perhaps with the axiom 'be liberal in what
yo except, but strict in what you send'). Many people don't actually
realize that 'xsd:sequence' dictates the order of elements and use it
just for 'any order is good enough'.

Might be a similar case with the WSDL: Order is defined, but due to
mis-interpretation of the spec it's not realized / enforced by all /
some of the components.
Anyways, modifiying SOAPpy might not be a bad idea: submit your changes
to the project, and you can write on your CV that you have contributed
to open-source projects! ;-)

I do hope for you that you will find something better to do than
pushing template-instances down a socket-hole.

Cheers,

--Tim

 

[Fredrik Lundh]

No you cannot fake it, because the ** form of passing arguments
constructs a new dictionary (and that dictionary is the standard
built-in type of dictionary, not your own fake-dictionary).

yeah, I thought he was passing a dictionary to a SOAPpy method. passing
custom types through the argument handling layer doesn't work.

</F>

 

[chosechu]

Anyways, modifiying SOAPpy might not be a bad idea: submit your changes
to the project, and you can write on your CV that you have contributed
to open-source projects! ;-)

Been there, done that. Don't worry, my contributions
to open-source projects is largely positive (though under
different aliases).

About SOAPpy: the author mentions something about
ordering (or not) parameters in the XML request. Quoting:

"There is an incompatibility with the way that Python and
SOAP handle method arguments: SOAP requires that
all arguments have names and that they are presented
in the same order as the method signature. [...]"

See SOAPpy/docs/MethodParameterNaming.txt for more
details.

I do hope for you that you will find something better to do than
pushing template-instances down a socket-hole.

It is not the first time I end up implementing some kind of
support for ASCII-based protocols with templatized strings
You would be surprised how easy it is to maintain when
the emitter is required to be as stable as possible. Just
have to be aware that you have a potential maintenance
nightmare somewhere out there.

 

[Bruno Desthuilliers]

Yes, sorry. Forgot the negation.


... which prompted my question.

And prompts yet another one: seems like it
is not possible with Python to modify behaviour
for base classes without recompiling the
interpreter. Forgive me for asking what must
surely have been asked already, but are there
plans to implement something like that,
<teasing>like Ruby</teasing>?

I would not feel too safe navigating in a source
where base object behaviour might have been
re-defined, but it sure is a powerful way of
adding behaviour to third-party code which you
may not have possibility to modify.

It's usually possible to modify third-parts classes behaviour on the fly
(googling for 'monkey-patching' should get you started). But true, this
doesn't work with builtins.

 

[chosechu]

It's usually possible to modify third-parts classes behaviour on the fly
(googling for 'monkey-patching' should get you started). But true, this
doesn't work with builtins.

I guess this stems from the fact that Python dictionaries
are C creatures and they are themselves used for internal
housekeeping, so no high-level modifications can be allowed
to disturb language internals. This all makes sense. It is sad
not to have the possibility to extend built-ins but on the other
hand it guarantees a constant (predictable) behaviour, which
I agree is something we should protect.

Thanks to all contributors for your nearly instantaneous
reaction times and help on the topic.