Extending the dict class

Discussion in 'Python' started by chosechu, Aug 29, 2006.

  1. chosechu

    chosechu Guest

    Hello Pythoneers:

    I need to pass a list of named arguments to a function in a given
    order,
    and make sure these named arguments are retrieved using keys() in the
    same order they were given. Example:

    keyargs={}
    keyargs['one']=1
    keyargs['two']=2
    keyargs['three']=3

    myfunc(**keyargs)
    -> myfunc would retrieve key arguments with keys() in the same order
    as they were set, i.e. keyargs.keys() == ['one', 'two', 'three']

    To achieve that, I subclassed dict and added the required lines
    in __init__(), __setitem__() and keys(). I then assigned dict to
    my new class but only get the desired behaviour for dictionaries
    created like:
    d=dict()
    but not for dictionaries created like:
    d={}
    or
    myfunc(**keyargs)

    Is it possible to force dictionary creation in these case to use
    my own dict class instead of the default one?

    I guess we can formulate this as a more generic question: if I
    want to modify the behaviour of the dictionary class, is there
    any way to do it interpreter-wide so that special dict constructors
    like those mentioned above use the modified version?

    Thanks for helping
    chosechu, Aug 29, 2006
    #1
    1. Advertising

  2. chosechu

    Duncan Booth Guest

    chosechu wrote:

    > Is it possible to force dictionary creation in these case to use
    > my own dict class instead of the default one?


    No

    > I guess we can formulate this as a more generic question: if I
    > want to modify the behaviour of the dictionary class, is there
    > any way to do it interpreter-wide so that special dict constructors
    > like those mentioned above use the modified version?


    Not without modifying and recompiling the interpreter.
    Duncan Booth, Aug 29, 2006
    #2
    1. Advertising

  3. chosechu

    chosechu Guest

    Duncan Booth wrote:
    > > Is it possible to force dictionary creation in these case to use
    > > my own dict class instead of the default one?

    >
    > No


    Ouch. I was expecting something like that, thanks for confirming it.

    If I may: this seems inconsistent to me. I have created an augmented
    version of the class with no possibility to extend 2 of the 3 legal
    constructors: {} and func(**d). While I agree changing base class
    behaviour is a bad idea, since we have the possibility to re-assign
    dict to another class we should be able to do it consistently, i.e.
    re-assing the two special constructors too.

    Just my 2c on the topic... I have found workarounds anyway.
    chosechu, Aug 29, 2006
    #3
  4. chosechu wrote:
    > Hello Pythoneers:
    >
    > I need to pass a list of named arguments to a function in a given
    > order,
    > and make sure these named arguments are retrieved using keys() in the
    > same order they were given. Example:
    >
    > keyargs={}
    > keyargs['one']=1
    > keyargs['two']=2
    > keyargs['three']=3
    >
    > myfunc(**keyargs)
    > -> myfunc would retrieve key arguments with keys() in the same order
    > as they were set, i.e. keyargs.keys() == ['one', 'two', 'three']


    I'm not sure to understand why you want to do so - perhaps you could
    tell more about your real use case ?

    Anyway, and since it's not directly possible, a possible workaround
    could be to pass a sequence of (key, value) tuples instead (possibly as
    *args). This of course requires that you can modify the implementation
    of myfunc().


    --
    bruno desthuilliers
    python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
    p in ''.split('@')])"
    Bruno Desthuilliers, Aug 29, 2006
    #4
  5. chosechu

    chosechu Guest


    > I'm not sure to understand why you want to do so - perhaps you could
    > tell more about your real use case ?


    Without diving into too many details: I am calling SOAPpy to build
    SOAP requests. When calling a proxy, named arguments are used
    to build up the corresponding XML like:

    proxy.call(alpha=1, beta=2, gamma=3)
    would end up like:
    <alpha>1</alpha> <beta>2</beta> <gamma>3</gamma>

    Unfortunately, since the arguments as retrieved with **k
    their order is lost in the call. The corresponding XML file
    re-arranges the parameter order randomly and the final
    XML request is unordered, which is not supported by a
    number of braindead SOAP servers (including one produced
    by a well-known software company I will not name here).

    SOAPpy cannot know in advance the argument names since
    they are server-dependent and SOAPpy is generic, so retrieving
    named arguments with **k seems like the sensible thing to do.
    Unfortunately this gets converted to a dict so looses all ordering
    when being received. Extending the base dict class to support ordering
    was one possible idea.

    > Anyway, and since it's not directly possible, a possible workaround
    > could be to pass a sequence of (key, value) tuples instead (possibly as
    > *args). This of course requires that you can modify the implementation
    > of myfunc().


    Yes, if I could simply modify myfunc() I would have workarounds.
    This would mean me modifying SOAPpy and specializing it for
    my needs.

    There are other ways to implement SOAP clients anyway so
    no real need. Just wanted to try out some metaclass stuff.
    chosechu, Aug 29, 2006
    #5
  6. chosechu

    Duncan Booth Guest

    chosechu wrote:

    > SOAPpy cannot know in advance the argument names since
    > they are server-dependent and SOAPpy is generic, so retrieving
    > named arguments with **k seems like the sensible thing to do.
    > Unfortunately this gets converted to a dict so looses all ordering
    > when being received. Extending the base dict class to support ordering
    > was one possible idea.


    If the order of the argument names matters, then it seems to me that should
    be handled by the SOAP library, not pushed onto the end user whoch should
    just be calling the function with the named arguments in the most
    convenient order.

    Shouldn't SOAPpy be able to get this information out of the WSDL?
    Duncan Booth, Aug 29, 2006
    #6
  7. "chosechu" wrote:

    > Yes, if I could simply modify myfunc() I would have workarounds.
    > This would mean me modifying SOAPpy and specializing it for
    > my needs.


    maybe you could fake it:

    class fakedict(dict):
    def __init__(self, *data):
    self.data = list(data)
    for k, v in data:
    self[k] = v
    def items(self):
    return self.data

    d = fakedict(("a", 1), ("b", 2), ("c", 3))

    print d => {'a': 1, 'c': 3, 'b': 2}
    print d.items() => [('a', 1), ('b', 2), ('c', 3)]
    print isinstance(d, dict) => True

    (the exact set of methods you need to override depends on how SOAPpy
    fetches the members).

    </F>
    Fredrik Lundh, Aug 29, 2006
    #7
  8. chosechu

    chosechu Guest

    Duncan Booth wrote:
    > If the order of the argument names matters, then it seems to me that should
    > be handled by the SOAP library, not pushed onto the end user whoch should
    > just be calling the function with the named arguments in the most
    > convenient order.
    >
    > Shouldn't SOAPpy be able to get this information out of the WSDL?


    Yes, SOAPpy could extract this from the WSDL specs.
    SOAPpy could also find another way to pass call parameters, but I kinda
    like
    the named parameters (seems really Python-like). Microsoft could also
    build SOAP services that parse XML without making ordering mandatory
    where
    nobody said it was.

    .... but we are living in a different dimension, one where I can extend
    the Python dict class but not touch 2 of its 3 constructors (of course
    the most
    useful ones).

    If you really want to know, I ended up at the socket level pushing
    templatized
    strings. Wish me luck with future server changes.
    chosechu, Aug 29, 2006
    #8
  9. chosechu

    chosechu Guest


    > (the exact set of methods you need to override depends on how SOAPpy
    > fetches the members).


    [fakedict class]
    This I did. And checking out the source it seems SOAPpy retrieves named
    parameters through keys(), which is the method I tried to overload.
    Trouble is:
    something happens to my fakedict object when getting through this:
    d = fakedict(...)
    SOAPmethod(**d)
    I have the impression **d acts like a dict constructor, taking over my
    fakedict class. I know I could find the answer by scrutinizing the
    source,
    call me lazy.
    chosechu, Aug 29, 2006
    #9
  10. chosechu

    Duncan Booth Guest

    chosechu wrote:

    >
    > Duncan Booth wrote:
    >> If the order of the argument names matters, then it seems to me that
    >> should be handled by the SOAP library, not pushed onto the end user
    >> whoch should just be calling the function with the named arguments in
    >> the most convenient order.
    >>
    >> Shouldn't SOAPpy be able to get this information out of the WSDL?

    >
    > Yes, SOAPpy could extract this from the WSDL specs.
    > SOAPpy could also find another way to pass call parameters, but I
    > kinda like
    > the named parameters (seems really Python-like). Microsoft could also
    > build SOAP services that parse XML without making ordering mandatory
    > where
    > nobody said it was.


    Indeed, the spec says that parameterOrder is a hint and may be safely
    ignored.

    >
    > ... but we are living in a different dimension, one where I can extend
    > the Python dict class but not touch 2 of its 3 constructors (of course
    > the most
    > useful ones).


    No, you weren't able to extend the builtin dict class nor touch any its
    constructor.

    All you did was to create a subclass with its own constructor and hide the
    name for the builtin dictionary type. The original type was still unchanged
    as you can see since anything which constructed a dictionary without using
    the name you had overwritten still got the original type.

    If you had looked at type(dict()) and type({}) after your subclassing, you
    would see that they are different types.
    Duncan Booth, Aug 29, 2006
    #10
  11. chosechu

    chosechu Guest

    Duncan Booth wrote:
    > No, you weren't able to extend the builtin dict class nor touch any its
    > constructor.


    Yes, sorry. Forgot the negation.

    > All you did was to create a subclass with its own constructor and hide the
    > name for the builtin dictionary type. The original type was still unchanged
    > as you can see since anything which constructed a dictionary without using
    > the name you had overwritten still got the original type.
    >
    > If you had looked at type(dict()) and type({}) after your subclassing, you
    > would see that they are different types.


    .... which prompted my question.

    And prompts yet another one: seems like it
    is not possible with Python to modify behaviour
    for base classes without recompiling the
    interpreter. Forgive me for asking what must
    surely have been asked already, but are there
    plans to implement something like that,
    <teasing>like Ruby</teasing>?

    I would not feel too safe navigating in a source
    where base object behaviour might have been
    re-defined, but it sure is a powerful way of
    adding behaviour to third-party code which you
    may not have possibility to modify.
    chosechu, Aug 29, 2006
    #11
  12. Fredrik Lundh wrote:
    > "chosechu" wrote:
    >
    > > Yes, if I could simply modify myfunc() I would have workarounds.
    > > This would mean me modifying SOAPpy and specializing it for
    > > my needs.

    >
    > maybe you could fake it:
    >

    No you cannot fake it, because the ** form of passing arguments
    constructs a new dictionary (and that dictionary is the standard
    built-in type of dictionary, not your own fake-dictionary).

    Just did some tests, redefining the dict-type:

    >>> native_dict = dict
    >>> type({}) == native_dict

    True
    >>> class my_dict(native_dict):

    .... pass
    ....
    >>> dict = my_dict
    >>> type({}) == native_dict

    True
    >>> type(dict()) == native_dict

    False
    >>> def foo(**kwargs):

    .... print type(kwargs) == native_dict
    ....
    >>> foo(x=1)

    True
    >>> foo(**dict())

    True
    >>>


    (NB: I also tried changing __builtins__.dict, no effect)

    As the OP said, when passing keyword-arguments, a new instance of a
    built-in dict is always generated and not the dict type bound to the
    dict-constructor.

    I do actually think that it would be useful in many circumstances to be
    able to replace the default-constructor of a particular so that custom
    objects are created for that type.

    OTOH, there are probably many more cases where doing so would be a very
    bad idea, not a good idea, and we would begin seeing an overwhelming
    number of cases of mis-use of such feature.

    Cheers,

    --Tim
    Tim N. van der Leeuw, Aug 29, 2006
    #12
  13. chosechu wrote:
    > Duncan Booth wrote:
    > > If the order of the argument names matters, then it seems to me that should
    > > be handled by the SOAP library, not pushed onto the end user whoch should
    > > just be calling the function with the named arguments in the most
    > > convenient order.
    > >
    > > Shouldn't SOAPpy be able to get this information out of the WSDL?

    >
    > Yes, SOAPpy could extract this from the WSDL specs.
    > SOAPpy could also find another way to pass call parameters, but I kinda
    > like
    > the named parameters (seems really Python-like). Microsoft could also
    > build SOAP services that parse XML without making ordering mandatory
    > where
    > nobody said it was.
    >

    Are you sure that the order is not defined in the WSDL still? I don't
    know WSDL, but in XML Schema for instance, order is defined when using
    'xsd:sequence' yet many parsers don't care for receiving elements in a
    different order from what's defined in the XSD which they're supposed
    to validate against (working perhaps with the axiom 'be liberal in what
    yo except, but strict in what you send'). Many people don't actually
    realize that 'xsd:sequence' dictates the order of elements and use it
    just for 'any order is good enough'.

    Might be a similar case with the WSDL: Order is defined, but due to
    mis-interpretation of the spec it's not realized / enforced by all /
    some of the components.
    Anyways, modifiying SOAPpy might not be a bad idea: submit your changes
    to the project, and you can write on your CV that you have contributed
    to open-source projects! ;-)

    I do hope for you that you will find something better to do than
    pushing template-instances down a socket-hole.

    Cheers,

    --Tim
    Tim N. van der Leeuw, Aug 29, 2006
    #13
  14. Tim N. van der Leeuw wrote:

    >> maybe you could fake it:
    >>

    > No you cannot fake it, because the ** form of passing arguments
    > constructs a new dictionary (and that dictionary is the standard
    > built-in type of dictionary, not your own fake-dictionary).


    yeah, I thought he was passing a dictionary to a SOAPpy method. passing
    custom types through the argument handling layer doesn't work.

    </F>
    Fredrik Lundh, Aug 29, 2006
    #14
  15. chosechu

    chosechu Guest

    Tim N. van der Leeuw wrote:
    > Anyways, modifiying SOAPpy might not be a bad idea: submit your changes
    > to the project, and you can write on your CV that you have contributed
    > to open-source projects! ;-)


    Been there, done that. Don't worry, my contributions
    to open-source projects is largely positive (though under
    different aliases).

    About SOAPpy: the author mentions something about
    ordering (or not) parameters in the XML request. Quoting:

    "There is an incompatibility with the way that Python and
    SOAP handle method arguments: SOAP requires that
    all arguments have names and that they are presented
    in the same order as the method signature. [...]"

    See SOAPpy/docs/MethodParameterNaming.txt for more
    details.

    > I do hope for you that you will find something better to do than
    > pushing template-instances down a socket-hole.


    It is not the first time I end up implementing some kind of
    support for ASCII-based protocols with templatized strings
    You would be surprised how easy it is to maintain when
    the emitter is required to be as stable as possible. Just
    have to be aware that you have a potential maintenance
    nightmare somewhere out there.
    chosechu, Aug 29, 2006
    #15
  16. chosechu wrote:
    > Duncan Booth wrote:
    >> No, you weren't able to extend the builtin dict class nor touch any its
    >> constructor.

    >
    > Yes, sorry. Forgot the negation.
    >
    >> All you did was to create a subclass with its own constructor and hide the
    >> name for the builtin dictionary type. The original type was still unchanged
    >> as you can see since anything which constructed a dictionary without using
    >> the name you had overwritten still got the original type.
    >>
    >> If you had looked at type(dict()) and type({}) after your subclassing, you
    >> would see that they are different types.

    >
    > ... which prompted my question.
    >
    > And prompts yet another one: seems like it
    > is not possible with Python to modify behaviour
    > for base classes without recompiling the
    > interpreter. Forgive me for asking what must
    > surely have been asked already, but are there
    > plans to implement something like that,
    > <teasing>like Ruby</teasing>?
    >
    > I would not feel too safe navigating in a source
    > where base object behaviour might have been
    > re-defined, but it sure is a powerful way of
    > adding behaviour to third-party code which you
    > may not have possibility to modify.


    It's usually possible to modify third-parts classes behaviour on the fly
    (googling for 'monkey-patching' should get you started). But true, this
    doesn't work with builtins.

    --
    bruno desthuilliers
    python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
    p in ''.split('@')])"
    Bruno Desthuilliers, Aug 29, 2006
    #16
  17. chosechu

    chosechu Guest

    Bruno Desthuilliers wrote:
    > It's usually possible to modify third-parts classes behaviour on the fly
    > (googling for 'monkey-patching' should get you started). But true, this
    > doesn't work with builtins.


    I guess this stems from the fact that Python dictionaries
    are C creatures and they are themselves used for internal
    housekeeping, so no high-level modifications can be allowed
    to disturb language internals. This all makes sense. It is sad
    not to have the possibility to extend built-ins but on the other
    hand it guarantees a constant (predictable) behaviour, which
    I agree is something we should protect.

    Thanks to all contributors for your nearly instantaneous
    reaction times and help on the topic.
    chosechu, Aug 29, 2006
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Skip Montanaro
    Replies:
    0
    Views:
    412
    Skip Montanaro
    Aug 15, 2003
  2. Alexander Kozlovsky

    dict!ident as equivalent of dict["ident"]

    Alexander Kozlovsky, May 21, 2006, in forum: Python
    Replies:
    5
    Views:
    357
    Alexander Kozlovsky
    May 22, 2006
  3. Paul Melis

    dict.has_key(x) versus 'x in dict'

    Paul Melis, Dec 6, 2006, in forum: Python
    Replies:
    48
    Views:
    1,319
    Kent Johnson
    Dec 15, 2006
  4. Ravi
    Replies:
    3
    Views:
    275
  5. Ravi
    Replies:
    4
    Views:
    427
    sturlamolden
    Mar 7, 2011
Loading...

Share This Page