default behavior

Discussion in 'Python' started by wheres pythonmonks, Jul 29, 2010.

  1. Why is the default value of an int zero?

    >>> x = int
    >>> print x

    <type 'int'>
    >>> x()

    0
    >>>


    How do I build an "int1" type that has a default value of 1?
    [Hopefully no speed penalty.]
    I am thinking about applications with collections.defaultdict.
    What if I want to make a defaultdict of defaultdicts of lists? [I
    guess my Perl background is showing -- I miss auto-vivification.]

    W
     
    wheres pythonmonks, Jul 29, 2010
    #1
    1. Advertising

  2. wheres pythonmonks

    Paul Rubin Guest

    wheres pythonmonks <> writes:
    > How do I build an "int1" type that has a default value of 1?
    > [Hopefully no speed penalty.]
    > I am thinking about applications with collections.defaultdict.


    You can supply an arbitary function to collections.defaultdict.
    It doesn't have to be a class. E.g.

    d = collections.defaultdict(lambda: 1)

    will do what you are asking.
     
    Paul Rubin, Jul 29, 2010
    #2
    1. Advertising

  3. Thanks. I presume this will work for my nested example as well. Thanks again.

    On Thu, Jul 29, 2010 at 2:18 PM, Paul Rubin <> wrote:
    > wheres pythonmonks <> writes:
    >> How do I build an "int1" type that has a default value of 1?
    >> [Hopefully no speed penalty.]
    >> I am thinking about applications with collections.defaultdict.

    >
    > You can supply an arbitary function to collections.defaultdict.
    > It doesn't have to be a class.  E.g.
    >
    >    d = collections.defaultdict(lambda: 1)
    >
    > will do what you are asking.
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >
     
    wheres pythonmonks, Jul 29, 2010
    #3
  4. wheres pythonmonks

    John Nagle Guest

    On 7/29/2010 11:12 AM, wheres pythonmonks wrote:
    > Why is the default value of an int zero?
    >
    >>>> x = int
    >>>> print x

    > <type 'int'>
    >>>> x()

    > 0
    >>>>

    >
    > How do I build an "int1" type that has a default value of 1?



    >>> class int1(object) :

    .... def __init__(self) :
    .... self.val = 1
    .... def __call__(self) :
    .... return(self.val)
    ....
    >>> x = int1()
    >>> x()

    1

    This isn't useful; you'd also have to define all the numeric operators
    for this type. And then there are mixed-type conversion issues.

    Inheriting from "int" is not too helpful, because you can't assign
    to the value of the base class. "self=1" won't do what you want.

    [Hopefully no speed penalty.]
    In your dreams. Although all numbers in CPython are "boxed",
    so there's more of a speed penalty with "int" itself than you
    might expect. There are some C libraries for handling large
    arrays if you really need to crunch numbers.

    John Nagle
     
    John Nagle, Jul 29, 2010
    #4
  5. wheres pythonmonks

    Peter Otten Guest

    wheres pythonmonks wrote:

    > How do I build an "int1" type that has a default value of 1?
    > [Hopefully no speed penalty.]
    > I am thinking about applications with collections.defaultdict.


    >>> from collections import defaultdict
    >>> d = defaultdict(1 .conjugate)
    >>> d["x"] += 2
    >>> d["x"]

    3

    Isn't that beautiful? Almost like home;)

    It is also fast:

    $ python -m timeit -s"one = lambda: 1" "one()"
    1000000 loops, best of 3: 0.213 usec per loop
    $ python -m timeit -s"one = 1 .conjugate" "one()"
    10000000 loops, best of 3: 0.0972 usec per loop

    Micro-optimisation, the best excuse for ugly code...

    Peter
     
    Peter Otten, Jul 30, 2010
    #5
  6. wheres pythonmonks

    Peter Otten Guest

    Duncan Booth wrote:

    > Peter Otten <> wrote:
    >
    >>>>> from collections import defaultdict
    >>>>> d = defaultdict(1 .conjugate)
    >>>>> d["x"] += 2
    >>>>> d["x"]

    >> 3
    >>
    >> Isn't that beautiful? Almost like home;)
    >>
    >> It is also fast:
    >>
    >> $ python -m timeit -s"one = lambda: 1" "one()"
    >> 1000000 loops, best of 3: 0.213 usec per loop
    >> $ python -m timeit -s"one = 1 .conjugate" "one()"
    >> 10000000 loops, best of 3: 0.0972 usec per loop
    >>
    >> Micro-optimisation, the best excuse for ugly code...
    >>

    >
    > Nice one, but if you are going to micro-optimise why not save a few
    > keystrokes while you're at it and use '1 .real' instead?


    >>> 1 .real

    1
    >>> 1 .conjugate

    <built-in method conjugate of int object at 0x1734298>
    >>> 1 .conjugate()


    real is a property, not a method. conjugate() was the first one that worked
    that was not __special__. I think it has the added benefit that it's likely
    to confuse the reader...

    Peter
     
    Peter Otten, Jul 30, 2010
    #6
  7. Instead of defaultdict for hash of lists, I have seen something like:


    m={}; m.setdefault('key', []).append(1)

    Would this be preferred in some circumstances?
    Also, is there a way to upcast a defaultdict into a dict? I have also
    heard some people use exceptions on dictionaries to catch key
    existence, so passing in a defaultdict (I guess) could be hazardous to
    health. Is this true?

    W




    On Fri, Jul 30, 2010 at 6:56 AM, Duncan Booth
    <> wrote:
    > Peter Otten <> wrote:
    >> real is a property, not a method. conjugate() was the first one that
    >> worked that was not __special__. I think it has the added benefit that
    >> it's likely to confuse the reader...
    >>

    > Ah, silly me, I should have realised that.
    >
    > Yes, micro-optimisations that are also micro-obfuscations are always the
    > best. :^)
    >
    > --
    > Duncan Booth http://kupuguy.blogspot.com
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >
     
    wheres pythonmonks, Jul 30, 2010
    #7
  8. On Fri, 30 Jul 2010 07:59:52 -0400, wheres pythonmonks wrote:

    > Instead of defaultdict for hash of lists, I have seen something like:
    >
    >
    > m={}; m.setdefault('key', []).append(1)
    >
    > Would this be preferred in some circumstances?


    Sure, why not? Whichever you prefer.

    setdefault() is a venerable old technique, dating back to Python 2.0, and
    not a newcomer like defaultdict.


    > Also, is there a way to upcast a defaultdict into a dict?


    "Upcast"? Surely it is downcasting. Or side-casting. Or type-casting.
    Whatever. *wink*

    Whatever it is, the answer is Yes:

    >>> from collections import defaultdict as dd
    >>> x = dd(int)
    >>> x[1] = 'a'
    >>> x

    defaultdict(<type 'int'>, {1: 'a'})
    >>> dict(x)

    {1: 'a'}



    > I have also heard some people use
    > exceptions on dictionaries to catch key existence, so passing in a
    > defaultdict (I guess) could be hazardous to health. Is this true?


    Yes, it is true that some people use exceptions on dicts to catch key
    existence. The most common reason to do so is to catch the non-existence
    of a key so you can add it:

    try:
    mydict[x] = mydict[x] + 1
    except KeyError:
    mydict[x] = 1


    If mydict is a defaultdict with the appropriate factory, then the change
    is perfectly safe because mydict[x] will not raise an exception when x is
    missing, but merely return 0, so it will continue to work as expected and
    all is good.

    Of course, if you pass it an defaultdict with an *inappropriate* factory,
    you'll get an error. So don't do that :) Seriously, you can't expect to
    just randomly replace a variable with some arbitrarily different variable
    and expect it to work. You need to know what the code is expecting, and
    not break those expectations too badly.

    And now you have at least three ways of setting missing values in a dict.
    And those wacky Perl people say that Python's motto is "only one way to
    do it" :)



    --
    Steven
     
    Steven D'Aprano, Jul 30, 2010
    #8
  9. wheres pythonmonks

    Peter Otten Guest

    wheres pythonmonks wrote:

    > Instead of defaultdict for hash of lists, I have seen something like:
    >
    >
    > m={}; m.setdefault('key', []).append(1)
    >
    > Would this be preferred in some circumstances?


    In some circumstances, sure. I just can't think of them at the moment.
    Maybe if your code has to work in Python 2.4.

    > Also, is there a way to upcast a defaultdict into a dict?


    dict(some_defaultdict)

    > I have also
    > heard some people use exceptions on dictionaries to catch key
    > existence, so passing in a defaultdict (I guess) could be hazardous to
    > health. Is this true?


    A problem could arise when you swap a "key in dict" test with a
    "try...except KeyError". This would be an implementation detail for a dict
    but affect the contents of a defaultdict:

    >>> from collections import defaultdict
    >>> def update(d):

    .... for c in "abc":
    .... try: d[c]
    .... except KeyError: d[c] = c
    ....
    >>> d = defaultdict(lambda:"-")
    >>> update(d)
    >>> d

    defaultdict(<function <lambda> at 0x7fd4ce32a320>, {'a': '-', 'c': '-', 'b':
    '-'})
    >>> def update2(d):

    .... for c in "abc":
    .... if c not in d:
    .... d[c] = c
    ....
    >>> d = defaultdict(lambda:"-")
    >>> update2(d)
    >>> d

    defaultdict(<function <lambda> at 0x7fd4ce32a6e0>, {'a': 'a', 'c': 'c', 'b':
    'b'})

    Peter
     
    Peter Otten, Jul 30, 2010
    #9
  10. On Fri, 30 Jul 2010 08:34:52 -0400, wheres pythonmonks wrote:

    > Sorry, doesn't the following make a copy?
    >
    >>>>> from collections import defaultdict as dd x = dd(int)
    >>>>> x[1] = 'a'
    >>>>> x

    >> defaultdict(<type 'int'>, {1: 'a'})
    >>>>> dict(x)

    >> {1: 'a'}
    >>
    >>
    >>

    >
    > I was hoping not to do that -- e.g., actually reuse the same underlying
    > data.



    It does re-use the same underlying data.

    >>> from collections import defaultdict as dd
    >>> x = dd(list)
    >>> x[1].append(1)
    >>> x

    defaultdict(<type 'list'>, {1: [1]})
    >>> y = dict(x)
    >>> x[1].append(42)
    >>> y

    {1: [1, 42]}

    Both the defaultdict and the dict are referring to the same underlying
    key:value pairs. The data itself isn't duplicated. If they are mutable
    items, a change to one will affect the other (because they are the same
    item). An analogy for C programmers would be that creating dict y from
    dict y merely copies the pointers to the keys and values, it doesn't copy
    the data being pointed to.

    (That's pretty much what the CPython implementation does. Other
    implementations may do differently, so long as the visible behaviour
    remains the same.)



    > Maybe dict(x), where x is a defaultdict is smart? I agree that a
    > defaultdict is safe to pass to most routines, but I guess I could
    > imagine that a try/except block is used in a bit of code where on the
    > key exception (when the value is absent) populates the value with a
    > random number. In that application, a defaultdict would have no random
    > values.


    If you want a defaultdict with a random default value, it is easy to
    provide:

    >>> import random
    >>> z = dd(random.random)
    >>> z[2] += 0
    >>> z

    defaultdict(<built-in method random of Random object at 0xa01e4ac>, {2:
    0.30707092626033605})


    The point which I tried to make, but obviously failed, is that any piece
    of code has certain expectations about the data it accepts. If take a
    function that expects an int between -2 and 99, and instead decide to
    pass a Decimal between 100 and 150, then you'll have problems: if you're
    lucky, you'll get an exception, if you're unlucky, it will silently give
    the wrong results. Changing a dict to a defaultdict is no different.

    If you have code that *relies* on getting a KeyError for missing keys:

    def who_is_missing(adict):
    for person in ("Fred", "Barney", "Wilma", "Betty"):
    try:
    adict[person]
    except KeyError:
    print person, "is missing"

    then changing adict to a defaultdict will cause the function to
    misbehave. That's not unique to dicts and defaultdicts.



    > Besides a slightly different favor, does the following have applications
    > not covered by defaultdict?
    >
    > m.setdefault('key', []).append(1)


    defaultdict calls a function of no arguments to provide a default value.
    That means, in practice, it almost always uses the same default value for
    any specific dict.

    setdefault takes an argument when you call the function. So you can
    provide anything you like at runtime.


    > I think I am unclear on the difference between that and:
    >
    > m['key'] = m.get('key',[]).append(1)


    Have you tried it? I guess you haven't, or you wouldn't have thought they
    did the same thing.

    Hint -- what does [].append(1) return?


    --
    Steven
     
    Steven D'Aprano, Jul 31, 2010
    #10
  11. >
    > Hint -- what does [].append(1) return?
    >


    Again, apologies from a Python beginner. It sure seems like one has
    to do gymnastics to get good behavior out of the core-python:

    Here's my proposed fix:

    m['key'] = (lambda x: x.append(1) or x)(m.get('key',[]))

    Yuck! So I guess I'll use defaultdict with upcasts to dict as needed.

    On a side note: does up-casting always work that way with shared
    (common) data from derived to base? (I mean if the data is part of
    base's interface, will b = base(child) yield a new base object that
    shares data with the child?)

    Thanks again from a Perl-to-Python convert!

    W


    On Fri, Jul 30, 2010 at 11:47 PM, Steven D'Aprano
    <> wrote:
    > On Fri, 30 Jul 2010 08:34:52 -0400, wheres pythonmonks wrote:
    >
    >> Sorry, doesn't the following make a copy?
    >>
    >>>>>> from collections import defaultdict as dd x = dd(int)
    >>>>>> x[1] = 'a'
    >>>>>> x
    >>> defaultdict(<type 'int'>, {1: 'a'})
    >>>>>> dict(x)
    >>> {1: 'a'}
    >>>
    >>>
    >>>

    >>
    >> I was hoping not to do that -- e.g., actually reuse the same underlying
    >> data.

    >
    >
    > It does re-use the same underlying data.
    >
    >>>> from collections import defaultdict as dd
    >>>> x = dd(list)
    >>>> x[1].append(1)
    >>>> x

    > defaultdict(<type 'list'>, {1: [1]})
    >>>> y = dict(x)
    >>>> x[1].append(42)
    >>>> y

    > {1: [1, 42]}
    >
    > Both the defaultdict and the dict are referring to the same underlying
    > key:value pairs. The data itself isn't duplicated. If they are mutable
    > items, a change to one will affect the other (because they are the same
    > item). An analogy for C programmers would be that creating dict y from
    > dict y merely copies the pointers to the keys and values, it doesn't copy
    > the data being pointed to.
    >
    > (That's pretty much what the CPython implementation does. Other
    > implementations may do differently, so long as the visible behaviour
    > remains the same.)
    >
    >
    >
    >> Maybe dict(x), where x is a defaultdict is smart?  I agree that a
    >> defaultdict is safe to pass to most routines, but I guess I could
    >> imagine that a try/except block is used in a bit of code where on the
    >> key exception (when the value is absent)  populates the value with a
    >> random number.  In that application, a defaultdict would have no random
    >> values.

    >
    > If you want a defaultdict with a random default value, it is easy to
    > provide:
    >
    >>>> import random
    >>>> z = dd(random.random)
    >>>> z[2] += 0
    >>>> z

    > defaultdict(<built-in method random of Random object at 0xa01e4ac>, {2:
    > 0.30707092626033605})
    >
    >
    > The point which I tried to make, but obviously failed, is that any piece
    > of code has certain expectations about the data it accepts. If take a
    > function that expects an int between -2 and 99, and instead decide to
    > pass a Decimal between 100 and 150, then you'll have problems: if you're
    > lucky, you'll get an exception, if you're unlucky, it will silently give
    > the wrong results. Changing a dict to a defaultdict is no different.
    >
    > If you have code that *relies* on getting a KeyError for missing keys:
    >
    > def who_is_missing(adict):
    >    for person in ("Fred", "Barney", "Wilma", "Betty"):
    >        try:
    >            adict[person]
    >        except KeyError:
    >            print person, "is missing"
    >
    > then changing adict to a defaultdict will cause the function to
    > misbehave. That's not unique to dicts and defaultdicts.
    >
    >
    >
    >> Besides a slightly different favor, does the following have applications
    >> not covered by defaultdict?
    >>
    >> m.setdefault('key', []).append(1)

    >
    > defaultdict calls a function of no arguments to provide a default value.
    > That means, in practice, it almost always uses the same default value for
    > any specific dict.
    >
    > setdefault takes an argument when you call the function. So you can
    > provide anything you like at runtime.
    >
    >
    >> I think I am unclear on the difference between that and:
    >>
    >> m['key'] = m.get('key',[]).append(1)

    >
    > Have you tried it? I guess you haven't, or you wouldn't have thought they
    > did the same thing.
    >
    > Hint -- what does [].append(1) return?
    >
    >
    > --
    > Steven
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >
     
    wheres pythonmonks, Jul 31, 2010
    #11
  12. On Sat, 31 Jul 2010 01:02:47 -0400, wheres pythonmonks wrote:


    >> Hint -- what does [].append(1) return?
    >>
    >>

    > Again, apologies from a Python beginner. It sure seems like one has to
    > do gymnastics to get good behavior out of the core-python:
    >
    > Here's my proposed fix:
    >
    > m['key'] = (lambda x: x.append(1) or x)(m.get('key',[]))
    >
    > Yuck!


    Yuk is right. What's wrong with the simple, straightforward solution?

    L = m.get('key', [])
    L.append(1)
    m['key'] = L


    Not everything needs to be a one-liner. But if you insist on making it a
    one-liner, that's what setdefault and defaultdict are for.



    > So I guess I'll use defaultdict with upcasts to dict as needed.


    You keep using that term "upcast". I have no idea what you think it
    means, so I have no idea whether or not Python does it. Perhaps you
    should explain what you think "upcasting" is.


    > On a side note: does up-casting always work that way with shared
    > (common) data from derived to base? (I mean if the data is part of
    > base's interface, will b = base(child) yield a new base object that
    > shares data with the child?)


    Of course not. It depends on the implementation of the class.


    --
    Steven
     
    Steven D'Aprano, Jul 31, 2010
    #12
  13. I think of an upcast as casting to the base-class (casting up the
    inheritance tree).
    http://en.wiktionary.org/wiki/upcast
    But really, what I am thinking of doing is overriding the virtual
    methods of a derived class with the base class behavior in an object
    that I can then pass into methods that are base/derived agnostic.

    defaultdict is the way to go.

    W

    <ps>
    <rant>

    Sadly, there are guidelines that I program by that are perhaps anti-pythonic:

    1. Don't use "extra" variables in code. Don't use global variables.
    Keep the scopes of local variables at a minimum to reduce state (the
    exception being for inner loops) or variables explicitly identified as
    part of the algorithm before implementation. [In python, just about
    everything is a variable which is terrifying to me. I never want the
    Alabama version of math.pi i.e.,
    http://www.snopes.com/religion/pi.asp, or math.sin being "666".]

    2. Use built-in functions/features as much as possible, as this are
    the most tested. Don't roll your own -- you're not that good, instead
    master the language. (How often do I invent a noun in English? Not
    even "upcast"!) [Plus, guys with phds probably already did what you
    need.] Use only very well known libraries -- numpy is okay (I hope!)
    for example. An exception can be made while interfacing external
    data, because others who create data may not have abided by rule #2.
    In most cases (except gui programming, which again tackles the
    external interfacing program) the more heavy-weight your API, the more
    wrong you are.

    3. In interpreted languages, avoid function calls, unless the
    function does something significant. [e.g., Functional call overhead
    tends to be worse that a dictionary lookup -- and yes I used timeit,
    the overhead can be 100%.] Small functions and methods (and
    callbacks) hamper good interpreted code. When writing functions, make
    them operate on lists/dicts.

    It is because of the above that I stopped writing object-oriented Perl.

    So I want "big" functions that do a lot of work with few variable
    names. Ideally, I'd create only variables that are relevant based on
    the description of the algorithm. [Oh yeah, real programming is done
    before the implementation in python or C++.]

    My problems are compounded by the lack of indention-based scope, but I
    see this as simply enforcing the full use of functional-programming
    approaches.

    </rant>
    </ps>

    On Sat, Jul 31, 2010 at 5:55 AM, Steven D'Aprano
    <> wrote:
    > On Sat, 31 Jul 2010 01:02:47 -0400, wheres pythonmonks wrote:
    >
    >
    >>> Hint -- what does [].append(1) return?
    >>>
    >>>

    >> Again, apologies from a Python beginner.  It sure seems like one has to
    >> do gymnastics to get good behavior out of the core-python:
    >>
    >> Here's my proposed fix:
    >>
    >>  m['key'] = (lambda x: x.append(1) or x)(m.get('key',[]))
    >>
    >> Yuck!

    >
    > Yuk is right. What's wrong with the simple, straightforward solution?
    >
    > L = m.get('key', [])
    > L.append(1)
    > m['key'] = L
    >
    >
    > Not everything needs to be a one-liner. But if you insist on making it a
    > one-liner, that's what setdefault and defaultdict are for.
    >
    >
    >
    >> So I guess I'll use defaultdict with upcasts to dict as needed.

    >
    > You keep using that term "upcast". I have no idea what you think it
    > means, so I have no idea whether or not Python does it. Perhaps you
    > should explain what you think "upcasting" is.
    >
    >
    >> On a side note:  does up-casting always work that way with shared
    >> (common) data from derived to base?  (I mean if the data is part of
    >> base's interface, will  b = base(child) yield a new base object that
    >> shares data with the child?)

    >
    > Of course not. It depends on the implementation of the class.
    >
    >
    > --
    > Steven
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >
     
    wheres pythonmonks, Jul 31, 2010
    #13
  14. wheres pythonmonks

    John Posner Guest

    On 7/31/2010 1:31 PM, John Posner wrote:
    >
    > Caveat -- there's another description of defaultdict here:
    >
    > http://docs.python.org/library/collections.html#collections.defaultdict
    >
    > ... and it's bogus. This other description claims that __missing__ is a
    > method of defaultdict, not of dict.


    Following is a possible replacement for the bogus description. Comments
    welcome. I intend to submit a Python doc bug, and I'd like to have a
    clean alternative to propose.

    --------------

    class collections.defaultdict([default_factory[, ...]])

    defaultdict is a dict subclass that can guarantee success on key
    lookups: if a key does not currently exist in a defaultdict object, a
    "default value factory" is called to provide a value for that key. The
    "default value factory" is a callable object (typically, a function)
    that takes no arguments. You specify this callable as the first argument
    to defaultdict(). Additional defaultdict() arguments are the same as for
    dict().

    The "default value factory" callable is stored as an attribute,
    default_factory, of the newly created defaultdict object. If you call
    defaultdict() with no arguments, or with None as the first argument, the
    default_factory attribute is set to None. You can reassign the
    default_factory attribute of an existing defaultdict object to another
    callable, or to None.

    When a lookup of a non-existent key is performed in a defaultdict
    object, its default_factory attribute is evaluated, and the resulting
    object is called:

    * If the call produces a value, that value is returned as the result of
    the lookup. In addition, the key-value pair is inserted into the
    defaultdict.

    * If the call raises an exception, it is propagated unchanged.

    * If the default_factory attribute evaluates to None, a KeyError
    exception is raised, with the non-existent key as its argument. (The
    defaultdict behaves exactly like a standard dict in this case.)
     
    John Posner, Aug 3, 2010
    #14
  15. wheres pythonmonks

    Ethan Furman Guest

    John Posner wrote:
    > On 7/31/2010 1:31 PM, John Posner wrote:
    >>
    >> Caveat -- there's another description of defaultdict here:
    >>
    >> http://docs.python.org/library/collections.html#collections.defaultdict
    >>
    >> ... and it's bogus. This other description claims that __missing__ is a
    >> method of defaultdict, not of dict.

    >
    > Following is a possible replacement for the bogus description. Comments
    > welcome. I intend to submit a Python doc bug, and I'd like to have a
    > clean alternative to propose.
    >
    > --------------
    >
    > class collections.defaultdict([default_factory[, ...]])
    >
    > defaultdict is a dict subclass that can guarantee success on key
    > lookups: if a key does not currently exist in a defaultdict object, a
    > "default value factory" is called to provide a value for that key. The
    > "default value factory" is a callable object (typically, a function)
    > that takes no arguments. You specify this callable as the first argument
    > to defaultdict(). Additional defaultdict() arguments are the same as for
    > dict().
    >
    > The "default value factory" callable is stored as an attribute,
    > default_factory, of the newly created defaultdict object. If you call
    > defaultdict() with no arguments, or with None as the first argument, the
    > default_factory attribute is set to None. You can reassign the
    > default_factory attribute of an existing defaultdict object to another
    > callable, or to None.
    >
    > When a lookup of a non-existent key is performed in a defaultdict
    > object, its default_factory attribute is evaluated, and the resulting
    > object is called:
    >
    > * If the call produces a value, that value is returned as the result of
    > the lookup. In addition, the key-value pair is inserted into the
    > defaultdict.
    >
    > * If the call raises an exception, it is propagated unchanged.
    >
    > * If the default_factory attribute evaluates to None, a KeyError
    > exception is raised, with the non-existent key as its argument. (The
    > defaultdict behaves exactly like a standard dict in this case.)


    I think mentioning how __missing__ plays into all this would be helpful.
    Perhaps in the first paragraph, after the colon:

    if a key does not currently exist in a defaultdict object, __missing__
    will be called with that key, which in turn will call a "default value
    factory" to provide a value for that key.

    ~Ethan~
     
    Ethan Furman, Aug 3, 2010
    #15
  16. wheres pythonmonks

    Ethan Furman Guest

    John Posner wrote:
    > On 7/31/2010 1:31 PM, John Posner wrote:
    >>
    >> Caveat -- there's another description of defaultdict here:
    >>
    >> http://docs.python.org/library/collections.html#collections.defaultdict
    >>
    >> ... and it's bogus. This other description claims that __missing__ is a
    >> method of defaultdict, not of dict.


    __missing__ isn't a method of dict:

    --> print dir(dict())
    ['__class__', '__cmp__', '__contains__', '__delattr__', '__delitem__',
    '__doc__', '__eq__', '__ge__', '__getattribute__', '__getitem__',
    '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__len__',
    '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__',
    '__repr__', '__setattr__', '__setitem__', '__str__', 'clear', 'copy',
    'fromkeys', 'get', 'has_key', 'items', 'iteritems', 'iterkeys',
    'itervalues', 'keys', 'pop', 'popitem', 'setdefault', 'update',
    'values']

    I will agree that the current defaultdict description does not make it
    clear that __missing__ can be defined for *any* subclass of dict,
    although the dict description does go over this... is that the confusion
    you are talking about? If not, could you explain?

    ~Ethan~
     
    Ethan Furman, Aug 3, 2010
    #16
  17. wheres pythonmonks

    Ethan Furman Guest

    Christian Heimes wrote:
    >> I just went and read the entry that had the bogus claim -- personally, I
    >> didn't see any confusion. I would like to point out the __missing__ is
    >> *not* part of dicts (tested on 2.5 and 2.6 -- don't have 2.7 installed yet).

    >
    > I beg your pardon but you are wrong. __missing__ is available for all
    > *subclasses* of dict since Python 2.5. See
    > http://svn.python.org/view/python/b...jects/dictobject.c?revision=81031&view=markup
    >
    >>>> class mydict(dict):

    > ... def __missing__(self, key):
    > ... print "__missing__", key
    > ... raise KeyError(key)
    > ...
    >>>> m = mydict()
    >>>> m[1]

    > __missing__ 1
    > Traceback (most recent call last):
    > File "<stdin>", line 1, in <module>
    > File "<stdin>", line 4, in __missing__
    > KeyError: 1


    Perhaps punctuation will help clarify my intent:

    __missing__ is *not* part of (dict)s, as shown by dir(dict()):

    ['__class__', '__cmp__', '__contains__', '__delattr__', '__delitem__',
    '__doc__', '__eq__', '__ge__', '__getattribute__', '__getitem__',
    '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__len__',
    '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__',
    '__repr__', '__setattr__', '__setitem__', '__str__', 'clear', 'copy',
    'fromkeys', 'get', 'has_key', 'items', 'iteritems', 'iterkeys',
    'itervalues', 'keys', 'pop', 'popitem', 'setdefault', 'update',
    'values']

    And, just to state what is hopefully obvious, if you don't create
    __missing__ yourself, it still isn't in the subclass:

    --> class somedict(dict):
    .... "Is __missing__ defined if I don't define it? Nope."
    ....
    --> sd = somedict()
    --> sd[1]
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    KeyError: 1
    --> dir(sd)
    ['__class__', '__cmp__', '__contains__', '__delattr__', '__delitem__',
    '__dict__', '__doc__', '__eq__', '__ge__', '__getattribute__',
    '__getitem__', '__gt__', '__hash__', '__init__', '__iter__', '__le__',
    '__len__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__',
    '__reduce_ex__', '__repr__', '__setattr__', '__setitem__', '__str__',
    '__weakref__', 'clear', 'copy', 'fromkeys', 'get', 'has_key', 'items',
    'iteritems', 'iterkeys', 'itervalues', 'keys', 'pop', 'popitem',
    'setdefault', 'update', 'values']

    ~Ethan~
     
    Ethan Furman, Aug 3, 2010
    #17
  18. wheres pythonmonks

    Ethan Furman Guest

    Christian Heimes wrote:
    >> Perhaps punctuation will help clarify my intent:
    >>
    >> __missing__ is *not* part of (dict)s, as shown by dir(dict()):

    >
    > Indeed, that's correct. Can we agree, that __missing__ is an optional
    > feature of the dict interface, that can be implemented in subclasses of
    > dict?


    Absolutely.

    ~Ethan~
     
    Ethan Furman, Aug 4, 2010
    #18
  19. wheres pythonmonks

    John Posner Guest

    On 8/3/2010 6:48 PM, Ethan Furman wrote:
    > Christian Heimes wrote:
    >>> I just went and read the entry that had the bogus claim --
    >>> personally, I didn't see any confusion. I would like to point out the
    >>> __missing__ is *not* part of dicts (tested on 2.5 and 2.6 -- don't
    >>> have 2.7 installed yet).

    >>
    >> I beg your pardon but you are wrong. __missing__ is available for all
    >> *subclasses* of dict since Python 2.5. See
    >> http://svn.python.org/view/python/b...jects/dictobject.c?revision=81031&view=markup
    >>
    >>
    >>>>> class mydict(dict):

    >> ... def __missing__(self, key):
    >> ... print "__missing__", key
    >> ... raise KeyError(key)
    >> ...
    >>>>> m = mydict()
    >>>>> m[1]

    >> __missing__ 1
    >> Traceback (most recent call last):
    >> File "<stdin>", line 1, in <module>
    >> File "<stdin>", line 4, in __missing__
    >> KeyError: 1

    >
    > Perhaps punctuation will help clarify my intent:
    >
    > __missing__ is *not* part of (dict)s, as shown by dir(dict()):
    >
    > ['__class__', '__cmp__', '__contains__', '__delattr__', '__delitem__',
    > '__doc__', '__eq__', '__ge__', '__getattribute__', '__getitem__',
    > '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__len__',
    > '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__',
    > '__repr__', '__setattr__', '__setitem__', '__str__', 'clear', 'copy',
    > 'fromkeys', 'get', 'has_key', 'items', 'iteritems', 'iterkeys',
    > 'itervalues', 'keys', 'pop', 'popitem', 'setdefault', 'update',
    > 'values']
    >
    > And, just to state what is hopefully obvious, if you don't create
    > __missing__ yourself, it still isn't in the subclass:


    Right, a __missing__ method does not magically appear in the subclass.
    Rather, the subclass is allowed (but not required) to define a method
    named __missing__, which will "magically" be called in certain situations.

    Here's a dict subclass that uses the "magic" for a purpose that has
    nothing to do with default values:

    class BadKeyTrackerDict(dict):
    def __init__(self, *args, **kwargs):
    dict.__init__(self, *args, **kwargs)
    self.bad_keys = set([])

    def __missing__(self, key):
    """
    add missing key to "bad keys" set
    """
    self.bad_keys.add(key)
    raise KeyError

    Note that "defaultdict" is nowhere in sight here. It's the dict class
    (or type) itself that provides the magic -- but only for its subclasses.

    >
    > --> class somedict(dict):
    > ... "Is __missing__ defined if I don't define it? Nope."
    > ...
    > --> sd = somedict()
    > --> sd[1]
    > Traceback (most recent call last):
    > File "<stdin>", line 1, in <module>
    > KeyError: 1
    > --> dir(sd)
    > ['__class__', '__cmp__', '__contains__', '__delattr__', '__delitem__',
    > '__dict__', '__doc__', '__eq__', '__ge__', '__getattribute__',
    > '__getitem__', '__gt__', '__hash__', '__init__', '__iter__', '__le__',
    > '__len__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__',
    > '__reduce_ex__', '__repr__', '__setattr__', '__setitem__', '__str__',
    > '__weakref__', 'clear', 'copy', 'fromkeys', 'get', 'has_key', 'items',
    > 'iteritems', 'iterkeys', 'itervalues', 'keys', 'pop', 'popitem',
    > 'setdefault', 'update', 'values']
    >
    > ~Ethan~


    -John
     
    John Posner, Aug 4, 2010
    #19
  20. wheres pythonmonks

    John Posner Guest

    On 8/2/2010 11:00 PM, John Posner wrote:
    > On 7/31/2010 1:31 PM, John Posner wrote:
    >>
    >> Caveat -- there's another description of defaultdict here:
    >>
    >> http://docs.python.org/library/collections.html#collections.defaultdict
    >>
    >> ... and it's bogus. This other description claims that __missing__ is a
    >> method of defaultdict, not of dict.

    >
    > Following is a possible replacement for the bogus description. Comments
    > welcome. I intend to submit a Python doc bug, and I'd like to have a
    > clean alternative to propose.



    After some off-list discussion with Ethan Furman (many thanks!), the
    Python Doc bug is submitted: #9536 at bugs.python.org.

    -John
     
    John Posner, Aug 6, 2010
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Stephen Walch

    Disabling default button behavior

    Stephen Walch, Nov 3, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    3,102
    Alvin Bruney
    Nov 3, 2003
  2. Stephen F Zelonis

    Strange Behavior on Default Web page

    Stephen F Zelonis, Apr 5, 2004, in forum: ASP .Net
    Replies:
    0
    Views:
    317
    Stephen F Zelonis
    Apr 5, 2004
  3. Denis Remezov
    Replies:
    4
    Views:
    359
  4. Mantorok Redgormor
    Replies:
    70
    Views:
    1,774
    Dan Pop
    Feb 17, 2004
  5. Pete Shinners

    importhook default behavior

    Pete Shinners, Jun 17, 2004, in forum: Python
    Replies:
    0
    Views:
    288
    Pete Shinners
    Jun 17, 2004
Loading...

Share This Page