Re: dictionary size changed during iteration

Discussion in 'Python' started by Peter Otten, Apr 20, 2011.

  1. Peter Otten

    Peter Otten Guest

    Laszlo Nagy wrote:

    > Given this iterator:
    >
    > class SomeIterableObject(object):
    > ....
    > ....
    >
    > def __iter__(self):
    > ukeys = self.updates.keys()
    > for key in ukeys:
    > if self.updates.has_key(key):
    > yield self.updates[key]
    > for rec in self.inserts:
    > yield rec
    > ....
    > ....
    >
    > How can I get this exception:
    >
    > RuntimeError: dictionary changed size during iteration
    >
    >
    > It is true that self.updates is being changed during the iteration. But
    > I have created the "ukeys" variable solely to prevent this kind of
    > error. Here is a proof of correctness:
    >
    >>>> d = {1:1,2:2}
    >>>> k = d.keys()
    >>>> del d[1]
    >>>> k

    > [1, 2]
    >>>> k is d.keys()

    > False
    >
    > So what is wrong with this iterator? Why am I getting this error message?


    The keys() method which used to return a list in 2.x was changed in 3.x to
    return a view object and to become more or less the equivalent of the old
    dict.iterkeys():

    >>> d = dict(a=1)
    >>> keys = d.keys()
    >>> keys

    dict_keys(['a'])
    >>> for k in keys:

    .... d["b"] = 42
    ....
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    RuntimeError: dictionary changed size during iteration
    >>> keys

    dict_keys(['a', 'b'])

    You now have to create the list explicitly to avoid the error:

    >>> d = dict(a=1)
    >>> keys = list(d.keys())
    >>> for k in keys:

    .... d["b"] = 42
    ....
    >>> d

    {'a': 1, 'b': 42}
    >>> keys

    ['a']
     
    Peter Otten, Apr 20, 2011
    #1
    1. Advertising

  2. Peter Otten

    Roy Smith Guest

    In article <>,
    Peter Otten <> wrote:

    > You now have to create the list explicitly to avoid the error:
    >
    > >>> d = dict(a=1)
    > >>> keys = list(d.keys())
    > >>> for k in keys:

    > ... d["b"] = 42
    > ...


    That works, but if d is large, it won't be very efficient because it has
    to generate a large list.

    If d is large, and the number of keys to be mutated is relatively small,
    a better solution may be to do it in two passes. The first loop
    traverses the iterator and builds a list of things to be changed. The
    second loop changes them.

    changes = [ ]
    for key in d.iterkeys():
    if is_bad(key):
    changes.append(key)
    for key in changes:
    d[key] = "I'm not dead yet"

    Both solutions are O(n), but the second may run significantly faster and
    use less memory.
     
    Roy Smith, Apr 22, 2011
    #2
    1. Advertising

  3. Peter Otten

    Laszlo Nagy Guest


    >> ...

    > That works, but if d is large, it won't be very efficient because it has
    > to generate a large list.

    It is not large. But I'm using Python 2.6 , not Python 3.

    I did not get this error again in the last two days. I'll post a new
    reply if I encounter it again. (It happened just a few times out of many
    thousand program invocations)
     
    Laszlo Nagy, May 6, 2011
    #3
  4. Peter Otten

    Paul Rubin Guest

    Roy Smith <> writes:
    > changes = [ ]
    > for key in d.iterkeys():
    > if is_bad(key):
    > changes.append(key)


    changes = list(k for k in d if is_bad(k))

    is a little bit more direct.
     
    Paul Rubin, May 7, 2011
    #4
  5. Peter Otten

    Roy Smith Guest

    In article <>,
    Paul Rubin <> wrote:

    > Roy Smith <> writes:
    > > changes = [ ]
    > > for key in d.iterkeys():
    > > if is_bad(key):
    > > changes.append(key)

    >
    > changes = list(k for k in d if is_bad(k))
    >
    > is a little bit more direct.


    This is true. I still file list comprehensions under "new fangled
    toys". While I use them, and appreciate their value, I admit they're
    not always the first thing that comes to my mind.

    OBTW,

    > changes = [k for k in d if is_bad(k)]


    is even more direct :)
     
    Roy Smith, May 7, 2011
    #5
  6. Peter Otten

    Hans Mulder Guest

    On 08/05/2011 00:12, Roy Smith wrote:
    > In article<>,
    > Paul Rubin<> wrote:
    >
    >> Roy Smith<> writes:
    >>> changes = [ ]
    >>> for key in d.iterkeys():
    >>> if is_bad(key):
    >>> changes.append(key)

    >>
    >> changes = list(k for k in d if is_bad(k))
    >>
    >> is a little bit more direct.

    >
    > This is true. I still file list comprehensions under "new fangled
    > toys". While I use them, and appreciate their value, I admit they're
    > not always the first thing that comes to my mind.
    >
    > OBTW,
    >
    >> changes = [k for k in d if is_bad(k)]

    >
    > is even more direct :)


    How about:

    changes = filter(is_bad, d)

    Or would that be too compact?

    -- HansM
     
    Hans Mulder, May 8, 2011
    #6
  7. Peter Otten

    Paul Rubin Guest

    Hans Mulder <> writes:
    > How about:
    > changes = filter(is_bad, d)
    > Or would that be too compact?


    I thought of writing something like that but filter in python 3 creates
    an iterator that would have the same issue of walking the dictionary
    while the dictionary is mutating.

    changes = list(filter(is_bad, d))

    should work.
     
    Paul Rubin, May 8, 2011
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Roman Suzi
    Replies:
    0
    Views:
    339
    Roman Suzi
    Jan 19, 2005
  2. Terry Reedy
    Replies:
    0
    Views:
    382
    Terry Reedy
    Jan 20, 2005
  3. robert
    Replies:
    29
    Views:
    1,073
    Raymond Hettinger
    Mar 14, 2006
  4. Jean-Paul Calderone
    Replies:
    0
    Views:
    376
    Jean-Paul Calderone
    Mar 13, 2006
  5. Robert Dailey
    Replies:
    6
    Views:
    399
    Terry Reedy
    Dec 9, 2008
Loading...

Share This Page