Modify dict/set during iteration?

M

metal

I used this quickndirty way, any good idea to solve this problem?

def miter(iterable):
cache = set()
while True:
try:
for x in iterable:
if x not in cache:
cache.add(x)
yield x
break
except RuntimeError, e:
# Not sure if there were any other RuntimeError
if 'changed size during iteration' in e.message:
continue
raise

s = set([1, 2, 3, 5, 6, 7])
for x in miter(s):
start, stop = x, x+1
while start-1 in s:
start -= 1
while stop in s:
stop += 1
for i in range(start, stop):
s.discard(i)
print '[%d:%d]' % (start, stop)

d = {'foo':1}
for x in miter(d):
if x == 'foo':
d['bar']=1
print x
 
S

Steven D'Aprano

I used this quickndirty way, any good idea to solve this problem?

It's not a problem that wants solving, it's a feature that wants paying
attention to.

As a general rule, you shouldn't modify data structures while you're
iterating over them, unless the data structure is advertised as safe to
modify while being iterated over, or you can do so in a way that is
guaranteed to be safe. When it comes to dicts and sets, neither of those
conditions hold.

Why do you want to do this? It seems like a lot of effort to avoid a
simple, straight-forward idiom:

make a copy of the dict or set
iterate over the copy, making changes to the original


What are you trying to accomplish by modifying objects while iterating
over them?

def miter(iterable): [...]
except RuntimeError, e:
# Not sure if there were any other RuntimeError
if 'changed size during iteration' in e.message:
continue

That is not safe. There's no guarantee that the error message won't
change, even between one minor version to another, or between one Python
implementation and another. If you insist on making such an unsafe test,
at the very least be as conservative in what you expect as possible:

if 'changed' in e.message and 'size' in e.message:

and hope that nobody runs your code with internationalised error messages.
 
M

metal

I used this quickndirty way, any good idea to solve this problem?

It's not a problem that wants solving, it's a feature that wants paying
attention to.

As a general rule, you shouldn't modify data structures while you're
iterating over them, unless the data structure is advertised as safe to
modify while being iterated over, or you can do so in a way that is
guaranteed to be safe. When it comes to dicts and sets, neither of those
conditions hold.

Why do you want to do this? It seems like a lot of effort to avoid a
simple, straight-forward idiom:

make a copy of the dict or set
iterate over the copy, making changes to the original

What are you trying to accomplish by modifying objects while iterating
over them?


def miter(iterable): [...]
        except RuntimeError, e:
            # Not sure if there were any other RuntimeError
            if 'changed size during iteration' in e.message:
                continue

That is not safe. There's no guarantee that the error message won't
change, even between one minor version to another, or between one Python
implementation and another. If you insist on making such an unsafe test,
at the very least be as conservative in what you expect as possible:

if 'changed' in e.message and 'size' in e.message:

and hope that nobody runs your code with internationalised error messages..

Yes the idoim rulz. I confused my self.

Maybe my real goal is the following:

def miter(iterable):
for x in tuple(iterable):
if x in iterable:
yield x
 
S

Steven D'Aprano

Maybe my real goal is the following:

def miter(iterable):
for x in tuple(iterable):
if x in iterable:
yield x


I don't think that does what you think it does. For some iterables it
doesn't do anything at all:
it = iter([1,2,3])
for item in miter(it):
.... print item
....
and for others it is redundant:
it = [1,2,3]
for item in miter(it): # same as `for item in it:`
.... print item
....
1
2
3

Could you explain what you are trying to accomplish?
 
D

Dave Angel

metal said:
Steven said:
I used this quickndirty way, any good idea to solve this problem?
It's not a problem that wants solving, it's a feature that wants paying
attention to.

As a general rule, you shouldn't modify data structures while you're
iterating over them, unless the data structure is advertised as safe to
modify while being iterated over, or you can do so in a way that is
guaranteed to be safe. When it comes to dicts and sets, neither of those
conditions hold.

Why do you want to do this? It seems like a lot of effort to avoid a
simple, straight-forward idiom:

make a copy of the dict or set
iterate over the copy, making changes to the original

What are you trying to accomplish by modifying objects while iterating
over them?



def miter(iterable):
[...]

except RuntimeError, e:
# Not sure if there were any other RuntimeError
if 'changed size during iteration' in e.message:
continue
That is not safe. There's no guarantee that the error message won't
change, even between one minor version to another, or between one Python
implementation and another. If you insist on making such an unsafe test,
at the very least be as conservative in what you expect as possible:

if 'changed' in e.message and 'size' in e.message:

and hope that nobody runs your code with internationalised error messages.

Yes the idoim rulz. I confused my self.

Maybe my real goal is the following:

def miter(iterable):
for x in tuple(iterable):
if x in iterable:
yield x
That works, though I'd change the parameter name to something to
indicate that it's a set or dictionary, not a general iterable.

As someone else pointed out, you can't use 'in' on a generator, for example.

Assuming a dict, what happens if a new item created in the dictionary,
but reusing an old key? It won't cause an exception, but you run the
risk of either processing some data twice, or some not-at-all. Whether
that matters or not depends on your use-case. Sometimes you need to
copy the keys, and sometimes you need to copy the keys and their values.

DaveA
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top