Strange Behavior with Old-Style classes and implicit __contains__

[rconradharris]

A co-worker of mine came across some interesting behavior in the
Python interpreter today and I'm hoping someone more knowledgeable in
Python internals can explain this to me.

First, we create an instance of an Old-Style class without defining a
__contains__ but instead define a __getitem__ method in which we raise
KeyError. Next we repeatedly use the 'in' operator to test to see
whether something, a string, an int, etc is an attribute of this new
instance.

Here's the strange part: The first test will return False as if the
__getitem__ was never called. The next test will raise a KeyError as
we'd expect. The test after that will again return False. This goes on
ad infinitum.

In Code:
Python 2.5 (r25:51908, Jan 21 2007, 03:10:25)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more
information. ... def __getitem__(self, key):
... raise KeyError
... Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __getitem__
KeyError


According to the language reference, if __contains__ isn't defined for
Old-Style classes and __getitem__ is defined, __getitem__ will be
called. So, how then can False ever be returned?

And to make matters worse, I've set up a situation where Python will
flat-out return incorrect results:

Python 2.5 (r25:51908, Jan 21 2007, 03:10:25)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information..... def __getitem__(self, key):
.... raise KeyError
....

foo = Foo()
"asdf" in foo False
1 in set([1,2,3]) <---- So the prior KeyError from another class is interacting and producing bad output

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __getitem__
KeyError

According to our cursory testing, this funny business doesn't happen
with New-Style Classes or when using PyPy.

If anyone can provide some insight into this, it would be greatly
appreciated.

Thanks,
Rick Harris

 

[Gabriel Genellina]

First, we create an instance of an Old-Style class without defining a
__contains__ but instead define a __getitem__ method in which we raise
KeyError. Next we repeatedly use the 'in' operator to test to see
whether something, a string, an int, etc is an attribute of this new
instance.

Here's the strange part: The first test will return False as if the
__getitem__ was never called. The next test will raise a KeyError as
we'd expect. The test after that will again return False. This goes on
ad infinitum.

In Code:
Python 2.5 (r25:51908, Jan 21 2007, 03:10:25)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more
information.... def __getitem__(self, key):
... raise KeyError
...Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __getitem__
KeyError

First I want to say that __getitem__ should raise IndexError, not
KeyError, to indicate "not found" - just to make clear the observed
behavior. (Using IndexError, you always get False, as expected).
Python 2.4 and 2.3 never return False, always showing the KeyError
exception, also as expected.

foo = Foo()
"asdf" in foo False
1 in set([1,2,3]) <---- So the prior KeyError from another class
is interacting and producing bad output

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __getitem__
KeyError

I have a displayhook installed, and it also were interfering with the
results; it appears that the next __getitem__ call after the KeyError was
raised (wherever it is called) "sees" the previous exception.

You should file a bug at http://sourceforge.net/bugs/?group_id=5470

 

[Steven D'Aprano]

A co-worker of mine came across some interesting behavior in the
Python interpreter today and I'm hoping someone more knowledgeable in
Python internals can explain this to me.

First, we create an instance of an Old-Style class without defining a
__contains__ but instead define a __getitem__ method in which we raise
KeyError. Next we repeatedly use the 'in' operator to test to see
whether something, a string, an int, etc is an attribute of this new
instance.

Here's the strange part: The first test will return False as if the
__getitem__ was never called. The next test will raise a KeyError as
we'd expect. The test after that will again return False. This goes on
ad infinitum.

I can confirm that. It looks like __getitem__ is only being called every
second time.

class Parrot:
def __getitem__(self, n):
print "Checking index %s..." % n
raise KeyError

def tester(n):
parrot = Parrot()
results = []
for i in range(n):
try:
results.append(i in parrot)
except KeyError:
results.append("KeyError")
return results



Here are the results under Python 2.5:
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
[False, 'KeyError', False, 'KeyError', False,
'KeyError', False, 'KeyError', False, 'KeyError']


And here are the results under Python 2.4.3:
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
['KeyError', 'KeyError', 'KeyError', 'KeyError', 'KeyError',
'KeyError', 'KeyError', 'KeyError', 'KeyError', 'KeyError']


Looks like a bug to me.

 

[Steven D'Aprano]

First I want to say that __getitem__ should raise IndexError, not
KeyError, to indicate "not found"

How do you know the Original Poster's class was meant to be a sequence
rather than a mapping?

 

[Scott David Daniels]

Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
[False, 'KeyError', False, 'KeyError', False,
'KeyError', False, 'KeyError', False, 'KeyError']


And here are the results under Python 2.4.3:
[works]

Looks like a bug to me.

No problem with 2.5.1c1 here.

--Scott David Daniels
(e-mail address removed)

 

[Gabriel Genellina]

En Thu, 12 Apr 2007 01:23:08 -0300, Steven D'Aprano

How do you know the Original Poster's class was meant to be a sequence
rather than a mapping?

Ouch, no, sorry, disregard that comment.