Tuples and immutability

[Eric Jacoboni]

Hi,

I'm using Python 3.3 and i have a problem for which i've still not found
any reasonable explanation...

a_tuple = ("spam", [10, 30], "eggs")
a_tuple[1] += [20]

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

Ok... I accept this message as += is a reassignment of a_tuple[1] and a
tuple is immutable...

But, then, why a_tuple is still modified?
('spam', [10, 30, 20], 'eggs')

I get a TypeError for an illegal operation, but this operation is still
completed?

 

[Zachary Ware]

Hi,

I'm using Python 3.3 and i have a problem for which i've still not found
any reasonable explanation...

a_tuple = ("spam", [10, 30], "eggs")
a_tuple[1] += [20]

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

Ok... I accept this message as += is a reassignment of a_tuple[1] and a
tuple is immutable...

But, then, why a_tuple is still modified?
('spam', [10, 30, 20], 'eggs')

I get a TypeError for an illegal operation, but this operation is still
completed?

You're not the first person to have this question

http://docs.python.org/3/faq/progra...em-raise-an-exception-when-the-addition-works

 

[Chris Angelico]

I'm using Python 3.3 and i have a problem for which i've still not found
any reasonable explanation...

a_tuple = ("spam", [10, 30], "eggs")
a_tuple[1] += [20]

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

Ok... I accept this message as += is a reassignment of a_tuple[1] and a
tuple is immutable...

But, then, why a_tuple is still modified?
('spam', [10, 30, 20], 'eggs')

I get a TypeError for an illegal operation, but this operation is still
completed?

This is a common confusion.

The += operator does two things. First, it asks the target to please
do an in-place addition of the other operand. Then, it takes whatever
result the target gives back, and assigns it back into the target. So
with a list, it goes like this:

foo = [10, 30]
foo.__iadd__([20]) [10, 30, 20]
foo = _

Note that the second step returned a three-element list. Then the +=
operator stuffs that back into the source. In the case of a list, it
does that addition by extending the list, and then returning itself.

The putting-back-into-the-target step fails with a tuple, because a
tuple's members can't be reassigned. But the list has already been
changed by that point. So, when you go to look at it, you see a
changed list.

You can call on this behaviour more safely by using the append() or
extend() methods.

a_tuple = ("spam", [10, 30], "eggs")
a_tuple[1].extend([20])
a_tuple

('spam', [10, 30, 20], 'eggs')

a_tuple = ("spam", [10, 30], "eggs")
a_tuple[1].append(20)
a_tuple

('spam', [10, 30, 20], 'eggs')

(append will add one more element, extend is roughly the same as you
had). Then you're not trying to assign to the tuple, but you're still
mutating the list.

Hope that helps!

ChrisA

 

[Marko Rauhamaa]

a_tuple[1] += [20]

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

[...]

But, then, why a_tuple is still modified?

That's because the += operator

1. modifies the list object in place

2. tries to replace the tuple slot with the list (even though the list
hasn't changed)

It's Step 2 that raises the exception, but the damage has been done
already.

One may ask why the += operator modifies the list instead of creating a
new object. That's just how it is with lists.

BTW, try:

a_tuple[1].append(20)
a_tuple[1].extend([20])

Try also:

a_tuple[1] = a_tuple[1]

Marko

 

[Eric Jacoboni]

Le 27/02/2014 17:13, Zachary Ware a écrit :

You're not the first person to have this question

http://docs.python.org/3/faq/progra...em-raise-an-exception-when-the-addition-works

Oh yes, i was aware of this explanation (thanks to Chris for his answer,
too)... and that's why i wrote "reasonable"
I know i should use append() or extend() and i understand the tricky
implications of += in this context. But, imho, it's far from being a
intuitive result, to say the least.

 

[Zachary Ware]

Le 27/02/2014 17:13, Zachary Ware a écrit :

Oh yes, i was aware of this explanation (thanks to Chris for his answer,
too)... and that's why i wrote "reasonable"
I know i should use append() or extend() and i understand the tricky
implications of += in this context. But, imho, it's far from being a
intuitive result, to say the least.

Well, once you understand what's actually going on, it's the result
that you should expect. The FAQ entry I linked to exists to help
people get to that point.

To answer your specific questions:

But, then, why a_tuple is still modified?

It is not. a_tuple is still "('spam', <list object at specific
address>, 'eggs')", exactly the same before and after the attempted
"a_tuple[1] += [20]". The change is internal to <list object at
specific address>.

I get a TypeError for an illegal operation, but this operation is still
completed?

Half completed. The extension of <list object at specific address>
happened as expected, but the assignment of <list object at specific
address> to a_tuple[1] didn't. It looks like it did, though, because
the assignment was just trying to assign the same object to the same
index.

 

[Chris Angelico]

Where is `.__iadd__()` called outside of `list += X`? If the only
difference from `.extend()` is that it returns `self`, but the list was
already modified anyway, why bother with reassignment?

Not everything handles += that way. You can't mutate the integer 5
into 7 because someone had 5 in x and wrote "x += 2". So it has to
reassign.

Actually, integers just don't define __iadd__, but the principle applies.

ChrisA

 

[Ian Kelly]

Where is `.__iadd__()` called outside of `list += X`? If the only
difference from `.extend()` is that it returns `self`, but the list was
already modified anyway, why bother with reassignment?

x += y is meant to be equivalent, except possibly in-place and more
efficient, than x = x + y. If you skip the assignment, and that
assignment is meaningful to whatever the left side may be (e.g.
assigning to a descriptor or something that invokes __setitem__ or
__setattr__), then the operation is not equivalent.

 

[John O'Hagan]

a_tuple[1] += [20]

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

[...]

But, then, why a_tuple is still modified?

That's because the += operator

1. modifies the list object in place

2. tries to replace the tuple slot with the list (even though the
list hasn't changed)

It's Step 2 that raises the exception, but the damage has been done
already.

One may ask why the += operator modifies the list instead of creating
a new object. That's just how it is with lists.

BTW, try:

a_tuple[1].append(20)
a_tuple[1].extend([20])

Try also:

a_tuple[1] = a_tuple[1]

[...]

Also try:

x = a_tuple[1] #What's in a name?
x is a_tuple[1] #Obviously, but:
x += [1] #No error
a_tuple[1] += [1] #Error

Same object, just a different name - but a different result. I get why,
but still find that odd.

 

[Marko Rauhamaa]

Same object, just a different name - but a different result. I get
why, but still find that odd.

The general principle is stated in the language specification:

<URL: http://docs.python.org/3.2/reference/simple_stmts.html
#augmented-assignment-statements>:

Also, when possible, the actual operation is performed in-place,
meaning that rather than creating a new object and assigning that to
the target, the old object is modified instead.

[...] with the exception of the possible in-place behavior, the
binary operation performed by augmented assignment [x += y] is the
same as the normal binary operations [x = x + y].

We should call += "dual-use technology."


Marko

 

[Joshua Landau]

a_tuple = ("spam", [10, 30], "eggs")
a_tuple[1] += [20]

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

Ok... I accept this message as += is a reassignment of a_tuple[1] and a
tuple is immutable...

But, then, why a_tuple is still modified?

This is a common confusion.

The += operator does two things. First, it asks the target to please
do an in-place addition of the other operand. Then, it takes whatever
result the target gives back, and assigns it back into the target. So
with a list, it goes like this:

foo = [10, 30]
foo.__iadd__([20]) [10, 30, 20]
foo = _

Would it be better to add a check here, such that if this gets raised
to the top-level it includes a warning ("Addition was inplace;
variable probably mutated despite assignment failure")?

 

[Chris Angelico]

Would it be better to add a check here, such that if this gets raised
to the top-level it includes a warning ("Addition was inplace;
variable probably mutated despite assignment failure")?

That'd require figuring out whether or not the variable was actually
mutated, and that's pretty hard to work out. So there's a FAQ entry,
which Zachary already posted:

http://docs.python.org/3/faq/progra...em-raise-an-exception-when-the-addition-works

Also, we just answer this question every now and then Presumably
more often on -tutor than here.

ChrisA

 

[Mark H. Harris]

('spam', [10, 30, 20], 'eggs')

I get a TypeError for an illegal operation, but this operation is still

completed?

hi Eric, others have answered your question specifically, but the issue (which is recurring) begs a couple of questions, which will also be recurring.... ehem.

This little snag (if it can be called that) is a side effect from two python inherent design considerations:

1) not to be a nudge, but dynamic typing is the first...

2) and the concept of immutability is the second.

I'll address the second first by asking a question... should an immutable type (object) be able to hold (contain) mutable objects ... should tuples be allowed to hold lists?

lists within a tuple should be converted to tuples. If you want a tuple to hold a list, make it a list in the first place. Tuples should not be changed... and as you point out... half changing a tuple is not a good condition if there is an exception...

Which brings me to my second point... dynamic binding... (and everything associated with that) is at play here too.... please, don't get me wrong, this is not flame bait and I'm not advocating static binding, nor do I wantstatic binding, nor do I want to open that can of worms again... just saying.

I really think this is a bug; honestly. IMHO it should be an error to use += with an immutable type and that means not at all. In other words, the list should not even be considered, because we're talking about changing a tuple... which should not be changed (nor should its members be changed).

Soooo.... the type does not support item assignment, yet the item got assigned. This is at least inconsistent... and to my small mind means... bug report.

 

[Eric Jacoboni]

Le 01/03/2014 01:22, Mark H. Harris a écrit :

I'll address the second first by asking a question... should an immutable type (object) be able to hold (contain) mutable objects ... should tuples be allowed to hold lists?

lists within a tuple should be converted to tuples. If you want a tuple to hold a list, make it a list in the first place.

You're perfectly right and that why i've corrected my code to use a list
of lists instead of tuple of list. I was hoping Python would prevent me
for such a mistake (because such a situation can't be cleary
intentional, isn't ?). Now, i will use tuple only with immutable items.


IMHO it should be an error to use += with an immutable type and that
means not at all.

In other words, the list should not even be considered, because we're
talking about changing a tuple...

which should not be changed (nor should its members be changed).

I agree with that too... My error was to first consider the list, then
the tuple... I should have considered the tuple first...
Anyway, the TypeError should rollback, not commit the mistake.

 

[Chris Angelico]

lists within a tuple should be converted to tuples. If you want a tuple to hold a list, make it a list in the first place. Tuples should not bechanged... and as you point out... half changing a tuple is not a good condition if there is an exception...

A tuple is perfectly fine containing a list. If you want a tuple to be
"recursively immutable", then you're talking about hashability, and
*then* yes, you need to convert everything into tuples - but a tuple
is not just an immutable list. The two are quite different in pupose.

I really think this is a bug; honestly. IMHO it should be an error to use += with an immutable type and that means not at all. In other words,the list should not even be considered, because we're talking about changing a tuple... which should not be changed (nor should its members be changed).

Definitely not! Incrementing an integer with += is a perfectly normal
thing to do:

x = 5
x += 1

It's just a matter of knowing the language and understanding what's going on.

ChrisA

 

[Mark H. Harris]

I agree with that too... My error was to first consider the list, then
the tuple... I should have considered the tuple first...
Anyway, the TypeError should rollback, not commit the mistake.

I believe so too, but I'm not one of the core devs. And they do not agree..

Ever since day one with me and python I have questioned whether a tuple even makes sense. Well, certainly it does, because it has so much less overhead and yet it acts like a list (which for so many situations thats really what we want anyway... a list, that never changes). Tuples are great, for what they are designed to do.

But now consider, why would I purposely want to place a mutable object within an immutable list? A valid question of high importance. Why indeed?

I really believe IMHO that the error should have come when you made the list an item of a tuple. An immutable object should have NO REASON to containa mutable object like list... I mean the whole point is to eliminate the overhead of a list ... why would the python interpreter allow you to place a mutable object within an immutable list in the first place. This is just philosophical, and yes, the core dev's are not going to agree with me on this either.

I think the situation is interesting for sure... and it will surface again,you can count on that.

Cheers

 

[Ian Kelly]

I really think this is a bug; honestly. IMHO it should be an error to use += with an immutable type and that means not at all. In other words,the list should not even be considered, because we're talking about changing a tuple... which should not be changed (nor should its members be changed).

How would you propose doing that? Bear in mind that while Python
knows that tuples specifically are immutable, it doesn't generally
know whether a type is immutable. The best it can do is try the
operation and raise a TypeError if it fails. So you can special-case
the code for += to fail earlier if the left side of the assignment is
an tuple index, but it's not currently possible to do the same for
arbitrary immutable user types.

Even if that is solved, how you do propose preventing the simple workaround:
tup = (0, [1, 2, 3], 4)
thelist = tup[1]
thelist += [5, 6]
which currently works just fine.

I don't think that entirely disallowing mutable objects inside tuples
is an acceptable solution. For one, it would mean never allowing
instances of custom classes. They could then basically only contain
strings, numbers, frozensets, and other tuples. For another, my
experience suggests that if I'm putting a list inside a tuple in the
first place, I am generally not relying on the immutability of that
tuple anyway, so I don't really see this as fixing an actual problem.
The usual reason for wanting an immutable object is to hash it, and a
tuple containing a list already can't be hashed.

 

[Ian Kelly]

Anyway, the TypeError should rollback, not commit the mistake.

The Python interpreter isn't a database. It can't rollback the object
because the operation that was performed may not be reversible.
Consider for example a socket class where the += operator is
overloaded to send a message on the socket. The message can't be
unsent.

 

[Mark H. Harris]

One very common example of tuples containing lists is when lists are
passed to any function that accepts *args, because the extra arguments
are passed in a tuple. A similarly common example is when returning
multiple objects from a function, and one of them happens to be a
list, because again they are returned in a tuple.

def f(*args):
print(args)
return (args[1:]

result = f(1, 2, 3, [4, 5]) (1, 2, 3, [4, 5])
print(result)

(2, 3, [4, 5])

I agree Ian... good points all. ... again, I'm not arguing with anyone... just saying that an error (whatever we mean by that) should not half-way-fail.... we are only pointing out the problem... we have not idea what the solution is yet.

Intuitively everyone can see that there is a problem here... the debate cannot be answered either because of the inherent design of python (almost all of which we love). So, as they say, what is a mother to do? ... I mean, some people's kids...

I don't know how I propose to handle the problem... I think the first step is getting everyone to agree that there IS a problem... then debate how to tackle the solution proposals.

marcus

 

[Mark H. Harris]

How would you propose doing that? Bear in mind that while Python
knows that tuples specifically are immutable, it doesn't generally
know whether a type is immutable.

hi Ian, consider the problem... its a "cake" and "eat it too" scenario...you can't have your cake and eat it too....

This is what I mean... the error message is telling the user that it cannot do what he has requested, and yet IT DID. You have one of two scenarios: 1) the message is arbitrarily lying and it really can assign an immutable's item... (and it did!) or 2) It really should NOT assign an immutables item (the message is truth) but the action was "allowed" anyway despite the protocol and accepted policy... in which case the two scenarios add upto a giant logical inconsistency... QED a bug.

There really is no way to get around this from the average user's perspective. And therefore, this is going to come up again and again and again... because coders don't like unexpected logical inconsistencies.

It is not helpful either to explain it away by knowing how the policy worksunder the covers... the average user of the python language should not have to understand the policy of the underlying substructure...

1) either they can't assign the list item of an immutable tuple type (and the action is a flaw, say bug

of

2) they really CAN set the immutable's list item type (which it did!) and the message is bogus.


Its one way or the other.... we can't have our cake and eat it too... thisis (one way or the other) a bug.

IMHO