var or inout parm?

[Marc 'BlackJack' Rintsch]

import numpy
t = (numpy.zeros(10),)
t

(array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),)>>> t[0] +=
1

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

Of course, the side effect occurs before the exception, so:

t[0]

array([ 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

Actually I would consider this to be a bug. The tuple is immutable, but
no mutation of the tuple is ever attempted.

No bug because a mutation *is* attempted. ``a += x`` calls `a.__iadd__`
which *always* returns the result which is *always* rebound to the name
`a`. Even with mutable objects where `__iadd__()` simply returns
`self`! It has to be this way because the compiler has no idea if the
object bound to `a` will be mutable or immutable when the code actually
runs.

In [252]: def f(a, x):
.....: a += x
.....:

In [253]: dis.dis(f)
2 0 LOAD_FAST 0 (a)
3 LOAD_FAST 1 (x)
6 INPLACE_ADD
7 STORE_FAST 0 (a)
10 LOAD_CONST 0 (None)
13 RETURN_VALUE

Ciao,
Marc 'BlackJack' Rintsch

 

[Hrvoje Niksic]

Actually I would consider this to be a bug. The tuple is immutable,
but no mutation of the tuple is ever attempted.

That's a common misconception about how += works in Python. It simply
*always* rebinds. Once you grok that, everything else follows.

 

[sturlamolden]

No bug because a mutation *is* attempted. ``a += x`` calls `a.__iadd__`
which *always* returns the result which is *always* rebound to the name
`a`. Even with mutable objects where `__iadd__()` simply returns
`self`!

No, a mutation is not attempted, even if __iadd__() always returns a
value. If a rebinding of a member to the same member is attempted, the
tuple should not raise an exception. The tuple should check that it is
actually being *mutated* before it raises any exception. There is an
attempted write to the tuple, but not an attempted mutation of the
tuple. The tuple should tell the difference. Immutability does not
imply inwriteability, if the write operation changes nothing. But the
tuple raises an exception on any write attempt.

 

[Hrvoje Niksic]

No, a mutation is not attempted, even if __iadd__() always returns a
value.

Mutation is attempted. A += x (where "A" could be anything valid at
the left-hand side of assignment, including item subscript) is not
implemented intuitivaly, as:

if hasattr(b, '__iadd__'):
A.__iadd__(x) # ignore return value
else:
A = A.__add__(x)

It is implemented as something like:

if hasattr(b, '__iadd__'):
newval = A.__iadd__(x)
else:
newval = A.__add__(x)
A = newval

So the only difference between __add__ and __iadd__ is that __iadd__
is only consulted on +=, where as __add__ is consulted on both + and
+= (in the latter case only if __iadd__ is missing).

The tuple should check that it is
actually being *mutated* before it raises any exception.

Tuple has no way to check that. What tuple sees is only the last
line:

t[0] = newval

At that point, the information about what is really going on is long
lost. The only thing tuple could do is detect that the same object is
being written that's already there, but tuple doesn't do that by
design.

 

[Steve Holden]

No, a mutation is not attempted, even if __iadd__() always returns a
value. If a rebinding of a member to the same member is attempted, the
tuple should not raise an exception. The tuple should check that it is
actually being *mutated* before it raises any exception. There is an
attempted write to the tuple, but not an attempted mutation of the
tuple. The tuple should tell the difference. Immutability does not
imply inwriteability, if the write operation changes nothing. But the
tuple raises an exception on any write attempt.

OK, so if you regard the current behavior as a bug explain how to modify
the tuple's __iadd__ method and the coding of the INPLACE_ADD operator.
At least in pseudocode.

Criticism is easy. Now demonstrate that it's *informed* criticism.
Enough of the "should". I am always suspicious of suggestions that say
what the interpreter "should" or "should not" do. It makes it sound as
though you can wave a magic wand to achieve the desired behavior.

The interpreter "should not" have a GIL. The tuple "should" check that
it is actually being mutated. How?

regards
Steve

 

[sturlamolden]

OK, so if you regard the current behavior as a bug explain how to modify
the tuple's __iadd__ method and the coding of the INPLACE_ADD operator.
At least in pseudocode.

Criticism is easy. Now demonstrate that it's *informed* criticism.
Enough of the "should". I am always suspicious of suggestions that say
what the interpreter "should" or "should not" do. It makes it sound as
though you can wave a magic wand to achieve the desired behavior.

The interpreter "should not" have a GIL.

The tuple "should" check that
it is actually being mutated. How?

In Python it would be something similar to:

def __setitem__(self, index, value):
if _buf[index] is not value: # given that _buf is the tuple's
internal buffer
raise TypeError, 'tuple' object does not support item
assignment

It should be the tuple's __setitem__ that was invoked here, not
__iadd__, or the parser is faulty.


S.M.

 

[sturlamolden]

def __setitem__(self, index, value):
if _buf[index] is not value: # given that _buf is the tuple's
internal buffer
raise TypeError, 'tuple' object does not support item
assignment

blæh, that should be self._buf[index]

 

[Steve Holden]

The interpreter "should not" have a GIL.

The tuple "should" check that
it is actually being mutated. How?

In Python it would be something similar to:

def __setitem__(self, index, value):
if _buf[index] is not value: # given that _buf is the tuple's
internal buffer
raise TypeError, 'tuple' object does not support item
assignment

It should be the tuple's __setitem__ that was invoked here, not
__iadd__, or the parser is faulty.

OK, so now you are proposing to alter the parser, and possibly the
implementation of the INPLACE_ADD opcode in eval.c, so can you give us
the patch for those, please?

This is exactly the point I was trying to make. It's easy to stand on
the sidelines and make sensible- or even intelligent-sounding
suggestions about how to change Python. But inside the interpreter
there's a maze of twisty little passages all alike (or similar
complexity), and hand-waving doesn't get you anywhere.

Discussion of such behavior as a "bug" is also pejorative, since the
current semantics are the way they are by design. You are therefore
disagreeing with the design of Python rather than discussing a bug in
its implementation.

That also raises the issues of how you get Jython and IronPython to
implement the same semantics, and whether you care about the millions of
lines of code that already assumes the current behavior.

You've got a lot of work to do ...

regards
Steve

 

[Marc 'BlackJack' Rintsch]

def __setitem__(self, index, value):
if _buf[index] is not value: # given that _buf is the tuple's
internal buffer
raise TypeError, 'tuple' object does not support item
assignment

blæh, that should be self._buf[index]

But then the error message is not true anymore because tuples *would*
support item assignment when the old and new value are the same. Which
leads to strange things like code that may or may not raise that
exception, depending on implementation details:

t = (1, 2)
t[0] = 1 # Maybe okay -- maybe not.
t[1] += 0 # Same here.

I'd find that quite odd.

Ciao,
Marc 'BlackJack' Rintsch

 

[sturlamolden]

OK, so now you are proposing to alter the parser, and possibly the
implementation of the INPLACE_ADD opcode in eval.c, so can you give us
the patch for those, please?

What? Take a look at the code again:

mytuple[0] += 1

should never attempt an __iadd__ on mytuple.

A sane parser would see this as:

tmp = mytuple.__getitem__(0)
tmp = tmp.__iadd__(1)
mytuple.__setitem__(0, tmp) # should this always raise an exception?

Discussion of such behavior as a "bug" is also pejorative, since the
current semantics are the way they are by design.

Right, this bug is by design. You learned that phrase from a guy in
Redmond?

 

[Steve Holden]

OK, so now you are proposing to alter the parser, and possibly the
implementation of the INPLACE_ADD opcode in eval.c, so can you give us
the patch for those, please?

What? Take a look at the code again:

mytuple[0] += 1

should never attempt an __iadd__ on mytuple.

A sane parser would see this as:

tmp = mytuple.__getitem__(0)
tmp = tmp.__iadd__(1)
mytuple.__setitem__(0, tmp) # should this always raise an exception?

Discussion of such behavior as a "bug" is also pejorative, since the
current semantics are the way they are by design.

Right, this bug is by design. You learned that phrase from a guy in
Redmond?

"It's not a bug, it's a feature" predates Microsoft by several years.

If I say you are ugly, that doesn't make it true. Neither does your
calling this a bug make it a bug.

The fact is that Python doesn't behave the way you want it to. If your
friend doesn't want to do what you do, does that make it a bug in his
behavior. You're being a little juvenile here.

regards
Steve

 

[sturlamolden]

OK, so now you are proposing to alter the parser, and possibly the
implementation of the INPLACE_ADD opcode in eval.c, so can you give us
the patch for those, please?

That is not where the problem resides.

 

[Arnaud Delobelle]

def __setitem__(self, index, value):
if _buf[index] is not value: # given that _buf is the tuple's
internal buffer
raise TypeError, 'tuple' object does not support item
assignment

blæh, that should be self._buf[index]

But then the error message is not true anymore because tuples *would*
support item assignment when the old and new value are the same. Which
leads to strange things like code that may or may not raise that
exception, depending on implementation details:

t = (1, 2)
t[0] = 1 # Maybe okay -- maybe not.
t[1] += 0 # Same here.

I'd find that quite odd.

Ciao,
Marc 'BlackJack' Rintsch

What I find a bit annoying is when you get both

* an exception
* a mutation

E.g.

t[1] *= 2

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

Now is t the same as before? Sometimes it is:

t (1, 2)
t[1] *= 2

Traceback (most recent call last):
(1, 2)

Sometimes not:

t (1, [2])
t[1] *= 2

Traceback (most recent call last):
(1, [2, 2])

I agree it's not a bug, but I never feel comfortable with it.

 

[Chris Rebert]

That is because integers are immutable. When x += 1 is done on an int,
there will be a rebinding. But try the same on say, a numpy array, and
the result will be different:

Yes, I know that. Did you not read the end of my email? Here it is again:

"""
If you were talking about lists rather than integers though, you'd be
absolutely correct, as the += ends up being a method call to __iadd__
instead of a plain assignment.
"""

Cheers,
Chris

 

[Steven D'Aprano]

OK, so now you are proposing to alter the parser, and possibly the
implementation of the INPLACE_ADD opcode in eval.c, so can you give us
the patch for those, please?

What? Take a look at the code again:

mytuple[0] += 1

should never attempt an __iadd__ on mytuple.

A sane parser would see this as:

tmp = mytuple.__getitem__(0)
tmp = tmp.__iadd__(1)
mytuple.__setitem__(0, tmp) # should this always raise an exception?

It is quite problematic if you do that. You've already pointed that out
yourself, by asking if "should this always raise an exception?". With the
behaviour you're asking for, there's ambiguity is what is allowed and
what isn't. You can try making __setitem__ cleverer, as in your earlier
post, but then:

t = (1, 2, 3)
t[0] += x

sometimes succeeds, sometimes fails, depending on the value of x.

And worse:

t[0] = y

also sometimes succeeds.

It's not clear that those occasional successes are actively harmful, but
nor is it clear that prohibiting them is harmful either. The best you get
from it is the ability to mutate mutable items in a tuple, but you can do
that already:

t = ([], None)
x = t[0]
x += ["mutated"]

So the current tuple implementation gives us consistent behaviour and
simplicity without preventing you from mutating elements if you really
want to. The only real cost is if you do this:

t[0] += ["mutated"]

you get an error and a mutation. Certainly a gotcha, but I wouldn't
describe it as a bug: it's pretty much unavoidable if you want to avoid
treating tuples as a special case.

Right, this bug is by design. You learned that phrase from a guy in
Redmond?

No. It merely means that just because you think it is a bug doesn't make
it so. It may even mean that there is no perfect solution, that *any*
behaviour on tuples containing mutable objects will be considered broken
by some people under some circumstances.

Trade-offs in software design are inevitable. There's little gain from
your proposal, and some cost, and the problem you are trying to solve
isn't a problem in practice. So if it's a bug, it's an insignificant one,
not worth fixing because the fix will invariably cause more problems than
the bug.

 

[Hrvoje Niksic]

What? Take a look at the code again:

mytuple[0] += 1

should never attempt an __iadd__ on mytuple.

A sane parser would see this as:

tmp = mytuple.__getitem__(0)
tmp = tmp.__iadd__(1)
mytuple.__setitem__(0, tmp) # should this always raise an exception?

What do you mean by "a sane parser"? This is exactly what happens in
current Python. Since tuple's __setitem__ always raises an exception
(and that is by design and not likely to change), everything is sane.

Saner (in this respect) behavior in the tuple example would require a
different protocol. I don't understand why Python doesn't just call
__iadd__ for side effect if it exists. The decision to also rebind
the result of __i*__ methods continues to baffle me. I guess it is a
result of a desire to enable the __i*__ methods to punt and return a
different instance after all, but if so, that design decision brings
more problems than benefits.

 

[Marc 'BlackJack' Rintsch]

Saner (in this respect) behavior in the tuple example would require a
different protocol. I don't understand why Python doesn't just call
__iadd__ for side effect if it exists. The decision to also rebind the
result of __i*__ methods continues to baffle me. I guess it is a result
of a desire to enable the __i*__ methods to punt and return a different
instance after all, but if so, that design decision brings more problems
than benefits.

How often do you use ``+=`` with mutable objects? I use it very often
with number types and sometimes with tuples, and there rebinding is
necessary. If I couldn't use it in this way: ``x = 0; x += z``, I'd
call that a bad design decision. It would be a quite useless operator
then.

Ciao,
Marc 'BlackJack' Rintsch

 

[Hrvoje Niksic]

How often do you use ``+=`` with mutable objects? I use it very
often with number types and sometimes with tuples, and there
rebinding is necessary. If I couldn't use it in this way: ``x = 0;
x += z``, I'd call that a bad design decision. It would be a quite
useless operator then.

Marc, I agree with you, I wasn't arguing that += shouldn't work for
immutable objects. If an object doesn't support __iadd__ at all (as
is already the case for numbers, strings, etc.), += should fall back
to __add__ followed by assignment. My point is that, *if they do*, it
would be better to just use __iadd__ and not attempt to also rebind.
Again, in pseudocode, I'd expect A += x to be implemented as:

lhsobj = A
if hasattr(lhsobj, '__iadd__'):
lhsobj.__iadd__(x) # lhsobj has __iadd__, just use it
else:
A = lhsobj.__add__(x) # fails if __add__ is not present either

That way tupleobj[0] += blah would work just fine for mutable objects
and would throw an exception for immutable objects. The current
situation where += first modifies the mutable object, and *then*
throws an exception feels like a design glitch.

 

[sturlamolden]

What do you mean by "a sane parser"?  This is exactly what happens in
current Python.  

Yes, but Steve Holden was suggesting mytuple.__iadd__ would be
invoked.

The decision to also rebind
the result of __i*__ methods continues to baffle me.  

Python methods always have a return value, even those that seem to do
not - they silently return None. Thus, __iadd__ must return self to
avoid rebinding to None.

 

[sturlamolden]

Python methods always have a return value, even those that seem to do
not - they silently return None. Thus, __iadd__ must return self to
avoid rebinding to None.

Except for immutable types, for which __iadd__ must return a new
instance.