Trying to understand += better

[Roy Smith]

If I've got an object foo, and I execute:

foo.bar += baz

exactly what happens if foo does not have a 'bar' attribute? It's
pretty clear that foo.__getattr__('bar') gets called first, but it's a
little murky after that. Assume for the moment that foo.__getattr__
('bar') returns an object x. I think the complete sequence of calls
is:

foo.__getattr__('bar') ==> x
x.__add__(baz) ==> y
foo.__setattr__('bar', y)

but I'm not 100% sure. It would be nice if it was, because that would
let me do some very neat magic in a system I'm working on

How would things change if X defined __iadd__()?

 

[Lie Ryan]

If I've got an object foo, and I execute:

foo.bar += baz

exactly what happens if foo does not have a 'bar' attribute? It's
pretty clear that foo.__getattr__('bar') gets called first, but it's a
little murky after that. Assume for the moment that foo.__getattr__
('bar') returns an object x. I think the complete sequence of calls
is:

foo.__getattr__('bar') ==> x
x.__add__(baz) ==> y
foo.__setattr__('bar', y)

but I'm not 100% sure. It would be nice if it was, because that would
let me do some very neat magic in a system I'm working on

How would things change if X defined __iadd__()?

The semantic of the in-place operator is something like:
x += y
becomes
x = x.__iadd__(y)

thus
foo.bar += baz
becomes
foo.bar = foo.bar.__iadd__(baz)

So the call sequence is,
foo.__getattr__('bar') ==> x
x.__iadd__(baz) ==> y
foo.__setattr__('bar', y)

the default definition of object.__iadd__ is something like this:
def __iadd__(self, other):
# this calls self.__add__ or other.__radd__ according to the
# operator call rule, may call __coerce__ or any other magics
# in operator calling
return self + other

 

[Steve Howell]

The semantic of the in-place operator is something like:
x += y
becomes
x = x.__iadd__(y)

thus
foo.bar += baz
becomes
foo.bar = foo.bar.__iadd__(baz)

So the call sequence is,
foo.__getattr__('bar') ==> x
x.__iadd__(baz) ==> y
foo.__setattr__('bar', y)

the default definition of object.__iadd__ is something like this:
def __iadd__(self, other):
     # this calls self.__add__ or other.__radd__ according to the
     # operator call rule, may call __coerce__ or any other magics
     # in operator calling
     return self + other

The __iadd__ method will often return self for mutable types. So, for
example, these two statements are NOT identical where lst is a list:

lst = lst + [3]
lst += [3]

The first statement is creating a whole new list; the second one
isn't.


http://docs.python.org/reference/datamodel.html

 

[Roy Smith]

The semantic of the in-place operator is something like:
x += y
becomes
x = x.__iadd__(y)

thus
foo.bar += baz
becomes
foo.bar = foo.bar.__iadd__(baz)

So the call sequence is,
foo.__getattr__('bar') ==> x
x.__iadd__(baz) ==> y
foo.__setattr__('bar', y)

I don't get where the __setattr__() call comes from in this situation. I
thought the whole idea of __iadd__(self, other) is that it's supposed to
mutate self. So, why is there another assignment happening after the
__iadd__() call?

 

[n00m]

The first statement is creating a whole new list;

Yes but *imo* not quite exactly so.
We can't think of 2 lists as of absolutely independent
things.

.... x = [[0]]
.... id(x)
19330632
.... id(x[0])
19316608
.... z = x + [3]
.... id(z)
19330312
.... id(z[0])
19316608
....
.... z[0] is x[0] # ?
True
.... x[0][0] = 1
....
.... z[0][0]
1

 

[Steve Howell]

The first statement is creating a whole new list;

Yes but *imo* not quite exactly so.
We can't think of 2 lists as of absolutely independent
things.
[...]

You are correct that two lists can both have the same mutable object
as items, and if you mutate that object, both lists will see that
mutation. Changing the innards of an item doesn't change the list, if
you think of the list as just a collection of ids, but your point is
well taken. It is probably easiest to understand all this with a more
elaborate example.

mutable = {'foo': 'bar'}
list1 = [mutable, 'bla']
list2 = list1 + ['another element']
list1 [{'foo': 'bar'}, 'bla']
list2

[{'foo': 'bar'}, 'bla', 'another element']

So list2 and list1 are no longer the same list, but...

mutable['foo'] = 'new value for mutable'
list1 [{'foo': 'new value for mutable'}, 'bla']
list2

[{'foo': 'new value for mutable'}, 'bla', 'another element']

They do still share a common element, as shown above.

But you can reassign the first element of list2 without affecting
list1:

list2[0] = 'only list 2'
list1 [{'foo': 'new value for mutable'}, 'bla']
list2

['only list 2', 'bla', 'another element']

Now look at fred_list and barney_list below. Since you use +=,
fred_list and barney_list stay tied together even when you *think* you
are only assigning a new value to barney_list[0].

fred_list = [0]
barney_list = fred_list
barney_list += [1]
barney_list [0, 1]
fred_list [0, 1]
barney_list[0] = 'barney'
barney_list ['barney', 1]
fred_list

['barney', 1]

 

[Steve Howell]

I don't get where the __setattr__() call comes from in this situation.  I
thought the whole idea of __iadd__(self, other) is that it's supposed to
mutate self.  So, why is there another assignment happening after the
__iadd__() call?

Non-mutable types can also use += syntax.

x = MagicalNumber(42)
x += 5 # x gets reassigned to MagicalNumber(47)

There is nothing that says that __iadd__ has to return self, but if
you are designing a mutable type, you will generally do that.

http://docs.python.org/reference/datamodel.html

'''
These methods are called to implement the augmented arithmetic
assignments (+=, -=, *=, /=, //=, %=, **=, <<=, >>=, &=, ^=, |=).
These methods should attempt to do the operation in-place (modifying
self) and return the result (which could be, but does not have to be,
self). If a specific method is not defined, the augmented assignment
falls back to the normal methods. For instance, to execute the
statement x += y, where x is an instance of a class that has an
__iadd__() method, x.__iadd__(y) is called. If x is an instance of a
class that does not define a __iadd__() method, x.__add__(y) and
y.__radd__(x) are considered, as with the evaluation of x + y.
'''

 

[Lie Ryan]

I don't get where the __setattr__() call comes from in this situation. I
thought the whole idea of __iadd__(self, other) is that it's supposed to
mutate self. So, why is there another assignment happening after the
__iadd__() call?

The formal name of the __iop__ is "agumented assignment". The name
doesn't talk about in-place; but it does talk about assignment. The
semantic defines that augmented assignment operation is done in-place
*when the left-hand side object supports it* (i.e. it does not require
in-place mutation).

"""
The __iadd__ hook should behave similar to __add__, returning the
result of the operation (which *could* be `self') which is to be
assigned to the variable `x'.
"""

For a more complete description, see:
http://www.python.org/dev/peps/pep-0203/

 

[Terry Reedy]

Except that the expression x is evaluated just once instead of twice.

I don't get where the __setattr__() call comes from in this situation.

Augmented *ASSIGNMENT* is a type of assignment.

The dis module can be used to see what CPython does. foo.bar += baz
2 0 LOAD_GLOBAL 0 (foo)
3 DUP_TOP
4 LOAD_ATTR 1 (bar)
7 LOAD_GLOBAL 2 (baz)
10 INPLACE_ADD
11 ROT_TWO
12 STORE_ATTR 1 (bar)
....
This amounts to what Roy said, with x and y being temporary entries on
the stack.

Terry Jan Reedy