bug or feature?

[beza1e1]

Coming back from a bug hunt, i am not sure what to think of this python
behaviour. Here is a demo program:

class A:
def __init__(self, lst=[]):
self.lst = lst

a = A()
b = A()
b.lst.append("hallo")
print a.lst # output: ["hallo"]

The point seems to be, that lst=[] creates a class attribute (correct
name?), which is shared by all instances of A. So a.lst ist the same
object as b.lst, despite the fact, that object a is different to object
b.

 

[Tomasz Lisowski]

Coming back from a bug hunt, i am not sure what to think of this python
behaviour. Here is a demo program:

class A:
def __init__(self, lst=[]):
self.lst = lst

a = A()
b = A()
b.lst.append("hallo")
print a.lst # output: ["hallo"]

The point seems to be, that lst=[] creates a class attribute (correct
name?), which is shared by all instances of A. So a.lst ist the same
object as b.lst, despite the fact, that object a is different to object
b.

It is an *instance attribute* by nature, since it does not reside in the
class object, but only in its instances. The truth is, that a.lst and
b.lst point to the same memory object, so it seems to behave much like
the class attribute

It is no more different from the simple fact, that two variables
(attributes) may point to the same memory object, like you see below:

a = b = []
a.append("hallo")
print b #output: ["hallo"]

In fact, it is usually a bad practice to assign instance attributes a
reference to the compound variable, existing in an external scope. Example:

aList = []

class A:
def __init__(self, lst): #no default attribute!
self.lst = lst

a = A(aList)
aList.append("hallo")
print a.lst #output: ["hallo"]

and your default value (, lst=[]) IS such an variable! The bad thing is,
that the value of the instance attribute 'lst' (example above) depends
on the external variable, which may be independently modified, thus
modifying unexpectedly the instance attribute. The safer approach, of
course is to write:

class A:
def __init__(self, lst): #no default attribute!
self.lst = lst[:] #take a copy

Summing up, is it an error, or a feature? I would say - a feature.
Everyone should be aware, that the argument default values are evaluated
once, and the same value (memory object) is reused at each instance
creation.

Best regards,
Tomasz Lisowski

 

[Simon Percivall]

Python.org General FAQ 1.4.21: Why are default values shared between
objects?
(http://www.python.org/doc/faq/general.html#why-are-default-values-shared-between-objects)

 

[skip]

beza1e1> class A:
beza1e1> def __init__(self, lst=[]):
beza1e1> self.lst = lst

Lists are mutable and default args are only evaluated once, at function
definition. If you want independent default args use:

class A:
def __init__(self, lst=None):
if lst is None:
lst = []
self.lst = lst

The same scheme would work for other mutable types (dicts, sets, etc).

This same question gets asked once a month or so. I'm sure this is in the
Python FAQ (check the website), but it was faster to reply than to look it
up...

Skip

 

[Steve Holden]

Coming back from a bug hunt, i am not sure what to think of this python
behaviour. Here is a demo program:

class A:
def __init__(self, lst=[]):
self.lst = lst

a = A()
b = A()
b.lst.append("hallo")
print a.lst # output: ["hallo"]

The point seems to be, that lst=[] creates a class attribute (correct
name?), which is shared by all instances of A. So a.lst ist the same
object as b.lst, despite the fact, that object a is different to object
b.

Interestingly I couldn't find this in the FAQ, though it *is* a
frequently-asked question [note: my not finding it doesn't guarantee
it's not there]. The nearest I could get was in


http://www.python.org/doc/faq/programming.html#my-program-is-too-slow-how-do-i-speed-it-up

which says:

"""Default arguments can be used to determine values once, at compile
time instead of at run time."""

The point is that the value of the keyword argument is determined when
the def statement is executed (which is to say when the function body is
being bound to its name).

If the default argument is (a reference to) a mutable object (such as a
list instance) then if one call to the function modifies that mutable
object, subsequent calls see the mutated instance as the default value.

regards
Steve

 

[Fredrik Lundh]

Interestingly I couldn't find this in the FAQ, though it *is* a
frequently-asked question [note: my not finding it doesn't guarantee
it's not there].

it's there:

http://www.python.org/doc/faq/general.html#why-are-default-values-shared-between-objects

(maybe "default values" should be changed to "default argument values")

it's also mentioned in chapter 4 of the tutorial:

http://docs.python.org/tut/node6.html#SECTION006710000000000000000

"*Important warning*: The default value is evaluated only once. This
makes a difference when the default is a mutable object such as a list,
dictionary, or instances of most classes. "

(the text then illustrates this with examples, and shows how to do things
instead)

and in the description of "def" in the language reference:

http://docs.python.org/ref/function.html

"*Default parameter values are evaluated when the function definition
is executed*. This means that the expression is evaluated once, when the
function is defined, and that that same "pre-computed" value is used for
each call. This is especially important to understand when a default para-
meter is a mutable object, such as a list or a dictionary: if the function
modifies the object (e.g. by appending an item to a list), the default
value is in effect modified."

(the text then shows how to do things instead)

</F>

 

[beza1e1]

Thanks for you answer! This copy trick is the most elegant solution i
think.

 

[Ben Sizer]

it's also mentioned in chapter 4 of the tutorial:

http://docs.python.org/tut/node6.html#SECTION006710000000000000000

"*Important warning*: The default value is evaluated only once. This
makes a difference when the default is a mutable object such as a list,
dictionary, or instances of most classes. "

Perhaps it would be a good idea if Python actually raised a warning
(SyntaxWarning?) if you use an unnamed list or dict as a default
argument. This would doubtless help quite a few beginners. And for
people who really do want that behaviour, working around the warning
should involve minimal extra code, with extra clarity thrown in for
free.

 

[Steven D'Aprano]

Coming back from a bug hunt, i am not sure what to think of this python
behaviour.

[snip code]

The point seems to be, that lst=[] creates a class attribute (correct
name?), which is shared by all instances of A. So a.lst ist the same
object as b.lst, despite the fact, that object a is different to object
b.

Not a bug, but not really a feature as such, it is a side-effect of
the way Python works. I guess that makes it a gotcha.

Argument defaults are set at compile time. You set the argument default to
a mutable object, an empty list. Every time you call the function,
you are appending to the same list.

This is not a problem if your argument default is a string, or a number,
or None, since these are all immutable objects that can't be changed.

I suppose someone might be able to come up with code that deliberately
uses this feature for good use, but in general it is something you want to
avoid. Instead of:

def spam(obj, L=[]):
L.append(obj)

do this:

def spam(obj, L=None):
if L is None: L = []
L.append(obj)

 

[Steve Holden]

beza1e1:

Coming back from a bug hunt, i am not sure what to think of this python
behaviour. Here is a demo program:

class A:
def __init__(self, lst=[]):
self.lst = lst

a = A()
b = A()
b.lst.append("hallo")
print a.lst # output: ["hallo"]

The point seems to be, that lst=[] creates a class attribute (correct
name?), which is shared by all instances of A. So a.lst ist the same
object as b.lst, despite the fact, that object a is different to object
b.

Steve Holden wrote:

Interestingly I couldn't find this in the FAQ, though it *is* a
frequently-asked question [note: my not finding it doesn't guarantee
it's not there].

it's there:

http://www.python.org/doc/faq/general.html#why-are-default-values-shared-between-objects

(maybe "default values" should be changed to "default argument values")

I couldn't believe it wasn't, but you're right: it should be easier to
find, and a change of wording may do that.

regards
Steve

 

[Steve Holden]

Fredrik Lundh wrote:




Perhaps it would be a good idea if Python actually raised a warning
(SyntaxWarning?) if you use an unnamed list or dict as a default
argument. This would doubtless help quite a few beginners. And for
people who really do want that behaviour, working around the warning
should involve minimal extra code, with extra clarity thrown in for
free.

This would have to be extended to any mutable object. How does the
compiler know which objects are mutable?

This would not be a good change.

regards
Steve

 

[Fredrik Lundh]

I suppose someone might be able to come up with code that deliberately
uses this feature for good use

argument binding is commonly used for optimization, and to give simple
functions persistent storage (e.g. memoization caches).

more importantly, it's the standard pydiom for passing object *values* (of
any kind) into an inner scope:

x = something

def myfunc(arg, x=x):
# myfunc needs the current value, not whatever x
# happens to be when the function is called

here's a typical gotcha:

for i in range(10):
def cb():
print "slot", i, "fired"
register_callback(slot=i, callback=cb)

to make this work as expected, you have to do

for i in range(10):
def cb(i=i):
print "slot", i, "fired"
register_callback(slot=i, callback=cb)

</F>