Empty list as default parameter

[Alex Panayotopoulos]

Hello all,

Maybe I'm being foolish, but I just don't understand why the following
code behaves as it does:

- = - = - = -

class listHolder:
def __init__( self, myList=[] ):
self.myList = myList

def __repr__( self ): return str( self.myList )

# debug: 'a' should contain 42, 'b' should be empty. But no.
a = listHolder()
a.myList.append( 42 )
b = listHolder()
print a
print b

- = - = - = -

I was expecting to see [42] then [], but instead I see [42] then [42]. It
seems that a and b share a reference to the same list object. Why?

 

[Fredrik Lundh]

Maybe I'm being foolish, but I just don't understand why the following
code behaves as it does:

- = - = - = -

class listHolder:
def __init__( self, myList=[] ):
self.myList = myList

def __repr__( self ): return str( self.myList )

# debug: 'a' should contain 42, 'b' should be empty. But no.
a = listHolder()
a.myList.append( 42 )
b = listHolder()
print a
print b

- = - = - = -

I was expecting to see [42] then [], but instead I see [42] then [42]. It
seems that a and b share a reference to the same list object. Why?

the default value expression is evaluated once, when the function
object is created, and the resulting object is bound to the argument.

if you want to create a new object on every call, you have to do
that yourself:

def __init__( self, myList=None):
if myList is None:
myList = [] # create a new list
self.myList = myList

or perhaps:

def __init__( self, myList=None):
self.myList = myList or []

see the description of the "def" statement for more info:

http://www.python.org/doc/current/ref/function.html

</F>

 

[Thorsten Pferdekämper]

class listHolder:
def __init__( self, myList=[] ):
self.myList = myList

def __repr__( self ): return str( self.myList )

# debug: 'a' should contain 42, 'b' should be empty. But no.
a = listHolder()
a.myList.append( 42 )
b = listHolder()
print a
print b

- = - = - = -

I was expecting to see [42] then [], but instead I see [42] then [42]. It
seems that a and b share a reference to the same list object. Why?

Hi,
AFAIK, the default parameter values are only instantiated once. So, the
default-myList is always the same object. The coding above should be
rewritten like...

class listHolder:
def __init__( self, myList=None ):
if myList = None:
self.myList = []
else:
self.myList = myList

Regards,
Thorsten

 

[anton muhin]

Hello all,

Maybe I'm being foolish, but I just don't understand why the following
code behaves as it does:

- = - = - = -

class listHolder:
def __init__( self, myList=[] ):
self.myList = myList

def __repr__( self ): return str( self.myList )

# debug: 'a' should contain 42, 'b' should be empty. But no.
a = listHolder()
a.myList.append( 42 )
b = listHolder()
print a
print b

- = - = - = -

I was expecting to see [42] then [], but instead I see [42] then [42]. It
seems that a and b share a reference to the same list object. Why?

Really common mistake: lists are _mutable_ objects and self.myList
references the same object as the default parameter. Therefore
a.myList.append modifies default value as well. Mutable defaults are
better avoided (except for some variants of memo pattern). Standard
trick is:

def __init__(self, myList = None):
if myList is None:
self.myList = []
else:
self.myList = myList

regards,
anton.

 

[Alex Panayotopoulos]

Really common mistake: lists are _mutable_ objects and self.myList
references the same object as the default parameter. Therefore
a.myList.append modifies default value as well.

It does?!
Ah, I've found it: listHolder.__init__.func_defaults

Hmm... this behaviour is *very* counter-intuitive. I expect that if I were
to define a default to an explicit object...

def process_tree1( start=rootNode )

....then I should indeed be able to process rootNode through manipulating
start. However, if I define a default as a new instance of an object...

def process_tree2( myTree=tree() )

....then the function should, IMHO, create a new object every time it is
entered. (By having func_defaults point to tree.__init__, or summat.)

Was there any reason that this sort of behaviour was not implemented?

Mutable defaults are better avoided (except for some variants of memo
pattern). Standard trick is:

def __init__(self, myList = None):
if myList is None:
self.myList = []
else:
self.myList = myList

Thank you. I shall use this in my code. (Although I would have preferred a
trick that uses less lines!)

 

[anton muhin]

On Fri, 21 Nov 2003, anton muhin wrote:


Was there any reason that this sort of behaviour was not implemented?

It was discussed several times (search for it, if you're interested),
but I don't remeber details.

Mutable defaults are better avoided (except for some variants of memo
pattern). Standard trick is:

def __init__(self, myList = None):
if myList is None:
self.myList = []
else:
self.myList = myList

Thank you. I shall use this in my code. (Although I would have preferred a
trick that uses less lines!)

if you wish a one-linear, somehting like this might work:
self.myList = myList or []

regards,
anton.

 

[Andrei]

Alex Panayotopoulos wrote on Fri, 21 Nov 2003 13:26:13 +0000:

Hmm... this behaviour is *very* counter-intuitive. I expect that if I were
to define a default to an explicit object...

I think so too. It's documented as a Python pitfall. Here are some more:
http://zephyrfalcon.org/labs/python_pitfalls.html

def __init__(self, myList = None):
if myList is None:
self.myList = []
else:
self.myList = myList

Thank you. I shall use this in my code. (Although I would have preferred a
trick that uses less lines!)

You could use the quasi-ternary trick:

mylist = None
(mylist is None and [[]] or [mylist])[0] []
mylist = []
(mylist is None and [[]] or [mylist])[0] []
mylist = [3,4]
(mylist is None and [[]] or [mylist])[0]

[3, 4]

Not that big an improvement really .

--
Yours,

Andrei

=====
Mail address in header catches spam. Real contact info (decode with rot13):
(e-mail address removed). Fcnz-serr! Cyrnfr qb abg hfr va choyvp cbfgf. V ernq
gur yvfg, fb gurer'f ab arrq gb PP.

 

[Robin Munn]

It does?!
Ah, I've found it: listHolder.__init__.func_defaults

Hmm... this behaviour is *very* counter-intuitive. I expect that if I were
to define a default to an explicit object...

def process_tree1( start=rootNode )

...then I should indeed be able to process rootNode through manipulating
start. However, if I define a default as a new instance of an object...

def process_tree2( myTree=tree() )

...then the function should, IMHO, create a new object every time it is
entered. (By having func_defaults point to tree.__init__, or summat.)

Was there any reason that this sort of behaviour was not implemented?

The reason for this behavior lies in the fact that Python is an
interpreted language, and that the class and def keywords are actually
considered statements. One creates a class object with a certain name,
the other creates a function object with a certain name. The def (or
class) statement is executed as soon as the entire function code or
class code has been parsed -- in other words, as soon as the interpreter
drops out of the indentation block of the def or class statement.
Therefore, any default values passed to function arguments in a def
statement will be evaluated once and only once, when the def statement
is executed.

Look at this, for example:


n = 5
def f(x = n):
return x
n = 3

print n # Prints 3
print f() # Prints 5

Note that the default argument to f() is the value of n when the def
statement was executed, not the value of n when f() is called.

 

[Dennis Lee Bieber]

Alex Panayotopoulos fed this fish to the penguins on Friday 21 November
2003 05:26 am:

Was there any reason that this sort of behaviour was not implemented?

Because, to be simplistic, the def statement is not a declaration, it
is an executable statement. Execution of def results in the generation
of a function object, which requires evaluating argument definitions.
Execution of def occurs /once/, during the initial load of the
module/file.

This is much different from things like old BASICs, where calling a
function causes the interpreter to rescan the source file to find the
function declaration.

Others can explain it much better.

--

 

[Peter Otten]

On Fri, 21 Nov 2003, anton muhin wrote:


Was there any reason that this sort of behaviour was not implemented?

It was discussed several times (search for it, if you're interested),
but I don't remeber details.

Mutable defaults are better avoided (except for some variants of memo
pattern). Standard trick is:

def __init__(self, myList = None):
if myList is None:
self.myList = []
else:
self.myList = myList

Thank you. I shall use this in my code. (Although I would have preferred
a trick that uses less lines!)

if you wish a one-linear, somehting like this might work:
self.myList = myList or []

This is dangerous, don't do it.
'just a marker'

seems to work. But:

[] or "just a marker"

'just a marker'

I. e. your one-liner will create a new list when the myList argument is not
provided or is provided and bound to an empty list.

Peter

 

[Peter Otten]

or perhaps:

def __init__( self, myList=None):
self.myList = myList or []

This treats empty and non-empty myList args differently:

mylist = []
(mylist or []) is mylist False
mylist = [1]
(mylist or []) is mylist

True

To be consistent you could do

self.myList = myList[:] or []

mylist = []
(mylist[:] or []) is mylist False
mylist = [1]
(mylist[:] or []) is mylist

False

or avoid the above pattern altogether.

Peter

 

[Alex Panayotopoulos]

It was discussed several times (search for it, if you're interested),

I have done so. It seems to be one of the most popular FAQs... 8^)

As I see it, this is not just a 'wart'; it's a major pitfall that seems
entirely out of place in python; everything up til now has made sense.[0]

The argument that this behaviour is useful for counters and such may have
been true in the past, but this was only ever a pseudo-generator hack. Now
we have real generators I don't see when you'd ever want to write
"foo(x=[])".

If it were up to me, I'd have def throw a warning wherever it sees a
square bracket, curly bracket or call to an __init__... But then I'm not a
python developer... 8^P 8^)

Okay, I've whinged enough, I think. No more posts from me on this subject.
8^)

if you wish a one-linear, somehting like this might work:
self.myList = myList or []

Thanks. That's sneaky without being confusing... I like.

 

[Alex Panayotopoulos]

On Fri, 21 Nov 2003, Peter Otten wrote:

[...]

To be consistent you could do

self.myList = myList[:] or []

TypeError. You can't slice None.
If you're going to be taking copies on entry, you might as well use

def __init__( self, myList=[] ):
self.myList = myList[:]

However, copying is inefficient. Not a problem in most cases, but
I've got a lot of objects to initialise. (This is for a GA). Solution: use
"myList or []", but carefully.

 

[Peter Otten]

On Fri, 21 Nov 2003, Peter Otten wrote:

[...]

To be consistent you could do

self.myList = myList[:] or []

TypeError. You can't slice None.
If you're going to be taking copies on entry, you might as well use

def __init__( self, myList=[] ):
self.myList = myList[:]

That was the the idea; if you are copying anyway, the [] default value does
no harm. Shame on me for not making this clear :-(

However, copying is inefficient. Not a problem in most cases, but
I've got a lot of objects to initialise. (This is for a GA). Solution: use
"myList or []", but carefully.

If you are mutating the myList attribute, the changes propagate to the list
given as a parameter only if that parameter was a non-empty list. Such a
behaviour will probably puzzle anyone but yourself (and yourself in a few
months), and what was gained by the shorter expression will be wasted on
documentation and/or debugging. Personally, I would avoid it even in
throwaway scripts.

Peter

 

[Stian =?iso-8859-1?Q?S=F8iland?=]

* Robin Munn spake thusly:

Look at this, for example:


n = 5
def f(x = n):
return x
n = 3

print n # Prints 3
print f() # Prints 5

Note that the default argument to f() is the value of n when the def
statement was executed, not the value of n when f() is called.

Wouldn't it be more logical for a programmer that x should evaluate
to '3' inside f()?

I can't see what is the purpose of binding default variables at
definition time instead of runtime.

I know perfectly well of the "param is None"-trick - and I've used it
far too often. I've never had any use of early binding of default
parameters except when making a remembering-function-for-fun:

>>> def remember(value=None, list=[]):

... if value is None:
... return list
... else:
... list.append(value)
... [1, 'Hello']

This example is in LISP presented as a way to do object orientation
without extending LISP. In Python it is an example of when you should
used an object instead.

Default variables should be meant as local names that can be overridden.
LISP, default variables are evaluated at runtime:


; This function just returns a new list, but prints
; "New" each time
[58]> (defun newlist () (print 'New ) ())
NEWLIST

; this function takes an optional parameter mylist, if it
; is not supplied, newlist is called and assigned to mylist
; The function returns mylist.
[59]> (defun getlist (&optional (mylist (newlist))) mylist)
GETLIST

; note how newlist() is called
[60]> (getlist)
NEW
NIL

; each time
[61]> (getlist)
NEW
NIL

; but not when the parameter is supplied
[62]> (getlist ())
NIL


This does not work in Python:
... print "New"
... return []
... ... return mylist
...
New

>>> getlist() []
>>> getlist()

[]

As one could see, newlist is called at definition time, and only once.

I think that default parameters should be evaluated each time the
function is called and the parameter is not supplied. This is a major
change, so it has to be delayed until 3.0 (possibly enabled by
__future__-imports)

 

[Fredrik Lundh]

However, if I define a default as a new instance of an object...

def process_tree2( myTree=tree() )

...then the function should, IMHO, create a new object every time it is
entered. (By having func_defaults point to tree.__init__, or summat.)

point to tree.__init__ ?

you haven't spent much time thinking about this, have you?

</F>

 

[Dennis Lee Bieber]

Stian Søiland fed this fish to the penguins on Saturday 22 November
2003 13:19 pm:

* Robin Munn spake thusly:

Wouldn't it be more logical for a programmer that x should evaluate
to '3' inside f()?

I can't see what is the purpose of binding default variables at
definition time instead of runtime.

Look at it as if it were:

n = 5
x = n
n = 3

print n
print x

The binding of the "default" is done "outside" of the function
definition itself, using the environment in place at that moment. And
since all Python variables are, in a way, Post-It notes stuck onto the
value, rather than fixed boxes into which a value is placed, the
behavior is understandable -- what you ask for would require
dynamically interpreting (the equivalent of exec() perhaps) the
argument list on every invocation.

Granted, I've never seen an example using the above structure -- in
those languages that allow for default values the default has always
been a constant, and that is about what Python does -- if you look on
the default assignment as being a C-style macro expansion (x = n
becomes x = 5).

If you are relying on a "default" that changes based upon the
execution environment then, to me, it isn't a default and you might as
well code it explicitly within the function (which puts you back into
the test for None situation).

--

 

[Bengt Richter]

It does?!
Ah, I've found it: listHolder.__init__.func_defaults

Hmm... this behaviour is *very* counter-intuitive. I expect that if I were
to define a default to an explicit object...

def process_tree1( start=rootNode )

...then I should indeed be able to process rootNode through manipulating
start. However, if I define a default as a new instance of an object...

def process_tree2( myTree=tree() )

...then the function should, IMHO, create a new object every time it is
entered. (By having func_defaults point to tree.__init__, or summat.)

Was there any reason that this sort of behaviour was not implemented?

Yes. The bindings of default values for call args are evaluated at define-time,
in the context of the definition, not at execution time, when the function is called.

If you want to specify what to call at execution time, you have to call it at execution time,
so you either have to pass a reference to the thing to call ('tree' in this case) or have it
otherwise visible from inside the function, e.g., the interface could be something like

def process_tree2(treemaker=tree):
myTree = treemaker()
...

Now if you wanted to pass a pre-existing tree to this kind of process_tree2, you'd have to
pass a callable that would execute to produce the tree, e.g., call it like

temp_treemaker = lambda: rootNode
process_tree2(temp_treemaker) # or just put the lambda expression right in the call

More likely, you'd want to be able to accept a tree, or make one if nothing was passed. E.g.,

def process_tree3(myTree=None):
if myTree is None: myTree = tree() # here tree must be visible in an enclosing scope
... # -- usually global, but not necessarily

a variant would be to have a default value of a single particular tree, as in your first example,
and then test whether something executable, like tree (not tree()) was being passed, e.g.,

def process_tree1( start=rootNode ):
if callable(start): start = start() # e.g., if called like process_tree1(tree)

Of course, if a node object is also callable, you have to make a different check ;-)

Mutable defaults are better avoided (except for some variants of memo
pattern). Standard trick is:

def __init__(self, myList = None):
if myList is None:
self.myList = []
else:
self.myList = myList

Thank you. I shall use this in my code. (Although I would have preferred a
trick that uses less lines!)

How about two less? I usually do it (with no trick ;-) like:

def __init__(self, myList = None):
if myList is None: myList = []
self.myList = myList

Regards,
Bengt Richter

 

[Paul Rubin]

Yes. The bindings of default values for call args are evaluated at
define-time, in the context of the definition, not at execution
time, when the function is called.

That's always seemed like a source of bugs to me, and ugly workarounds
for the bugs. Is there any corresponding disadvantage to eval'ing the
defaults at runtime as is done in other languages?

 

[Bengt Richter]

That's always seemed like a source of bugs to me, and ugly workarounds
for the bugs. Is there any corresponding disadvantage to eval'ing the
defaults at runtime as is done in other languages?

What languages are you thinking of? A concrete example for comparison would
clarify things. Would you have default expressions effectively passed in as
the bodies of lambdas (which might mean creating closures, depending on what was
referenced) and then executed to create the local bindings prior to the first line in
a function or method? It would certainly be inefficient for all the cases where you
just wanted a static default (unless you special cased those to work as now -- but
remember, only bare names and constant literals could be special cased that way. An
expression like os.RD_ONLY (yes that is an expression!) would have to be passed as
lambda: os.RDONLY). So you'd have to counter that by making bare-name bindings prior
to calls, like tmp_mode=os.RD_ONLY; os.open('foo.txt', tmp_mode); #etc

I expect the use cases balance favor the current methodology. A cursory grep though
defs of /lib/*py shows no defaults at all to be the majority (>80%?) and of the defs
with '=' in the same line (983 of 5443), most seem to be =None or =<some int> or =<some string>
to the tiring eyeball. I didn't process it.

Regards,
Bengt Richter