Bizarre method keyword-arg bug.

[Jasper]

I'm stumped. I'm calling a method that has keyword args, but not
setting them, and yet one of them starts off with data?!

The class definition begins like so:

class BattleIntentionAction( BattleAction ):
def __init__( self, factionName, location, tactic='hold',
targetFacName='', terrainArgs=[], garrisonIds=[] ):
self.terrainArgs = terrainArgs
print terrainArgs

The constructor is called somewhere else, like so:
act = BattleIntentionAction( facName, self.location )


During this object's construction, terrainArgs is set to a list with
values corresponding to a previously created BattleIntentionAction!
Even more bizarre, the terrainArgs param is a testing formality, and
doesn't actually get used anywhere in my code -- the corresponding
attribute is always modified after object creation. Furthermore, this
doesn't happen with the other keyword args...

Obviously, I'm glossing over a ton of code here, but I'm having a
tough time isolating this problem, as it seems to be very dependent on
events leading up to it. It feels like the sort of memory stomping
bug I remember seeing from days of yore when I hacked C++. :-(


I frankly don't understand how "terrainArgs" can have a value if
nothing is passed for it on the calling invocation, short of some
obscure compiler bug (this is Python 2.4.3). Am I being naive? Is
there some way I could be bringing this about myself?

I can easily work around this weirdness by having the caller set
terrainArgs explicitly, but I can't shake the sensation that this
"fix" just masks some deeper flaw in my code.


Arg!
-Jasper

 

[Simon Brunning]

I'm stumped. I'm calling a method that has keyword args, but not
setting them, and yet one of them starts off with data?!

<http://www.python.org/doc/faq/general/#why-are-default-values-shared-between-objects>

--
Cheers,
Simon B.
(e-mail address removed)
http://www.brunningonline.net/simon/blog/
GTalk: simon.brunning | MSN: small_values | Yahoo: smallvalues | Twitter: brunns

 

[Fredrik Lundh]

I'm stumped. I'm calling a method that has keyword args, but not
setting them, and yet one of them starts off with data?!

The class definition begins like so:

class BattleIntentionAction( BattleAction ):
def __init__( self, factionName, location, tactic='hold',
targetFacName='', terrainArgs=[], garrisonIds=[] ):
self.terrainArgs = terrainArgs
print terrainArgs

The constructor is called somewhere else, like so:
act = BattleIntentionAction( facName, self.location )

During this object's construction, terrainArgs is set to a list with
values corresponding to a previously created BattleIntentionAction!

default argument values are evaluated when the function object is
created (by the "def" statement, that is), not when the resulting
function is called. if you mutate the default values, the mutations
will stick.

this is explained in the FAQ, the tutorial, and the reference manual,
and hopefully in your favourite python book as well; see e.g.

http://docs.python.org/tut/node6.html#SECTION006710000000000000000
http://docs.python.org/ref/function.html

</F>

 

[Diez B. Roggisch]

I'm stumped. I'm calling a method that has keyword args, but not
setting them, and yet one of them starts off with data?!

The class definition begins like so:

class BattleIntentionAction( BattleAction ):
def __init__( self, factionName, location, tactic='hold',
targetFacName='', terrainArgs=[], garrisonIds=[] ):
self.terrainArgs = terrainArgs
print terrainArgs

The constructor is called somewhere else, like so:
act = BattleIntentionAction( facName, self.location )


During this object's construction, terrainArgs is set to a list with
values corresponding to a previously created BattleIntentionAction!
Even more bizarre, the terrainArgs param is a testing formality, and
doesn't actually get used anywhere in my code -- the corresponding
attribute is always modified after object creation. Furthermore, this
doesn't happen with the other keyword args...

Obviously, I'm glossing over a ton of code here, but I'm having a
tough time isolating this problem, as it seems to be very dependent on
events leading up to it. It feels like the sort of memory stomping
bug I remember seeing from days of yore when I hacked C++. :-(


I frankly don't understand how "terrainArgs" can have a value if
nothing is passed for it on the calling invocation, short of some
obscure compiler bug (this is Python 2.4.3). Am I being naive? Is
there some way I could be bringing this about myself?

I can easily work around this weirdness by having the caller set
terrainArgs explicitly, but I can't shake the sensation that this
"fix" just masks some deeper flaw in my code.

This is a FAQ:

http://effbot.org/pyfaq/why-are-default-values-shared-between-objects.htm

Diez

 

[Fredrik Lundh]

default argument values are evaluated when the function object is
created (by the "def" statement, that is), not when the resulting
function is called. if you mutate the default values, the mutations
will stick.

and yes, workarounds and further details are provided here:

http://effbot.org/zone/default-values.htm

</F>

 

[Jasper]

<http://www.python.org/doc/faq/general/#why-are-default-values-shared-...>

Uggg! /That's/ an intuitive side-effect/wart. :-/

Thanks for sorting me out!

-Jasper

 

[Fredrik Lundh]

Uggg! /That's/ an intuitive side-effect/wart. :-/

it's done that way on purpose, of course, because evaluating a full
closure for each default argument at every call would greatly hurt
performance (and lead to another set of surprises, of course).

please don't label things that you don't understand and haven't spent
any time reflecting over as bugs or warts; that's disrespectful to the
designers and probably not good for your blood pressure.

</F>

 

[Jasper]

it's done that way on purpose, of course, because evaluating a full
closure for each default argument at every call would greatly hurt
performance (and lead to another set of surprises, of course).

please don't label things that you don't understand and haven't spent
any time reflecting over as bugs or warts; that's disrespectful to the
designers and probably not good for your blood pressure.

</F>

I understand it's done that way on purpose, and that there are
tradeoffs
involved, but frankly your /guess/ that I don't understand is wrong.
Having
used Python for some 15 years, I'm hardly a neophyte -- it's pure
serendipity
that this hasn't bitten me before.

I can see the elegance from a language design perspective, the speed
advantage,
etc. Nonetheless, it's an unintuitive wart, hurting Python's clarity
-- as evidence
I'll point out all the warnings that need to be sprinkled through the
various docs.
And no, the alternative /does not/ have an equivalent set of surprises
-- it's not
like Python is unique in having default arguments.

Frankly, if I wanted speed, I wouldn't be using python, and if I
wanted clever tricks,
I'd use Perl. Surprise caching as a side-effect is /very/ Perl-like.

-Jasper

 

[Paul Boddie]

it's done that way on purpose, of course, because evaluating a full
closure for each default argument at every call would greatly hurt
performance (and lead to another set of surprises, of course).

Having had the opportunity to reflect on this recently, I'd agree that
the current behaviour is probably the better outcome in many cases,
although one usually only sees people having problems with this when
using literals (lists mostly, and often empty lists), so there's
always the question of how people perceive those literals, whether
they consider them sufficiently "low cost" to be evaluated for each
call, and so on. Indeed, issues of binding don't apply to such
literals, and I imagine that this conceals the possibility of
surprising behaviour (in the general case with names which could refer
to different things at different times) and the rationale for
implementing a mechanism which is consequently less complicated (both
for the developers and in terms of predicting program behaviour).

please don't label things that you don't understand and haven't spent
any time reflecting over as bugs or warts; that's disrespectful to the
designers and probably not good for your blood pressure.

Well, in the page of "Python warts" that I compiled when it was
claimed that Python 3000 addresses such issues in Python 2.x, the
"Mutable default arguments" entry lists at least one experienced
Python author who agrees with the inquirer's assertion:

http://wiki.python.org/moin/PythonWarts

Paul

 

[Jasper]

Well, in the page of "Python warts" that I compiled when it was
claimed that Python 3000 addresses such issues in Python 2.x, the
"Mutable default arguments" entry lists at least one experienced
Python author who agrees with the inquirer's assertion:

http://wiki.python.org/moin/PythonWarts

Paul

Not surprising, as it's fairly non-standard. I'd even argue that
calling them "default arguments" is a misnomer -- they're more akin to
static variables.

It doesn't help that the solution to get the expected behavior
involves adding boiler-plate code all over.

-Jasper

 

[Paul Boddie]

Not surprising, as it's fairly non-standard.  I'd even argue that
calling them "default arguments" is a misnomer -- they're more akin to
static variables.

Indeed, default parameter values are occasionally suggested for that
purpose, although it has often been said that one shouldn't really use
them for that, either because there are often superior alternatives to
static variables, or because (as at least claimed in years gone by)
the behaviour may change one day. I think the latter explanation has
little substance now, at least for implementations compatible with
CPython.

It doesn't help that the solution to get the expected behavior
involves adding boiler-plate code all over.

Yes, it's a case of avoiding the full extent of the feature because it
doesn't do what one might expect. Personally, I only really use None,
numbers and strings as defaults, with the boilerplate you mention in
the function to get the initialisation that would have been provided
by the defaults. There is, however, a useful pattern which arises from
such a conservative approach: adopting None as a default (or perhaps a
special value) means that one has a way of explicitly indicating that
the default is desired, rather than omitting a parameter - something
which may not always be convenient. The boilerplate then loads the
appropriate default which may be stored in a more convenient location:
as a module global or in a class or instance attribute.

Ultimately, I suppose one could enforce some kind of least surprising
"best practice" by limiting default parameter values to being literals
of immutable objects or names, as opposed to expressions, thus
eliminating some potential confusion. Perhaps the various static code
checking tools provide guidance on this matter.

Paul

 

[Steven D'Aprano]

Ultimately, I suppose one could enforce some kind of least surprising
"best practice" by limiting default parameter values to being literals
of immutable objects or names, as opposed to expressions, thus
eliminating some potential confusion.

-1

Firstly, I *like* the ability to use mutable objects as default
arguments. I don't do it often, but when I do, I do it deliberately. I
find it useful.

Secondly, I think forbidding expressions as default arguments would be
far worse than the so-called "problem" you wish to fix. It would make
such simple default arguments as these unnecessarily complicated:

def foo(x=2**64, sentinel=object())

Please don't try to "fix" this feature.

 

[Steven D'Aprano]

It doesn't help that the solution to get the expected behavior involves
adding boiler-plate code all over.

Expected by who?

Please don't assume that everyone has had their intuition shaped by
exposure to the same languages yours has been shaped by. What surprises
you is obvious to me.

In a previous post, you asserted that the alternative behaviour (having
default arguments re-evaluated each time the function is called) can't
possibly be surprising. You wrote:

"And no, the alternative /does not/ have an equivalent set of surprises
-- it's not like Python is unique in having default arguments."

That's simply not true. I would find this behaviour very surprising, and
I bet you would too:

.... print obj
.... Traceback (most recent call last):
...
NameError: name 'x' is not defined

 

[Fredrik Lundh]

Having used Python for some 15 years, I'm hardly a neophyte -- it's pure
serendipity that this hasn't bitten me before.

Using languages and spending time reflecting over how they're put
together are two very different things.

</F>

 

[Fredrik Lundh]

Expected by who?

Blub programmers.

</F>

 

[Fredrik Lundh]

You may find the above surprising, but Common Lisp users expect the default
argument expression to be evaluated anew when need by a function call:

I find the Lisp approach more reasonable. Also, an argument based on
performance for Python's current behavior seems dubious, given the
language's other performance robbing design choices.

well, I'd say an argument based on "Common Lisp users" is a lot more
dubious ;-)

(and some of us are actually capable of writing pretty fast code in
Python. slowing the language down (or crippling it) because some blub
programmers screams "bug!" instead of looking things up in the handbook
when they stumble upon something they haven't seen before doesn't strike
me as a good use of anyone's time.)

</F>