Keyword arguments - strange behaviour?

Discussion in 'Python' started by brian.bird@securetrading.com, Dec 21, 2004.

  1. Guest

    Can anyone explain the behaviour of python when running this script?

    >>> def method(n, bits=[]):

    .... bits.append(n)
    .... print bits
    ....
    >>> method(1)

    [1]
    >>> method(2)

    [1, 2]

    It's the same in python 1.5, 2.3 and 2.4 so it's not a bug. But I
    expected the variable "bits" to be re-initialised to an empty list as
    each method was called. Whenever I explain optional keyword arguments
    to someone I have usually (wrongly as it turns out) said it is
    equivalent to:

    >>> def method(n, bits=None):

    .... if bits is None:
    .... bits=[]
    .... bits.append(n)
    .... print bits
    ....
    >>> method(1)

    [1]
    >>> method(2)

    [2]

    Is there a good reason why these scripts are not the same? I can
    understand how/why they are different, it's just not what I expected.
    (It seems strange to me that the result of the first method can only be
    determined if you know how many times it has already been called)

    Is this behaviour what you would (should?) intuitively expect?
    Thanks,

    Brian
     
    , Dec 21, 2004
    #1
    1. Advertising

  2. deelan Guest

    wrote:
    > Can anyone explain the behaviour of python when running this script?

    (...)
    >
    > Is there a good reason why these scripts are not the same? I can
    > understand how/why they are different, it's just not what I expected.
    > (It seems strange to me that the result of the first method can only be
    > determined if you know how many times it has already been called)


    see "python gotchas":
    <http://www.ferg.org/projects/python_gotchas.html#bct_sec_5>

    bye.

    --
    @prefix foaf: <http://xmlns.com/foaf/0.1/> .
    <#me> a foaf:person ; foaf:nick "deelan" ;
    foaf:weblog <http://www.netspyke.com/> .
     
    deelan, Dec 21, 2004
    #2
    1. Advertising

  3. wrote:

    > Can anyone explain the behaviour of python when running this script?


    the language reference has the full story:

    http://www.python.org/doc/2.4/ref/function.html

    "Default parameter values are evaluated when the function
    definition is executed"

    </F>
     
    Fredrik Lundh, Dec 21, 2004
    #3
  4. Duncan Booth Guest

    wrote:

    > Can anyone explain the behaviour of python when running this script?
    >
    >>>> def method(n, bits=[]):

    > ... bits.append(n)
    > ... print bits
    > ...
    >>>> method(1)

    > [1]
    >>>> method(2)

    > [1, 2]
    >
    > It's the same in python 1.5, 2.3 and 2.4 so it's not a bug. But I
    > expected the variable "bits" to be re-initialised to an empty list as
    > each method was called. Whenever I explain optional keyword arguments
    > to someone I have usually (wrongly as it turns out) said it is
    > equivalent to:


    <snipped erroneous comparison>

    No, it is closer to:

    # Assume you have done this earlier:
    import new
    def _realmethod(n, bits):
    bits.append(n)
    print bits

    # The def is roughly equivalent to this:
    _defaultArgs = ([], )
    method = new.function(_realmethod.func_code, globals(), 'method',
    _defaultArgs)

    Each time you re-execute the def you get a new set of default arguments
    evaluated, but the result of the evaluation is simply a value passed in to
    the function constructor.

    The code object is compiled earlier then def creates a new function object
    from the code object, the global dictionary, the function name, and the
    default arguments (and any closure as well, but its a bit harder to
    illustrate that this way).
     
    Duncan Booth, Dec 21, 2004
    #4
  5. Guest

    Re: Keyword arguments - strange behaviour?

    Thanks, this makes perfect sense. The phrase which sums it up neatly is
    "Default parameter values are evaluated when the function definition is
    executed"

    However, is there a good reason why default parameters aren't evaluated
    as the function is called? (apart from efficiency and backwards
    compatibility)? Is this something that's likely to stay the same in
    python3.0?

    I'm really looking for a neat way to do the following:

    def method(a,b,opt1=None,opt2=None,opt3="",opt4=None):
    if opt1 is None: opt1=[]
    if opt2 is None: opt2={}
    if opt4 is None: opt4=[]

    Python syntax is normally so neat but this just looks a mess if there
    are lots of parameters.
     
    , Dec 21, 2004
    #5
  6. Re: Keyword arguments - strange behaviour?

    <> wrote:

    > However, is there a good reason why default parameters aren't evaluated
    > as the function is called? (apart from efficiency and backwards compatibility)?


    how would you handle this case:

    def function(arg=otherfunction(value)):
    return arg

    </F>
     
    Fredrik Lundh, Dec 21, 2004
    #6
  7. Guest

    Re: Keyword arguments - strange behaviour?

    def function(arg=otherfunction(value)):
    return arg

    My expectation would have been that otherfunction(value) would be
    called if (and only if) the arg keyword parameter was missing from the
    function() call (ie. the optional value is evaluated the lazy way).
    Also, otherfunction would be called each and every time this function()
    is called without the arg keyword. (At least, I would have assumed this
    before today)

    Still, I can see why it's been implemented the way it has, it just
    seems a shame there isn't a neat shortcut to default lots of optional
    arguments to new mutable objects. And since I'm not the only one to
    fall into this trap it makes me wonder why the default behaviour isn't
    made to be what most people seem to expect?
     
    , Dec 21, 2004
    #7
  8. Re: Keyword arguments - strange behaviour?

    <> wrote:

    > def function(arg=otherfunction(value)):
    > return arg
    >
    > My expectation would have been that otherfunction(value) would be
    > called if (and only if) the arg keyword parameter was missing from the
    > function() call (ie. the optional value is evaluated the lazy way).


    what otherfunction? what value?

    </F>
     
    Fredrik Lundh, Dec 21, 2004
    #8
  9. Re: Keyword arguments - strange behaviour?

    Hi,

    I cannot see any strange behavior. this code works exacly as you and I
    suspect:

    >>> def otherfunction(x) :

    .... return x
    ....
    >>> def function(arg=otherfunction(5)) :

    .... return arg
    ....
    >>> function(3)

    3
    >>> function()

    5

    Or is this not what you excepted?

    - harold -

    On 21.12.2004, at 15:47, wrote:

    > def function(arg=otherfunction(value)):
    > return arg
    >
    > My expectation would have been that otherfunction(value) would be
    > called if (and only if) the arg keyword parameter was missing from the
    > function() call (ie. the optional value is evaluated the lazy way).
    > Also, otherfunction would be called each and every time this function()
    > is called without the arg keyword. (At least, I would have assumed this
    > before today)
    >
    > Still, I can see why it's been implemented the way it has, it just
    > seems a shame there isn't a neat shortcut to default lots of optional
    > arguments to new mutable objects. And since I'm not the only one to
    > fall into this trap it makes me wonder why the default behaviour isn't
    > made to be what most people seem to expect?
    >
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >
    >

    --
    Freunde, nur Mut,
    Lächelt und sprecht:
    Die Menschen sind gut --
    Bloß die Leute sind schlecht.
    -- Erich Kästner
     
    harold fellermann, Dec 21, 2004
    #9
  10. Steve Holden Guest

    Re: Keyword arguments - strange behaviour?

    harold fellermann wrote:

    > Hi,
    >
    > I cannot see any strange behavior. this code works exacly as you and I
    > suspect:
    >
    > >>> def otherfunction(x) :

    > .... return x
    > ....
    > >>> def function(arg=otherfunction(5)) :

    > .... return arg
    > ....
    > >>> function(3)

    > 3
    > >>> function()

    > 5
    >
    > Or is this not what you excepted?
    >


    Channelling the effbot, I think he was asking what namespace context you
    expected the expression "arg=otherfunction(x)" to be evaluated in when
    it's used at the time of a function call to dynamically create a new
    default value for arg.

    regards
    Steve
    --
    Steve Holden http://www.holdenweb.com/
    Python Web Programming http://pydish.holdenweb.com/
    Holden Web LLC +1 703 861 4237 +1 800 494 3119
     
    Steve Holden, Dec 21, 2004
    #10
  11. Re: Keyword arguments - strange behaviour?

    "harold fellermann" wrote:

    > I cannot see any strange behavior. this code works exacly as you and I suspect:


    you seem to have missed some of the posts that led up to the one you
    replied to. (most importantly, the first one).

    </F>
     
    Fredrik Lundh, Dec 21, 2004
    #11
  12. Re: Keyword arguments - strange behaviour?

    wrote:
    > However, is there a good reason why default parameters aren't evaluated
    > as the function is called? (apart from efficiency and backwards
    > compatibility)?


    So, one of my really common use cases that takes advantage of the fact
    that default parameters are evaluated at function definition time:

    def foo(bar, baz, matcher=re.compile(r'...')):
    ...
    text = matcher.sub(r'...', text)
    ...

    If default parameters were evaluated when the function was called, my
    regular expression would get re-compiled every time foo() was called.
    This would be inefficient, especially if foo() got called a lot. If
    Python 3000 changed the evaluation time of default parameters, I could
    rewrite this code as:

    class foo(object):
    matcher=re.compile(r'...')
    def __new__(self, bar, baz, matcher=None):
    if matcher is None:
    matcher = self.matcher
    ...
    text = matcher.sub(r'...', text)
    ...

    But that seems like a lot of work to do something that used to be pretty
    simple...

    Steve
     
    Steven Bethard, Dec 21, 2004
    #12
  13. Guest

    Re: Keyword arguments - strange behaviour?

    > Channelling the effbot, I think he was asking what namespace context
    you
    > expected the expression "arg=otherfunction(x)" to be evaluated in

    when
    > it's used at the time of a function call to dynamically create a new
    > default value for arg.


    Thanks, I realise now that's what was meant. I think I was just
    assuming otherfunction() would be globally available - but that's not
    something I want to go into since I can see now there will be problems
    trying to define what python should do if it evaluated defaults at the
    function call :)

    I think I can now explain to someone why there are good reasons that
    the default params are evaluated at the definition. However, I still
    can't give a nice looking solution on how to re-write a function to
    have empty mutable values as default arguments: eg.

    def method(a,b,opt1=[],opt2=None,opt3="",opt4={})

    How could I re-write this (especially if there are perhaps 20 optional
    parameters,any number of which may have mutable defaults) without
    writing 20 "if opt1 is None: opt1=[]" statements?

    Brian
     
    , Dec 23, 2004
    #13
  14. Fuzzyman Guest

    Re: Keyword arguments - strange behaviour?

    wrote:
    > > Channelling the effbot, I think he was asking what namespace

    context
    > you
    > > expected the expression "arg=otherfunction(x)" to be evaluated in

    > when
    > > it's used at the time of a function call to dynamically create a

    new
    > > default value for arg.

    >
    > Thanks, I realise now that's what was meant. I think I was just
    > assuming otherfunction() would be globally available - but that's not
    > something I want to go into since I can see now there will be

    problems
    > trying to define what python should do if it evaluated defaults at

    the
    > function call :)
    >
    > I think I can now explain to someone why there are good reasons that
    > the default params are evaluated at the definition. However, I still
    > can't give a nice looking solution on how to re-write a function to
    > have empty mutable values as default arguments: eg.
    >
    > def method(a,b,opt1=[],opt2=None,opt3="",opt4={})
    >
    > How could I re-write this (especially if there are perhaps 20

    optional
    > parameters,any number of which may have mutable defaults) without
    > writing 20 "if opt1 is None: opt1=[]" statements?
    >
    > Brian


    One way that is *slightly neater* is to simply collect all keyword
    arguments as a dictionary. Then compare with a dictionary of defaults
    for any missing keywords.

    def afunction(**keywargs):
    defaults = {'param1' : [], 'param2' : {}, 'param3' : []}
    for entry in defaults:
    if not keywargs.has_key(entry):
    keywargs[entry] = defaults[entry]

    This keeps all your defaults in a single dictionary and avoids a really
    long function definition.
    Regards,

    Fuzzy
    http://www.voidspace.org.uk/python/index.shtml
     
    Fuzzyman, Dec 23, 2004
    #14
  15. Fuzzyman Guest

    Re: Keyword arguments - strange behaviour?

    wrote:
    > > Channelling the effbot, I think he was asking what namespace

    context
    > you
    > > expected the expression "arg=otherfunction(x)" to be evaluated in

    > when
    > > it's used at the time of a function call to dynamically create a

    new
    > > default value for arg.

    >
    > Thanks, I realise now that's what was meant. I think I was just
    > assuming otherfunction() would be globally available - but that's not
    > something I want to go into since I can see now there will be

    problems
    > trying to define what python should do if it evaluated defaults at

    the
    > function call :)
    >
    > I think I can now explain to someone why there are good reasons that
    > the default params are evaluated at the definition. However, I still
    > can't give a nice looking solution on how to re-write a function to
    > have empty mutable values as default arguments: eg.
    >
    > def method(a,b,opt1=[],opt2=None,opt3="",opt4={})
    >
    > How could I re-write this (especially if there are perhaps 20

    optional
    > parameters,any number of which may have mutable defaults) without
    > writing 20 "if opt1 is None: opt1=[]" statements?
    >
    > Brian


    One way that is *slightly neater* is to simply collect all keyword
    arguments as a dictionary. Then compare with a dictionary of defaults
    for any missing keywords.

    def afunction(**keywargs):
    defaults = {'param1' : [], 'param2' : {}, 'param3' : []}
    for entry in defaults:
    if not keywargs.has_key(entry):
    keywargs[entry] = defaults[entry]

    This keeps all your defaults in a single dictionary and avoids a really
    long function definition.
    Regards,

    Fuzzy
    http://www.voidspace.org.uk/python/index.shtml
     
    Fuzzyman, Dec 23, 2004
    #15
  16. Fuzzyman Guest

    Re: Keyword arguments - strange behaviour?

    Steven Bethard wrote:
    > wrote:
    > > However, is there a good reason why default parameters aren't

    evaluated
    > > as the function is called? (apart from efficiency and backwards
    > > compatibility)?

    >
    > So, one of my really common use cases that takes advantage of the

    fact
    > that default parameters are evaluated at function definition time:
    >
    > def foo(bar, baz, matcher=re.compile(r'...')):
    > ...
    > text = matcher.sub(r'...', text)
    > ...
    >


    Surely "re.compile(r'...')" is effectively a constant ? So your above
    code is equivalent to :

    aConst = re.compile(r'...')
    def foo(bar, baz, matcher=aConst):
    ....
    text = matcher.sub(r'...', text)
    ....

    I agree that dynamic evaluation of default arguments seems much more
    pythonic, as well as getting rid of a common python gotcha.

    Regards,

    Fuzzy
    http://www.voidspace.org.uk/python/index.shmtl

    > If default parameters were evaluated when the function was called, my


    > regular expression would get re-compiled every time foo() was called.


    > This would be inefficient, especially if foo() got called a lot. If
    > Python 3000 changed the evaluation time of default parameters, I

    could
    > rewrite this code as:
    >
    > class foo(object):
    > matcher=re.compile(r'...')
    > def __new__(self, bar, baz, matcher=None):
    > if matcher is None:
    > matcher = self.matcher
    > ...
    > text = matcher.sub(r'...', text)
    > ...
    >
    > But that seems like a lot of work to do something that used to be

    pretty
    > simple...
    >
    > Steve
     
    Fuzzyman, Dec 23, 2004
    #16
  17. Re: Keyword arguments - strange behaviour?

    Fuzzyman wrote:
    > Steven Bethard wrote:
    >
    >> wrote:
    >>
    >>>However, is there a good reason why default parameters aren't
    >>>evaluated as the function is called? (apart from efficiency
    >>>and backwards compatibility)?

    >>
    >>So, one of my really common use cases that takes advantage of the
    >>fact that default parameters are evaluated at function definition
    >>time:
    >>
    >>def foo(bar, baz, matcher=re.compile(r'...')):
    >> ...
    >> text = matcher.sub(r'...', text)
    >> ...

    >
    > Surely "re.compile(r'...')" is effectively a constant ? So your above
    > code is equivalent to :
    >
    > aConst = re.compile(r'...')
    > def foo(bar, baz, matcher=aConst):
    > ...
    > text = matcher.sub(r'...', text)
    > ...


    Basically, yes. Though you add an extra name to the module or class
    namespace by defining 'aConst'. I consider this a minor pollution of
    the namespace since 'matcher' is only used in the function, not the
    module or class.

    But if we're arguing for equivalencies, the oft-asked for

    def foo(lst=[]):
    ...

    where [] is evaluated for each function call, is equivalent for most
    purposes to:

    def foo(lst=None):
    if lst is None:
    lst = []

    There's a minor pollution of the range of values 'lst' can take on -- if
    you need lst to be able to take on the value None, you'll want to
    rewrite this something like:

    no_value = object()
    def foo(lst=no_value):
    if lst is no_value:
    lst = []

    My point here is that there's always going to be some equivalent
    expression (turing completeness etc.) I was just pointing out a quite
    reasonable use of the current default parameter evaluation system.

    Steve
     
    Steven Bethard, Dec 23, 2004
    #17
  18. Fuzzyman Guest

    Re: Keyword arguments - strange behaviour?

    Steven Bethard wrote:
    > Fuzzyman wrote:
    > > Steven Bethard wrote:
    > >
    > >> wrote:
    > >>
    > >>>However, is there a good reason why default parameters aren't
    > >>>evaluated as the function is called? (apart from efficiency
    > >>>and backwards compatibility)?
    > >>
    > >>So, one of my really common use cases that takes advantage of the
    > >>fact that default parameters are evaluated at function definition
    > >>time:
    > >>
    > >>def foo(bar, baz, matcher=re.compile(r'...')):
    > >> ...
    > >> text = matcher.sub(r'...', text)
    > >> ...

    > >
    > > Surely "re.compile(r'...')" is effectively a constant ? So your

    above
    > > code is equivalent to :
    > >
    > > aConst = re.compile(r'...')
    > > def foo(bar, baz, matcher=aConst):
    > > ...
    > > text = matcher.sub(r'...', text)
    > > ...

    >
    > Basically, yes. Though you add an extra name to the module or class
    > namespace by defining 'aConst'. I consider this a minor pollution of


    > the namespace since 'matcher' is only used in the function, not the
    > module or class.
    >
    > But if we're arguing for equivalencies, the oft-asked for
    >
    > def foo(lst=[]):
    > ...
    >
    > where [] is evaluated for each function call, is equivalent for most
    > purposes to:
    >
    > def foo(lst=None):
    > if lst is None:
    > lst = []
    >
    > There's a minor pollution of the range of values 'lst' can take on --

    if
    > you need lst to be able to take on the value None, you'll want to
    > rewrite this something like:
    >
    > no_value = object()
    > def foo(lst=no_value):
    > if lst is no_value:
    > lst = []
    >
    > My point here is that there's always going to be some equivalent
    > expression (turing completeness etc.) I was just pointing out a

    quite
    > reasonable use of the current default parameter evaluation system.
    >
    > Steve


    Sure.. but you also gave an example of an alternative that was complex,
    and used it's complexity as an argument against having default
    arguments dynamically evaluated. What I was saying is that there is an
    alternative that is at least as readable (if not more) than your
    example.

    Having default arguments evaluated when the function is defined is
    unintuitive and unpythonic.
    Regards,


    Fuzzy
    http://www.voidspace.org.uk/python/index.shtml
     
    Fuzzyman, Dec 23, 2004
    #18
  19. Steve Holden Guest

    Re: Keyword arguments - strange behaviour?

    Fuzzyman wrote:

    [...]
    >
    > Having default arguments evaluated when the function is defined is
    > unintuitive and unpythonic.


    With due respect that's a matter of opinion.

    Having default arguments evaluated when the function is defined is the
    result of several design decisions in Python, not the least of which is
    the decision to have "def" be an executable statement. Ever wondered how
    come you can def the same function name several times in the same
    module? It's simply because on of def's side-effects is the binding of
    its name in the current namespace.

    Unfortunately there's absolutely no guarantee that the namespace context
    when a function is called will bear any relation to the namespace
    context when it was defined (particularly given that definition and call
    can take place in entirely separate modules). So, how would one specify
    a "deferred expression" to be evaluated when the function is called
    rather than when it is defined? There presumably has to be some
    technique that's different from the current ones, since presumably you
    would also want to retain the option of having the default evaluated at
    definition time.

    I still haven't seen any reasonable suggestions as to how one might
    exactly specify the execution context for such a deferred expression,
    let alone how to spell the deferred expression itself. Would you have us
    provide a code object that can be eval()'d?

    regards
    Steve
    --
    Steve Holden http://www.holdenweb.com/
    Python Web Programming http://pydish.holdenweb.com/
    Holden Web LLC +1 703 861 4237 +1 800 494 3119
     
    Steve Holden, Dec 23, 2004
    #19
  20. Re: Keyword arguments - strange behaviour?

    Fuzzyman wrote:
    >>>Steven Bethard wrote:
    >>>>
    >>>>So, one of my really common use cases that takes advantage of the
    >>>>fact that default parameters are evaluated at function definition
    >>>>time:
    >>>>
    >>>>def foo(bar, baz, matcher=re.compile(r'...')):
    >>>> ...
    >>>> text = matcher.sub(r'...', text)
    >>>> ...

    > Sure.. but you also gave an example of an alternative that was complex,


    Interesting. I would have thought that my example was pretty simple.
    Maybe it would be helpful to generalize it to:

    def foo(bar, baz, spam=badger(x, y, z)):
    ...

    All it does is use a default value that was produced by a function call.
    I'm surprised you haven't run into this situation before...

    Of course, what is complex or simple is a matter of personal opinion. I
    use this pattern so often that it's quite simple to me, but I guess I
    can understand that if you don't use such a pattern, it might seem
    foreign to you.

    Steve
     
    Steven Bethard, Dec 23, 2004
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Edward Diener
    Replies:
    14
    Views:
    5,084
    Josiah Carlson
    Apr 6, 2004
  2. matthewperpick
    Replies:
    8
    Views:
    270
    matthewperpick
    Apr 17, 2007
  3. Replies:
    6
    Views:
    478
    Peter Otten
    May 10, 2007
  4. Hamilton, William

    RE: keyword checker - keyword.kwlist

    Hamilton, William, May 10, 2007, in forum: Python
    Replies:
    4
    Views:
    372
  5. Peter Motzfeldt
    Replies:
    1
    Views:
    168
Loading...

Share This Page