final version of PEP-318 "Decorators for functions and methods"

Discussion in 'Python' started by Anthony Baxter, Sep 3, 2004.

  1. To go along with the 2.4a3 release, here's an updated version of
    the decorator PEP. It describes the state of decorators as they
    are in 2.4a3.

    PEP: 318
    Title: Decorators for Functions and Methods
    Version: $Revision: 1.34 $
    Last-Modified: $Date: 2004/09/03 09:32:50 $
    Author: Kevin D. Smith, Jim Jewett, Skip Montanaro, Anthony Baxter
    Status: Draft
    Type: Standards Track
    Content-Type: text/x-rst
    Created: 05-Jun-2003
    Python-Version: 2.4
    Post-History: 09-Jun-2003, 10-Jun-2003, 27-Feb-2004, 23-Mar-2004, 30-Aug-2004,


    This document is meant to describe the decorator syntax and the
    process that resulted in the decisions that were made. It does not
    attempt to cover the huge number of potential alternative syntaxes,
    nor is it an attempt to exhaustively list all the positives and
    negatives of each form.


    The current method for transforming functions and methods (for instance,
    declaring them as a class or static method) is awkward and can lead to
    code that is difficult to understand. Ideally, these transformations
    should be made at the same point in the code where the declaration
    itself is made. This PEP introduces new syntax for transformations of a
    function or method declaration.


    The current method of applying a transformation to a function or method
    places the actual transformation after the function body. For large
    functions this separates a key component of the function's behavior from
    the definition of the rest of the function's external interface. For

    def foo(self):
    perform method operation
    foo = classmethod(foo)

    This becomes less readable with longer methods. It also seems less
    than pythonic to name the function three times for what is conceptually
    a single declaration. A solution to this problem is to move the
    transformation of the method closer to the method's own declaration.
    While the new syntax is not yet final, the intent is to replace::

    def foo(cls):
    foo = synchronized(lock)(foo)
    foo = classmethod(foo)

    with an alternative that places the decoration in the function's

    def foo(cls):

    Modifying classes in this fashion is also possible, though the benefits
    are not as immediately apparent. Almost certainly, anything which could
    be done with class decorators could be done using metaclasses, but
    using metaclasses is sufficiently obscure that there is some attraction
    to having an easier way to make simple modifications to classes. For
    Python 2.4, only function/method decorators are being added.

    Why Is This So Hard?

    Two decorators (``classmethod()`` and ``staticmethod()``) have been
    available in Python since version 2.2. It's been assumed since
    approximately that time that some syntactic support for them would
    eventually be added to the language. Given this assumption, one might
    wonder why it's been so difficult to arrive at a consensus. Discussions
    have raged off-and-on at times in both comp.lang.python and the
    python-dev mailing list about how best to implement function decorators.
    There is no one clear reason why this should be so, but a few problems
    seem to be most divisive.

    * Disagreement about where the "declaration of intent" belongs.
    Almost everyone agrees that decorating/transforming a function at the
    end of its definition is suboptimal. Beyond that there seems to be no
    clear consensus where to place this information.

    * Syntactic constraints. Python is a syntactically simple language
    with fairly strong constraints on what can and can't be done without
    "messing things up" (both visually and with regards to the language
    parser). There's no obvious way to structure this information so
    that people new to the concept will think, "Oh yeah, I know what
    you're doing." The best that seems possible is to keep new users from
    creating a wildly incorrect mental model of what the syntax means.

    * Overall unfamiliarity with the concept. For people who have a
    passing acquaintance with algebra (or even basic arithmetic) or have
    used at least one other programming language, much of Python is
    intuitive. Very few people will have had any experience with the
    decorator concept before encountering it in Python. There's just no
    strong preexisting meme that captures the concept.

    * Syntax discussions in general appear to cause more contention than
    almost anything else. Readers are pointed to the ternary operator
    discussions that were associated with PEP 308 for another example of


    There is general agreement that syntactic support is desirable to
    the current state of affairs. Guido mentioned `syntactic support
    for decorators`_ in his DevDay keynote presentation at the `10th
    Python Conference`_, though `he later said`_ it was only one of
    several extensions he proposed there "semi-jokingly". `Michael Hudson
    raised the topic`_ on ``python-dev`` shortly after the conference,
    attributing the initial bracketed syntax to an earlier proposal on
    ``comp.lang.python`` by `Gareth McCaughan`_.

    ... _syntactic support for decorators:
    ... _10th python conference:
    ... _michael hudson raised the topic:
    ... _he later said:
    ... _gareth mccaughan:

    Class decorations seem like an obvious next step because class
    definition and function definition are syntactically similar,
    however Guido remains unconvinced, and class decorators will almost
    certainly not be in Python 2.4.

    The discussion continued on and off on python-dev from February
    2002 through July 2004. Hundreds and hundreds of posts were made,
    with people proposing many possible syntax variations. Guido took
    a list of proposals to `EuroPython 2004`_, where a discussion took
    place. Subsequent to this, he decided that we'd have the `Java-style`_
    @decorator syntax, and this appeared for the first time in 2.4a2.
    Barry Warsaw named this the 'pie-decorator' syntax, in honor of the
    Pie-thon Parrot shootout which was occured around the same time as
    the decorator syntax, and because the @ looks a little like a pie.
    Guido `outlined his case`_ on Python-dev, including `this piece`_
    on some of the (many) rejected forms.

    ... _EuroPython 2004:
    ... _outlined his case:
    ... _this piece:
    ... _Java-style:

    On the name 'Decorator'

    There's been a number of complaints about the choice of the name
    'decorator' for this feature. The major one is that the name is not
    consistent with its use in the `GoF book`_. The name 'decorator'
    probably owes more to its use in the compiler area -- a syntax tree is
    walked and annotated. It's quite possible that a better name may turn

    ... _GoF book:

    Design Goals

    The new syntax should

    * work for arbitrary wrappers, including user-defined callables and
    the existing builtins ``classmethod()`` and ``staticmethod()``. This
    requirement also means that a decorator syntax must support passing
    arguments to the wrapper constructor

    * work with multiple wrappers per definition

    * make it obvious what is happening; at the very least it should be
    obvious that new users can safely ignore it when writing their own

    * be a syntax "that ... [is] easy to remember once explained"

    * not make future extensions more difficult

    * be easy to type; programs that use it are expected to use it very

    * not make it more difficult to scan through code quickly. It should
    still be easy to search for all definitions, a particular definition,
    or the arguments that a function accepts

    * not needlessly complicate secondary support tools such as
    language-sensitive editors and other "`toy parser tools out

    * allow future compilers to optimize for decorators. With the hope of
    a JIT compiler for Python coming into existence at some point this
    tends to require the syntax for decorators to come before the function

    * move from the end of the function, where it's currently hidden, to
    the front where it is more `in your face`_

    Andrew Kuchling has links to a bunch of the discussions about
    motivations and use cases `in his blog`_. Particularly notable is `Jim
    Huginin's list of use cases`_.

    ... _toy parser tools out there:
    ... _in your face:
    ... _in his blog:
    ... _Jim Huginin's list of use cases:

    Current Syntax

    The current syntax for function decorators as implemented in Python
    2.4a2 is::

    def func(arg1, arg2, ...):

    This is equivalent to::

    def func(arg1, arg2, ...):
    func = dec2(dec1(func))

    without the intermediate assignment to the variable ``func``. The
    decorators are near the function declaration. The @ sign makes it clear
    that something new is going on here.

    The decorator statement is limited in what it can accept -- arbitrary
    expressions will not work. Guido preferred this because of a `gut

    ... _gut feeling:

    Syntax Alternatives

    There have been `a large number`_ of different syntaxes proposed --
    rather than attempting to work through these individual syntaxes, it's
    worthwhile to break the syntax discussion down into a number of areas.
    Attempting to discuss `each possible syntax`_ individually would be an
    act of madness, and produce a completely unwieldy PEP.

    ... _a large number:
    ... _each possible syntax:

    Decorator Location

    The first syntax point is the location of the decorators. For the
    following examples, we use the @syntax used in 2.4a2.

    Decorators before the def statement are the first alternative, and the
    syntax used in 2.4a2::

    def foo(arg1,arg2):

    def bar(low,high):

    There have been a number of objections raised to this location -- the
    primary one is that it's the first real Python case where a line of code
    has an effect on a following line. The syntax available in 2.4a3
    requires one decorator per line (in a2, multiple decorators could be
    specified on the same line).

    People also complained that the syntax quickly got unwieldy when
    multiple decorators were used. The point was made, though, that the
    chances of a large number of decorators being used on a single function
    were small and thus this was not a large worry.

    Some of the advantages of this form are that the decorators live outside
    the method body -- they are obviously executed at the time the function
    is defined.

    Another advantage is that a prefix to the function definition fits
    the idea of knowing about a change to the semantics of the code before
    the code itself, thus you know how to interpret the code's semantics
    properly without having to go back and change your initial perceptions
    if the syntax did not come before the function definition.

    Guido decided `he preferred`_ having the decorators on the line before
    the 'def', because it was felt that a long argument list would mean that
    the decorators would be 'hidden'

    ... _he preferred:

    The second form is the decorators between the def and the function name,
    or the function name and the argument list::

    def @classmethod foo(arg1,arg2):

    def @accepts(int,int),@returns(float) bar(low,high):

    def foo @classmethod (arg1,arg2):

    def bar @accepts(int,int),@returns(float) (low,high):

    There are a couple of objections to this form. The first is that it
    breaks easily 'greppability' of the source -- you can no longer search
    for 'def foo(' and find the definition of the function. The second,
    more serious, objection is that in the case of multiple decorators, the
    syntax would be extremely unwieldy.

    The next form, which has had a number of strong proponents, is to have
    the decorators between the argument list and the trailing ``:`` in the
    'def' line::

    def foo(arg1,arg2) @classmethod:

    def bar(low,high) @accepts(int,int),@returns(float):

    Guido `summarized the arguments`_ against this form (many of which also
    apply to the previous form) as:

    - it hides crucial information (e.g. that it is a static method)
    after the signature, where it is easily missed

    - it's easy to miss the transition between a long argument list and a
    long decorator list

    - it's cumbersome to cut and paste a decorator list for reuse, because
    it starts and ends in the middle of a line

    ... _summarized the arguments:

    The next form is that the decorator syntax goes inside the method body at
    the start, in the same place that docstrings currently live:

    def foo(arg1,arg2):

    def bar(low,high):

    The primary objection to this form is that it requires "peeking inside"
    the method body to determine the decorators. In addition, even though
    the code is inside the method body, it is not executed when the method
    is run. Guido felt that docstrings were not a good counter-example, and
    that it was quite possible that a 'docstring' decorator could help move
    the docstring to outside the function body.

    The final form is a new block that encloses the method's code. For this
    example, we'll use a 'decorate' keyword, as it makes no sense with the
    @syntax. ::

    def foo(arg1,arg2):

    def bar(low,high):

    This form would result in inconsistent indentation for decorated and
    undecorated methods. In addition, a decorated method's body would start
    three indent levels in.

    Syntax forms

    * ``@decorator``::

    def foo(arg1,arg2):

    def bar(low,high):

    The major objections against this syntax are that the @ symbol is
    not currently used in Python (and is used in both IPython and Leo),
    and that the @ symbol is not meaningful. Another objection is that
    this "wastes" a currently unused character (from a limited set) on
    something that is not perceived as a major use.

    * ``|decorator``::

    def foo(arg1,arg2):

    def bar(low,high):

    This is a variant on the @decorator syntax -- it has the advantage
    that it does not break IPython and Leo. Its major disadvantage
    compared to the @syntax is that the | symbol looks like both a capital
    I and a lowercase l.

    * list syntax::

    def foo(arg1,arg2):

    [accepts(int,int), returns(float)]
    def bar(low,high):

    The major objection to the list syntax is that it's currently
    meaningful (when used in the form before the method). It's also
    lacking any indication that the expression is a decorator.

    * list syntax using other brackets (``<...>``, ``[[...]]``, ...)::

    def foo(arg1,arg2):

    <accepts(int,int), returns(float)>
    def bar(low,high):

    None of these alternatives gained much traction. The alternatives
    which involve square brackets only serve to make it obvious that the
    decorator construct is not a list. They do nothing to make parsing any
    easier. The '<...>' alternative presents parsing problems because '<'
    and '>' already parse as un-paired. They present a further parsing
    ambiguity because a right angle bracket might be a greater than symbol
    instead of a closer for the decorators.

    * ``decorate()``

    The ``decorate()`` proposal was that no new syntax be implemented
    -- instead a magic function that used introspection to manipulate
    the following function. Both Jp Calderone and Philip Eby produced
    implementations of functions that did this. Guido was pretty firmly
    against this -- with no new syntax, the magicness of a function like
    this is extremely high:

    Using functions with "action-at-a-distance" through sys.settraceback
    may be okay for an obscure feature that can't be had any other
    way yet doesn't merit changes to the language, but that's not
    the situation for decorators. The widely held view here is that
    decorators need to be added as a syntactic feature to avoid the
    problems with the postfix notation used in 2.2 and 2.3. Decorators
    are slated to be an important new language feature and their
    design needs to be forward-looking, not constrained by what can be
    implemented in 2.3.

    * _`new keyword (and block)`

    This idea was the consensus alternate from comp.lang.python (more
    on this in `Community Consensus`_ below.) Robert Brewer wrote up a
    detailed `J2 proposal`_ document outlining the arguments in favor of
    this form. The initial issues with this form are:

    - It requires a new keyword, and therefore a ``from __future__
    import decorators`` statement.

    - The choice of keyword is contentious. However ``using`` emerged
    as the consensus choice, and is used in the proposal and

    - The keyword/block form produces something that looks like a normal
    code block, but isn't. Attempts to use statements in this block
    will cause a syntax error, which may confuse users.

    A few days later, Guido `rejected the proposal`_ on two main grounds,

    ... the syntactic form of an indented block strongly
    suggests that its contents should be a sequence of statements, but
    in fact it is not -- only expressions are allowed, and there is an
    implicit "collecting" of these expressions going on until they can
    be applied to the subsequent function definition. ...

    and secondly:

    ... the keyword starting the line that heads a block
    draws a lot of attention to it. This is true for "if", "while",
    "for", "try", "def" and "class". But the "using" keyword (or any
    other keyword in its place) doesn't deserve that attention; the
    emphasis should be on the decorator or decorators inside the suite,
    since those are the important modifiers to the function definition
    that follows. ...

    Readers are invited to read `the full response`_.

    .. _J2 proposal:

    .. _rejected the proposal:

    .. _the full response:

    * Other forms

    There are plenty of other variants and proposals on `the wiki page`_.

    ... _the wiki page:

    Why @?

    There is some history in Java using @ initially as a marker in `Javadoc
    comments`_ and later in Java 1.5 for `annotations`_, which are similar
    to Python decorators. The fact that @ was previously unused as a token
    in Python also means it's clear there is no possibility of such code
    being parsed by an earlier version of Python, leading to possibly subtle
    semantic bugs. It also means that ambiguity of what is a decorator
    and what isn't is removed. That said, @ is still a fairly arbitrary
    choice. Some have suggested using | instead.

    For syntax options which use a list-like syntax (no matter where it
    appears) to specify the decorators a few alternatives were proposed:
    ``[|...|]``, ``*[...]*``, and ``<...>``.

    ... _Javadoc comments:
    ... _annotations:

    Current Implementation, History

    Guido asked for a volunteer to implement his preferred syntax, and Mark
    Russell stepped up and posted a `patch`_ to SF. This new syntax was
    available in 2.4a2. ::

    def func(arg1, arg2, ...):

    This is equivalent to::

    def func(arg1, arg2, ...):
    func = dec2(dec1(func))

    though without the intermediate creation of a variable named ``func``.

    The version implemented in 2.4a2 allowed multiple ``@decorator`` clauses
    on a single line. In 2.4a3, this was tightened up to only allowing one
    decorator per line.

    A `previous patch`_ from Michael Hudson which implements the
    list-after-def syntax is also still kicking around.

    ... _patch:
    ... _previous patch:

    After 2.4a2 was released, in response to community reaction, Guido
    stated that he'd re-examine a community proposal, if the community
    could come up with a community consensus, a decent proposal, and an
    implementation. After an amazing number of posts, collecting a vast
    number of alternatives in the `Python wiki`_, a community consensus
    emerged (below). Guido `subsequently rejected`_ this alternate form,
    but added:

    In Python 2.4a3 (to be released this Thursday), everything remains
    as currently in CVS. For 2.4b1, I will consider a change of @ to
    some other single character, even though I think that @ has the
    advantage of being the same character used by a similar feature
    in Java. It's been argued that it's not quite the same, since @
    in Java is used for attributes that don't change semantics. But
    Python's dynamic nature makes that its syntactic elements never mean
    quite the same thing as similar constructs in other languages, and
    there is definitely significant overlap. Regarding the impact on
    3rd party tools: IPython's author doesn't think there's going to be
    much impact; Leo's author has said that Leo will survive (although
    it will cause him and his users some transitional pain). I actually
    expect that picking a character that's already used elsewhere in
    Python's syntax might be harder for external tools to adapt to,
    since parsing will have to be more subtle in that case. But I'm
    frankly undecided, so there's some wiggle room here. I don't want
    to consider further syntactic alternatives at this point: the buck
    has to stop at some point, everyone has had their say, and the show
    must go on.

    ... _Python wiki:
    ... _subsequently rejected:

    Community Consensus

    This section documents the rejected J2 syntax, and is included for
    historical completeness.

    The consensus that emerged on comp.lang.python was the proposed J2
    syntax (the "J2" was how it was referenced on the PythonDecorators wiki
    page): the new keyword ``using`` prefixing a block of decorators before
    the ``def`` statement. For example::

    def func(cls):

    The main arguments for this syntax fall under the "readability counts"
    doctrine. In brief, they are:

    * A suite is better than multiple @lines. The ``using`` keyword and
    block transforms the single-block ``def`` statement into a
    multiple-block compound construct, akin to try/finally and others.

    * A keyword is better than punctuation for a new token. A keyword
    matches the existing use of tokens. No new token category is
    necessary. A keyword distinguishes Python decorators from Java
    annotations and .Net attributes, which are significantly different

    Robert Brewer wrote a `detailed proposal`_ for this form, and Michael
    Sparks produced `a patch`_.

    ... _detailed proposal:
    ... _a patch:

    As noted previously, Guido rejected this form, outlining his problems
    with it in `a message`_ to python-dev and comp.lang.python.

    ... _a message:


    Much of the discussion on ``comp.lang.python`` and the ``python-dev``
    mailing list focuses on the use of decorators as a cleaner way to use
    the ``staticmethod()`` and ``classmethod()`` builtins. This capability
    is much more powerful than that. This section presents some examples of

    1. Define a function to be executed at exit. Note that the function
    isn't actually "wrapped" in the usual sense. ::

    def onexit(f):
    import atexit
    return f

    def func():

    Note that this example is probably not suitable for real usage, but
    is for example purposes only.

    2. Define a class with a singleton instance. Note that once the class
    disappears enterprising programmers would have to be more creative to
    create more instances. (From Shane Hathaway on ``python-dev``.) ::

    def singleton(cls):
    instances = {}
    def getinstance():
    if cls not in instances:
    instances[cls] = cls()
    return instances[cls]
    return getinstance

    class MyClass:

    3. Add attributes to a function. (Based on an example posted by
    Anders Munch on ``python-dev``.) ::

    def attrs(**kwds):
    def decorate(f):
    for k in kwds:
    setattr(f, k, kwds[k])
    return f
    return decorate

    author="Guido van Rossum")
    def mymethod(f):

    4. Enforce function argument and return types. Note that this
    copies the func_name attribute from the old to the new function.
    func_name was made writable in Python 2.4a3::

    def accepts(*types):
    def check_accepts(f):
    assert len(types) == f.func_code.co_argcount
    def new_f(*args, **kwds):
    for (a, t) in zip(args, types):
    assert isinstance(a, t), \
    "arg %r does not match %s" % (a,t)
    return f(*args, **kwds)
    new_f.func_name = f.func_name
    return new_f
    return check_accepts

    def returns(rtype):
    def check_returns(f):
    def new_f(*args, **kwds):
    result = f(*args, **kwds)
    assert isinstance(result, rtype), \
    "return value %r does not match %s" % (result,rtype)
    return result
    new_f.func_name = f.func_name
    return new_f
    return check_returns

    @accepts(int, (int,float))
    def func(arg1, arg2):
    return arg1 * arg2

    5. Declare that a class implements a particular (set of) interface(s).
    This is from a posting by Bob Ippolito on ``python-dev`` based on
    experience with `PyProtocols`_. ::

    def provides(*interfaces):
    An actual, working, implementation of provides for
    the current implementation of PyProtocols. Not
    particularly important for the PEP text.
    def provides(typ):
    declareImplementation(typ, instancesProvide=interfaces)
    return typ
    return provides

    class IBar(Interface):
    """Declare something about IBar here"""

    class Foo(object):
    """Implement something here..."""

    .. _PyProtocols:

    Of course, all these examples are possible today, though without
    syntactic support.

    Open Issues

    1. It's not yet certain that class decorators will be incorporated
    into the language at a future point. Guido expressed skepticism about
    the concept, but various people have made some `strong arguments`_
    (search for ``PEP 318 -- posting draft``) on their behalf in
    ``python-dev``. It's exceedingly unlikely that class decorators
    will be in Python 2.4.

    .. _strong arguments:

    2. The choice of the ``@`` character will be re-examined before
    Python 2.4b1.


    This document has been placed in the public domain.

    Local Variables:
    mode: indented-text
    indent-tabs-mode: nil
    sentence-end-double-space: t
    fill-column: 70
    Anthony Baxter, Sep 3, 2004
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John Roth

    PEP 318 - PyFIT comments

    John Roth, Aug 4, 2004, in forum: Python
    Nick Vargish
    Aug 6, 2004
  2. Edward K. Ream

    Confused about pep 318

    Edward K. Ream, Aug 4, 2004, in forum: Python
    Shalabh Chaturvedi
    Aug 6, 2004
  3. Delaney, Timothy C (Timothy)

    RE: Decorator syntax (was Re: PEP 318 - PyFIT comments)

    Delaney, Timothy C (Timothy), Aug 5, 2004, in forum: Python
    =?ISO-8859-1?Q?I=F1igo?= Serna
    Aug 6, 2004
  4. Hallvard B Furuseth

    pep-318 questions

    Hallvard B Furuseth, Aug 8, 2004, in forum: Python
    Hallvard B Furuseth
    Aug 28, 2004
  5. Tim Hochberg
    Tim Hochberg
    Aug 10, 2004

Share This Page