Proposed PEP: Treating Builtins as Constants in the Standard Library

Discussion in 'Python' started by Raymond Hettinger, Apr 18, 2004.

  1. Comments are invited on the following proposed PEP.


    Raymond Hettinger

    -------------------------------------------------------



    PEP: 329
    Title: Treating Builtins as Constants in the Standard Library
    Author: Raymond Hettinger <>
    Status: Pre-PEP


    Abstract
    ========

    This proposal is to add a private function for treating builtin
    references as constants and to apply that function throughout
    the standard library.


    Motivation
    ==========

    The library contains code such as _len=len which is intended to create
    fast local references instead of a slower global lookups. Though
    necessary for performance, these constructions clutter the code and
    are usually incomplete (missing many opportunities).

    If the proposal is adopted, those constructions could be eliminated
    from the code base and at the same time improve upon their results
    in terms of performance.

    There are currently over a hundred instances of ``while 1`` in the
    library. They were not replaced with the more readable ``while True``
    because of performance reasons (the compiler cannot eliminate the
    test because True is not known to always be a constant). Conversion
    of True to a constant will clarify the code while retaining performance.

    Many other basic Python operations run much slower because of global
    lookups. In a try/except statement, the matching exception must be
    re-looked up on every call (whether or not the exception is raised).
    Similarly, simple identity tests such as ``while input is not None``
    require the None variable to be re-looked up on every pass.
    Builtin lookups are especially egregious because the enclosing
    global scope must be checked first. These lookup chains devour cache
    space that is best used elsewhere.

    In short, if the proposal is adopted, the code will become cleaner
    and performance will improve across the board.


    Proposal
    ========

    Add an internal module called _bind.py which contains two functions,
    bind_builtins() and bind_builtins_and_globals().

    For each module in the standard library, add a pair of lines near
    the near of the code::

    import _bind, sys
    _bind.bind_builtins(sys.modules[__name__])

    In a few select cases (such as sre_compile) where the module variables
    are just a collection of constants, the globals will also be converted
    to constants using bind_builtins_and_globals(). This will not be
    applied anywhere there would be cause for a user to externally
    change the value of one of the module variables.


    Question and Answers
    ====================

    Q. Will this make everyone divert their attention to optimization issues?

    A. No, it is internal only (not user visible). Also, it reduces the need
    to think about optimization because it is done automatically.


    Q. What if the module defines a variable shadowing a builtin?

    A. This does happen. For instance, True can be redefined at the module
    level as `True = (1==1)`. The sample implementation below detects the
    shadowing and leaves the global lookup unchanged.


    Q. How do you know this works?

    A. I implemented it, applied it to every module in library, and ran the
    test suite without exception.


    Q. Are you the first person to recognize that most global lookups are for
    values that never change?

    A. No, this has long been known. Skip provides an especially eloquent
    explanation in [1]_.


    Q. Why not make this a public module so that users can optimize builtins
    or builtins and globals in their own code?

    A. The ASPN recipe [2]_ is provided for this purpose. It takes a modicum
    of skill to use correctly and not inadvertently tranform a non-constant.


    Q. What if I have an overpowering urge to import a library module,
    write a new shadowing builtin, and insert it into the module's namespace?

    A. Your problems are less acute than with a module written in C
    because you can always comment out the automatic binding.


    Q. More realistically, what if I want to replace the builtins module and
    supply my own implementations?

    A. Either do this before importing a module, or just reload the module,
    or disable _bind.py.


    Q. How susceptible is this module to changes in Python's byte coding?

    A. It imports opcode.py to protect against renumbering. Also, it uses
    LOAD_CONST and LOAD_GLOBAL which are fundamental and have been around
    forever. That notwithstanding, the coding scheme could change
    dramatically and this implementation would have to change along with
    modules like `dis` which also rely on the current coding scheme.



    Sample Implementation
    =====================

    Here is a sample implementation for _bind.py::

    from types import ClassType, FunctionType
    from opcode import opmap, HAVE_ARGUMENT, EXTENDED_ARG
    LOAD_GLOBAL, LOAD_CONST = opmap['LOAD_GLOBAL'], opmap['LOAD_CONST']

    __all__ = ['bind_builtins', 'bind_builtins_and_globals']

    def bind(f, builtin_only=True):
    """ Return a new function with global references converted to constants.

    If builtin_only is True (the default), will convert all builtin
    references unless they have been overridden in the function's globals.
    If builtin_only is False, will also convert other module variable
    references into constants.

    """
    import __builtin__
    env = vars(__builtin__).copy()
    if builtin_only:
    stoplist = f.func_globals # do not replace overridden builtins
    else:
    stoplist = {}
    env.update(f.func_globals)

    co = f.func_code
    newcode = map(ord, co.co_code)
    newconsts = list(co.co_consts)
    codelen = len(newcode)

    i = 0
    while i < codelen:
    opcode = newcode
    if opcode == EXTENDED_ARG:
    return f # handle only the simple cases
    if opcode == LOAD_GLOBAL:
    oparg = newcode[i+1] + (newcode[i+2] << 8)
    name = co.co_names[oparg]
    if name in env and name not in stoplist:
    value = env[name]
    try:
    pos = newconsts.index(value)
    except ValueError:
    pos = len(newconsts)
    newconsts.append(value)
    newcode = LOAD_CONST
    newcode[i+1] = pos & 0xFF
    newcode[i+2] = pos >> 8
    i += 1
    if opcode >= HAVE_ARGUMENT:
    i += 2

    codestr = ''.join(map(chr, newcode))
    if codestr == co.co_code:
    return f # no changes were found
    codeobj = type(co)(co.co_argcount, co.co_nlocals, co.co_stacksize,
    co.co_flags, codestr, tuple(newconsts), co.co_names,
    co.co_varnames, co.co_filename, co.co_name,
    co.co_firstlineno, co.co_lnotab, co.co_freevars,
    co.co_cellvars)
    return type(f)(codeobj, f.func_globals, f.func_name, f.func_defaults,
    f.func_closure)

    def bind_all(module_or_class, builtin_only=True):
    """Recursively apply bind() to functions in a module or class.

    Use as the last line of the module (after everything is defined, but
    before test code or lines defining shorthand references to functions.

    If builtin_only is True (the default), will convert all builtin
    references unless they have been overridden in the function's globals.

    If builtin_only is False, will also convert other module variable
    references into constants. This should only be used in modules
    where the values of the module variables are truly constant.

    """
    for k, v in vars(module_or_class).items():
    if type(v) is FunctionType:
    newv = bind(v)
    setattr(module_or_class, k, newv)
    elif type(v) in (type, ClassType):
    bind_all(v, builtin_only)

    import sys
    bind_all(sys.modules[__name__], False) # Binder, bind thyself!

    # Self-documenting API (eliminates the builtin_only switch)
    bind_builtins = bind_all
    bind_builtins_and_globals = lambda m: bind_all(m, False)


    The final code should have some minor embellishments:

    * Automatic detection of a non-CPython environment that does not have
    bytecodes. In that situation, the bind functions would simply
    return the original function unchanged. This assures that the two
    line additions to library modules do not impact other implementations.

    * Add a few lines to handle reading and writing of EXTENDED_ARG. While
    this almost never arises in practice, it is best to have the code as
    adaptable as possible.


    References
    ==========

    ... [1] Optimizing Global Variable/Attribute Access
    http://www.python.org/peps/pep-0266.html

    ... [2] ASPN Recipe for a non-private implementation
    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/277940

    Copyright
    =========

    This document has been placed in the public domain.



    ...
    Local Variables:
    mode: indented-text
    indent-tabs-mode: nil
    sentence-end-double-space: t
    fill-column: 70
    End:
     
    Raymond Hettinger, Apr 18, 2004
    #1
    1. Advertising

  2. Raymond Hettinger

    John Roth Guest

    "Raymond Hettinger" <> wrote in message
    news:...
    > Comments are invited on the following proposed PEP.
    >
    >
    > Raymond Hettinger


    It certainly looks interesting. What's the possibility
    of also eliminating the extra function call on global
    functions like len() at the same time?

    John Roth
     
    John Roth, Apr 18, 2004
    #2
    1. Advertising

  3. Raymond Hettinger

    Carl Banks Guest

    Raymond Hettinger wrote:
    > PEP: 329
    > Title: Treating Builtins as Constants in the Standard Library
    > Author: Raymond Hettinger <>
    > Status: Pre-PEP
    >
    >
    > Abstract
    > ========
    >
    > This proposal is to add a private function for treating builtin
    > references as constants and to apply that function throughout
    > the standard library.





    Two things:

    1. The new _bind module needs to have a more specific (and
    appropriate) name than _bind. The function in the module could
    still be named bind, but the module itself should be named
    something like

    _optimize_globals_as_consts_very_unsafe

    (Ok, but you get the idea.)

    2. The module being optimized should have a special attribute, say
    __variable__, that identifies any globals that should not be
    optimized in this way:

    __variable__ = ['global_that_might_change']


    Other than that, +1. I'm definitely in favor of optimization hacks
    whenever it doesn't intrude too much on the code. It would be nice if
    it were safer.


    --
    CARL BANKS http://www.aerojockey.com/software
    "If you believe in yourself, drink your school, stay on drugs, and
    don't do milk, you can get work."
    -- Parody of Mr. T from a Robert Smigel Cartoon
     
    Carl Banks, Apr 18, 2004
    #3
  4. Re: Proposed PEP: Treating Builtins as Constants in the StandardLibrary

    rh> Q. Why not make this a public module so that users can optimize
    rh> builtins or builtins and globals in their own code?

    rh> A. The ASPN recipe [2]_ is provided for this purpose. It takes a
    rh> modicum of skill to use correctly and not inadvertently tranform
    rh> a non-constant.

    I thought one of the basic tenets of Python was "we're all adults here".
    Programmers have unfettered access to the new module with which they can
    wreak havoc. I don't see this proposal as somehow any more dangerous.

    Skip
     
    Skip Montanaro, Apr 18, 2004
    #4
  5. Raymond Hettinger

    Yermat Guest

    Re: Proposed PEP: Treating Builtins as Constants in the StandardLibrary

    Raymond Hettinger wrote:
    > Comments are invited on the following proposed PEP.
    >
    >
    > Raymond Hettinger
    >
    > -------------------------------------------------------
    >
    >
    >
    > PEP: 329
    > Title: Treating Builtins as Constants in the Standard Library
    > Author: Raymond Hettinger <>
    > Status: Pre-PEP
    >
    >
    > Abstract
    > ========
    >
    > This proposal is to add a private function for treating builtin
    > references as constants and to apply that function throughout
    > the standard library.


    Is it really a good thing to do it manually?
    Shouldn't be done automatically by a simple analyser?

    --
    Yermat
     
    Yermat, Apr 19, 2004
    #5
  6. Raymond Hettinger

    Peter Hansen Guest

    Re: Proposed PEP: Treating Builtins as Constants in the Standard

    Yermat wrote:

    > Raymond Hettinger wrote:
    >> PEP: 329
    >> Title: Treating Builtins as Constants in the Standard Library
    >> Author: Raymond Hettinger <>
    >>
    >> Abstract
    >> ========
    >>
    >> This proposal is to add a private function for treating builtin
    >> references as constants and to apply that function throughout
    >> the standard library.

    >
    > Is it really a good thing to do it manually?
    > Shouldn't be done automatically by a simple analyser?


    Not requiring that it be done manually, or at least not providing
    a simple means of disabling it, would prevent some kinds of
    automated testing. The most easily demonstrated one is where
    the builtin open/file calls are replaced with a simulated ("mock")
    file system which applies to *all* modules, not just a single
    module under test. This functionality is critical to some people
    using Python with the Test-Driven Development approach to software.

    -Peter
     
    Peter Hansen, Apr 19, 2004
    #6
  7. Re: Proposed PEP: Treating Builtins as Constants in the Standard

    On Mon, 19 Apr 2004 08:09:56 -0400, Peter Hansen <>
    wrote:

    [snip]
    ....
    >automated testing. The most easily demonstrated one is where
    >the builtin open/file calls are replaced with a simulated ("mock")
    >file system which applies to *all* modules, not just a single
    >module under test. This functionality is critical to some people
    >using Python with the Test-Driven Development approach to software.


    Isn't this the use case for AOP you were looking for?
    --dang
     
    Daniel 'Dang' Griffith, Apr 20, 2004
    #7
  8. Raymond Hettinger

    Peter Hansen Guest

    AOP use case? (was Re: Proposed PEP: Treating Builtins as Constantsin the Standard)

    Daniel 'Dang' Griffith wrote:

    > On Mon, 19 Apr 2004 08:09:56 -0400, Peter Hansen <>
    > wrote:
    >
    > [snip]
    > ...
    >
    >>automated testing. The most easily demonstrated one is where
    >>the builtin open/file calls are replaced with a simulated ("mock")
    >>file system which applies to *all* modules, not just a single
    >>module under test. This functionality is critical to some people
    >>using Python with the Test-Driven Development approach to software.

    >
    >
    > Isn't this the use case for AOP you were looking for?
    > --dang


    As you might imagine from my previous postings, I have no idea.
    I don't understand well enough what AOP really is to say,
    obviously (given my requests for compelling use cases so I can
    understand why I would even want to investigate it).

    Can you describe how AOP would solve this problem? As I
    understand it so far, it has nothing to do with mock objects...

    -Peter
     
    Peter Hansen, Apr 20, 2004
    #8
  9. Re: AOP use case? (was Re: Proposed PEP: Treating Builtins as Constants in the Standard)

    On Tue, 20 Apr 2004 10:33:36 -0400, Peter Hansen <>
    wrote:

    >Daniel 'Dang' Griffith wrote:
    >
    >> On Mon, 19 Apr 2004 08:09:56 -0400, Peter Hansen <>
    >> wrote:
    >> ...
    >>>automated testing. The most easily demonstrated one is where
    >>>the builtin open/file calls are replaced with a simulated ("mock")
    >>>file system which applies to *all* modules, not just a single
    >>>module under test. This functionality is critical to some people
    >>>using Python with the Test-Driven Development approach to software.

    >>
    >> Isn't this the use case for AOP you were looking for?
    >> --dang

    >
    >As you might imagine from my previous postings, I have no idea.
    >I don't understand well enough what AOP really is to say,
    >obviously (given my requests for compelling use cases so I can
    >understand why I would even want to investigate it).
    >
    >Can you describe how AOP would solve this problem? As I
    >understand it so far, it has nothing to do with mock objects...


    Probably not.
    I'm not an "AOP person", but from what I understand, you could use
    an AOP-enabled language to replace the real file system (or whatever)
    objects with the mock objects. This is roughly analagous to the
    standard logging or auditing examples, wherein a program is doing what
    it needs to do, meanwhile a cross-cutting aspect intercepts certain
    operations and performing additional, or different, operations.
    --dang
     
    Daniel 'Dang' Griffith, Apr 21, 2004
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Raymond Hettinger
    Replies:
    0
    Views:
    315
    Raymond Hettinger
    Apr 19, 2004
  2. James J. Besemer

    Proposed new PEP: print to expand generators

    James J. Besemer, Jun 4, 2006, in forum: Python
    Replies:
    3
    Views:
    302
    Bruno Desthuilliers
    Jun 5, 2006
  3. Paul Rubin

    proposed PEP: iterator splicing

    Paul Rubin, Apr 15, 2007, in forum: Python
    Replies:
    8
    Views:
    278
    Georg Brandl
    Apr 15, 2007
  4. Xeno Campanoli
    Replies:
    2
    Views:
    134
    Eric Hodel
    Jan 25, 2008
  5. G Matthew J

    proposed new standard: JSON++

    G Matthew J, Jul 19, 2005, in forum: Javascript
    Replies:
    16
    Views:
    214
    G Matthew J
    Jul 31, 2005
Loading...

Share This Page