Compile time evaluation (aka eliminating default argument hacks)

[Nick Coghlan]

Time for another random syntax idea. . .

So, I was tinkering in the interactive interpreter, and came up with the
following one-size-fits-most default argument hack:

Py> x = 1
Py> def _build_used():
.... y = x + 1
.... return x, y
....
Py> def f(_used = _build_used()):
.... x, y = _used
.... print x, y
....
Py> f()
1 2
Py> x = 3
Py> f()
1 2

Works pretty well in terms of getting early binding, compile-time evaluation of
expensive values and psuedo-constants, initialising a local with a shadowed
value from an outer scope, sharing mutable values between invocations, and well,
really, anything that default argument hacks are used for. It has the benefit of
allowing use of a full suite at compile time, instead of the expressions one is
usually limited to in default arguments hacks. It also slightly improves the
argspec pollution situation by only using one argument instead of several.

However, it's still ugly as sin, still pollutes the functions argspec, the lists
of names in the assignment statement and the return statement have to be kept in
sync manually, and you're now polluting the outer namespace as well. Not to
mention the fact that the contents of the compile-time functions are miles away
from where the results are used.

But consider a syntax like the following:

def f():
use x, y from:
y = x + 1 # [1]
print x, y

[1] I'll grant that the binding of x from the outer scope here is more than a
little obscure. However, I could see 'use x from: pass' becoming an idiom for
early binding, in which case the example 'use' block could be written:
use y from: y = x + 1
use x from: pass

Then mixing early binding with early evaluation in the same 'use' block might
simply be considered bad style, and discouraged (although not prevented).

Essentially, the function is compiled as usual, and emits code at the location
of the 'use' statement equivalent to that for "x, y = <const>". The relevant
entry in co_consts is populated by executing the body of the 'use' statement
with an implicit "return x, y" at the end. The environment for that execution is
the same as that for any function defined at the same level as the containing
scope of the 'use' statement (e.g. module level in the example).

Multiple 'use' blocks would be allowed in a scope. A 'use' block at module level
would simply mean that the result of calling the block gets marshalled into the
compiled module file, rather than the block itself.

You could get performance improvements on *any* function, simply by moving code
which doesn't depend on the functions arguments inside a 'use' block. For
modules, data structures initialised inside a using block could simply be
unmarshalled rather than built anew.

Cheers,
Nick.

P.S. I didn't search the archive, because I couldn't figure out any search terms
for the topic that weren't swamped by irrelevant hits.

 

[Steven Bethard]

Time for another random syntax idea. . .

So, I was tinkering in the interactive interpreter, and came up with the
following one-size-fits-most default argument hack:
[snip]

But consider a syntax like the following:

def f():
use x, y from:
y = x + 1 # [1]
print x, y
[snip]

Essentially, the function is compiled as usual, and emits code at the
location of the 'use' statement equivalent to that for "x, y = <const>".
The relevant entry in co_consts is populated by executing the body of
the 'use' statement with an implicit "return x, y" at the end. The
environment for that execution is the same as that for any function
defined at the same level as the containing scope of the 'use' statement
(e.g. module level in the example).

So just to clarify, the issue you're trying to address is when you want
early binding like function default arguments get, but you don't want to
declare the names as function arguments?

If I understand you right, I like the idea, though I'm undecided on your
syntax for it at the moment...

STeVe

 

[Nick Coghlan]

So just to clarify, the issue you're trying to address is when you want
early binding like function default arguments get, but you don't want to
declare the names as function arguments?

Basically, yeah. Although I later realised I got the name of the feature I want
wrong - default arguments are evaluated when the def statement is executed, not
when the code is compiled. So it's a matter of being able to execute some code
in the functions local namespace at compile time.

I have my doubts about the feasibility of the syntax I posted, too. A version
which the Python compiler and parser should actually be able to handle would be
to tack an optional clause on to the front of the function definition statement:

using x, y from:
# Again, the right syntax for early binding is not clear
y = x + 2
def f():
print x, y

(Similar to some of the ideas about statement local namespaces that were kicking
around a while back, but in this case specific to function definitions)

Anyway, if others agree that the ability to execute a suite at def exeuction
time to preinitialise a function's locals without resorting to bytecode hacks is
worth having, finding a decent syntax is the next trick

Cheers,
Nick.

 

[Nick Coghlan]

Basically, yeah. Although I later realised I got the name of the feature
I want wrong - default arguments are evaluated when the def statement is
executed, not when the code is compiled. So it's a matter of being able
to execute some code in the functions local namespace at compile time.

D'oh - "definition execution time", not compile time.

Cheers,
Nick.

 

[Steven Bethard]

Anyway, if others agree that the ability to execute a suite at def
exeuction time to preinitialise a function's locals without resorting to
bytecode hacks is worth having, finding a decent syntax is the next
trick

I'm not certain how many use cases really require a full suite, though
being able to define default values for function locals in the same way
that default values can be defined for function arguments would be nice.

Worth looking at is the thread:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/58f53fe8bcc49664/

STeVe

 

[Nick Coghlan]

Worth looking at is the thread:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/58f53fe8bcc49664/

Huh - I thought I put something in the original post saying "without resorting
to bytecode hacks", but I must have deleted it before sending the message.

Anyway, that thread is what the comment about bytecode hacks would have been
referring to

Although, once the AST compiler goes in, if the AST for a function is attached
via a special attribute like __ast__, then perhaps such manipulations could be
rewritten to work on the AST instead of the bytecode. If that happens, they
might become less hackish.

So I guess I should wait and see what the next few months holds in terms of
AST-hacks

Cheers,
Nick.

 

[Bengt Richter]

Well, what if the bytecode hacks were part of a builtin decorator? Of course,
someone would have to make sure versions were updated, but that's true of
such things as the dis module too.

I'm not certain how many use cases really require a full suite, though
being able to define default values for function locals in the same way
that default values can be defined for function arguments would be nice.

Enjoy ;-)

Worth looking at is the thread:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/58f53fe8bcc49664/

Before I went away I guess I sort of promised to post "my" (obviously, credits/thanks to Raymond) hack,
so here it is. BTW it also tries to modify the line map and update signature for currying,
but it is _minimally_ tested:
(Note that this way of currying does not result in nested calls and closures, so currying should
result in speedup rather than slowdown, even without applying Raymond's optimizing decorator ;-)

----< presets.py >-------------------------------------------------------------------------
# presets.py -- a decorator to preset function local variables without a default-argument hack or closure
# also does currying, with adjustment of argument count, eliminating named arguments from right.
# 20050310 09:22:15 -- alpha 0.01 release -- bokr
# Released to the public domain WITH NO WARRANTIES of any kind by Bengt Richter
# Released to c.l.py for testing and/or further development by the interested.
# Byte code munging based on cannibalizing Raymond Hettinger's make_constants optimizing decorator (presets.py
# doesn't do the optimizations, though make_constants should be able to process the output of presets if applied
# outermost).
#
if __import__('sys').version_info[:2] != (2, 4):
raise SystemExit, 'presets.py requires version 2.4 at least, and maybe exactly.'

from opcode import opmap, HAVE_ARGUMENT, EXTENDED_ARG, hasjabs
globals().update(opmap)

class ShouldNotHappenError(Exception): pass

def presets(verbose=False, **presets):
"""
Print preset change info if verbose.
All keyword values are injected into the decorated function's
local namespace as intial assignments to local variables.
A function may make use of the variables before apparently setting them.
Global references will be overridden and made into local preset variable
references if they are present as keyword arguments.
"""
return lambda f: _presets(f, False, verbose, **presets)

def curry(verbose=False, **curry):
"""
return a function with named arguments replaced with given expression values
and eliminated from signature. Multiple arguments may be eliminated but names
must be taken from the right of the signature without skipping.
"""
return lambda f: _curry(f, verbose, **curry)

def _curry(f, verbose, **curry):
try:
co = f.func_code
except AttributeError:
return f # Jython doesn't have a func_code attribute.
if not curry: return f # nothing to do
names = co.co_names
varnames = list(co.co_varnames)[:co.co_argcount] # for indexing local names
if len(curry) > len(varnames):
raise ValueError, 'too many curry values %r vs %r'%(curry.keys(), varnames)
for n, name in enumerate(varnames[::-1]):
if n >= len(curry): break
if name not in curry:
raise ValueError, 'must supply %r before others in arg list %r'%(name, varnames)
return _presets(f, True, verbose, **curry)


def _presets(f, curry=False, verbose=False, **presets):
try:
co = f.func_code
except AttributeError:
return f # Jython doesn't have a func_code attribute.
if not presets: return f # nothing to do
newcode = map(ord, co.co_code)
newconsts = list(co.co_consts)
names = co.co_names
codelen = len(newcode)
varnames = list(co.co_varnames) # for indexing local names
nvarnames = len(varnames) # for later check if any added
prenames = tuple(sorted(presets))
nseq = len(prenames)
pretuple = tuple(presets[name] for name in prenames)
pos = len(newconsts)
newconsts.append(nseq > 1 and pretuple or pretuple[0])
if verbose: print '\npresets: -- "name(?)" means name may be unused'
# generate the code to set presets (by unpacking the constant tuple of values if more than one value)
precode = [LOAD_CONST, pos&0xFF, pos >> 8] # single value or tuple to unpack
if nseq > 1: precode.extend([ UNPACK_SEQUENCE, nseq&0xff, nseq>>8])
for name in prenames:
try: ix = varnames.index(name) # look for local name
except ValueError:
ix = len(varnames)
varnames.append(name) # make sure there is a local variable as such for the preset name
if verbose:
print '%12s%s = %r' % (name, '(?)'*(name not in names), presets[name])
precode.extend([STORE_FAST, ix&0xff, ix>>8])
if verbose: print
precodelen = len(precode)

# Change preset-name global references to local names and references
# adjust absolute jumps for length of presetting code
i = 0
posdict = {}
while i < codelen:
opcode = newcode
if opcode in (EXTENDED_ARG, STORE_GLOBAL):
return f # XXX ?? for simplicity, only preset for common cases ??
if opcode in (LOAD_GLOBAL, LOAD_NAME, LOAD_CLOSURE, LOAD_DEREF):
oparg = newcode[i+1] + (newcode[i+2] << 8)
if opcode in (LOAD_GLOBAL, LOAD_NAME):
name = co.co_names[oparg]
else:
name = (co.co_cellvars + co.co_freevars)[oparg]
if name in presets:
# change code to LOAD_FAST of new local
try: ix = varnames.index(name)
except ValueError:
raise ShouldNotHappenError, "--> can't find new local %r in %r" % (name, varnames)
else:
newcode = LOAD_FAST
newcode[i+1] = ix&0xff
newcode[i+2] = ix>>8

elif precodelen and opcode in hasjabs: # JUMP_ABSOLUTE or CONTINUE_LOOP
jdest = newcode[i+1] + (newcode[i+2]<<8)
jdest += precodelen # abs positions have moved down by presetting-code's length
newcode[i+1] = jdest&0xff
newcode[i+2] = jdest>>8
i += 1
if opcode >= HAVE_ARGUMENT:
i += 2

newlocals = len(varnames)-nvarnames
codestr = ''.join(map(chr, precode+newcode))
argcount = co.co_argcount
defaults = f.func_defaults
if curry:
argcount -= len(presets)
defaults = defaults and defaults[:len(presets)]
codeobj = type(co)(argcount, co.co_nlocals+newlocals, co.co_stacksize,
co.co_flags, codestr, tuple(newconsts), co.co_names,
tuple(varnames), co.co_filename, co.co_name,
co.co_firstlineno, '%c%c%c%c%s'%(0, 0, precodelen, 2, co.co_lnotab[2:]), co.co_freevars,
co.co_cellvars)
return type(f)(codeobj, f.func_globals, f.func_name, defaults, f.func_closure)

def test():
@presets(
verbose=True,
decotime=__import__('time').ctime(),
ver = __import__('sys').version,
test_unused = 'verbose output should show postfixed (?) on this variable',
comment = 'XXX the presets decorator needs much more testing!'
)
def foo():
print
print 'Note: foo was decorated on %s' % decotime
print 'Python version %s' % ver
print
print comment
print
foo()
print
print 'Curried def bar(x, y):return x*y with y=111, printing bar(2), bar(3):'
@curry(y=111)
def bar(x, y): return x*y
print bar(2), bar(3)
return foo, bar


if __name__ == '__main__':
test()
-------------------------------------------------------------------------------------------

Results:

[ 9:47] C:\pywk\clp>py24 ..\ut\presets.py

presets: -- "name(?)" means name may be unused
comment = 'XXX the presets decorator needs much more testing!'
decotime = 'Thu Mar 10 09:47:27 2005'
test_unused(?) = 'verbose output should show postfixed (?) on this variable'
ver = '2.4b1 (#56, Nov 3 2004, 01:47:27) \n[GCC 3.2.3 (mingw special 20030504-1)]'


Note: foo was decorated on Thu Mar 10 09:47:27 2005
Python version 2.4b1 (#56, Nov 3 2004, 01:47:27)
[GCC 3.2.3 (mingw special 20030504-1)]

XXX the presets decorator needs much more testing!


Curried def bar(x, y):return x*y with y=111, printing bar(2), bar(3):
222 333

Regards,
Bengt Richter

 

[Carl Banks]

Anyway, if others agree that the ability to execute a suite at def exeuction
time to preinitialise a function's locals without resorting to bytecode hacks is
worth having, finding a decent syntax is the next trick

Workarounds:

1. Just use a freaking global, especially if it's just a stateless
piece of data. Give it a long decriptive name, make it all caps,
precede it with an underscore, and put a comment nearby saying that
it's only for use with a certain function. If someone uses it for
another reason anyways, BFD, we're all adults here. If you need more
speed, assign it a local alias.


2.

.. def call_and_replace_with_result(func):
.. return func()
..
.. @call_and_replace_with_result
.. def function_with_predefined_locals():
.. x = initialize()
.. y = initialize()
.. def real_function(a,b,c):
.. use(x,y,a,b,c)
.. return real_function


I wouldn't use it, though, since a global variable is much more
readable and not much more dangerous. I could, however, see myself
using the slightly more complicated descriptor such as this (for a
wholly different reason, though):

.. def call_with_open_file(filename):
.. def descriptor(func):
.. flo = open(filename)
.. try: f(flo)
.. finally: f.close()
.. return None
.. return descriptor

 

[Carl Banks]

I could, however, see myself
using the slightly more complicated descriptor such as this (for a
wholly different reason, though):

. def call_with_open_file(filename):
. def descriptor(func):
. flo = open(filename)
. try: f(flo)
. finally: f.close()
. return None
. return descriptor

Apparently, I don't know the difference between a descriptor and a
decorator.