An idiom for code generation with exec

[George Sakkis]

First of all, I see absolutely no connection between your question and
the text you quote. Is there? Or did you pick one post randomly to
post your question on?

Second, yes - I have profiled my code.

Third, this is a very typical torture path one has to go through when
asking about code generation. It is true of almost all communities,
except Lisp, perhaps. You have to convince everyone that you have a
real reason to do what you do. The simple norm of getting a reply to
your question doesn't work when you get to code generation. I wonder
why is it so. How many people have been actually "burned" by bad code
generation techniques, and how many are just parroting "goto is evil"
because it's the accepted thing to say. This is an interesting point
to ponder.

It's not as much that many people have been burned but that, like
goto, 99% of the time there are better alternatives. Off the top of my
head, two recurring threads in c.l.py related to dynamic code
generation and evaluation are:
- Asking how to dynamically generate variable names ("for i in
xrange(10): exec 'x%d = %d' % (i,i)") instead of using a regular
dictionary.
- Using function names instead of the actual function objects and
calling eval(), not knowing that functions are first-class objects (or
not even familiar with what that means).

So even if your use case belongs to the exceptional 1% where dynamic
code generation is justified, you should expect people to question it
by default.

George

 

[eliben]

Thanks for all the replies in this post. Just to conclude, I want to
post a piece of code I wrote to encapsulate function creation in this
way:

def create_function(code):
""" Create and return the function defined in code.
"""
m = re.match('\s*def\s+([a-zA-Z_]\w*)\s*\(', code)
if m:
func_name = m.group(1)
else:
return None

d = {}
exec code.strip() in globals(), d
return d[func_name]

Although the 'def' matching in the beginning looks a bit shoddy at
first, it should work in all cases.

Eli

 

[Bruno Desthuilliers]

eliben a écrit :

d = {}
execcode in globals(), d
return d['foo']

My way:

return function(compile(code, '<string>', 'exec'), globals())

With some help from the guys at IRC I came to realize your way doesn't
do the same. It creates a function that, when called, creates 'foo' on
globals(). This is not exactly what I need.

I possibly messed up a couple things in the arguments, flags etc - I
very seldom use compile() and function(). The point was that it didn't
require any extra step.

 

[Maric Michaud]

Le Monday 23 June 2008 09:22:29 Bruno Desthuilliers, vous avez écrit :

I possibly messed up a couple things in the arguments, flags etc - I
very seldom use compile() and function(). The point was that  it didn't
require any extra step.

In the argument list of function type, the code object in first place is
expected to be created directly (no exec - eval) with the python type 'code'
(either found as types.CodeType or new.code).


In [24]: types.CodeType?
Type: type
Base Class: <type 'type'>
String Form: <type 'code'>
Namespace: Interactive
Docstring:
code(argcount, nlocals, stacksize, flags, codestring, constants, names,
varnames, filename, name, firstlineno, lnotab[, freevars[,
cellvars]])

Create a code object. Not for the faint of heart.

^^^^^^^^^^^^^^^

Even if it looks more "object oriented", I'm not sure it's actually the good
solution for the original problem. I think these interface are not a
replacement for the quick eval-exec idiom but more intended to make massive
code generation programs object oriented and closer to python internals.

AFAIK, the only use case I see code generation (eval - exec, playing with code
objects) as legitime in python is in programs that actually do code
generation, that is, parse and compile code from textual inputs (application
buillders).

If code generation is not the best, and I fail to see any performance issue
that could explain such a choice, except a misunderstanding of
what "compilation" means in python, just don't use it, use closures or
callable instances, there are many way to achieve this.

 

[Bruno Desthuilliers]

Maric Michaud a écrit :

Le Monday 23 June 2008 09:22:29 Bruno Desthuilliers, vous avez écrit :

In the argument list of function type, the code object in first place is
expected to be created directly (no exec - eval) with the python type 'code'

Which is what compile returns. But indeed, re-reading compile's doc more
carefully, I'm afraid that the code object it returns may not be usable
the way I thought. My bad. <OP>sorry</OP>

(snip)

 

[Fuzzyman]

A simple .strip() doesn't work if the code comprises multiple lines:


...     return """
...     x = 42
...     if x > 0:
...             print x
...     """
...>>> exec "if 1:\n" + f().rstrip()
42

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 2
    if x > 0:
    ^
IndentationError: unexpected indent

You can of course split the code into lines, calculate the indentation of
the first non-white line, remove that indentation from all lines and then
rejoin.

textwrap.dedent will do all that for you...

Michael Foord
http://www.ironpythoninaction.com/

 

[eliben]

If code generation is not the best, and I fail to see any performance issue

that could explain such a choice, except a misunderstanding of
what "compilation" means in python, just don't use it, use closures or
callable instances, there are many way to achieve this.

And while we're on the topic of what compilation means in Python, I'm
not sure I fully understand the difference between compiled (.pyc)
code and exec-ed code. Is the exec-ed code turned to bytecode too,
i.e. it will be as efficient as compile-d code ?

Eli

 

[Terry Reedy]

And while we're on the topic of what compilation means in Python,

It depends on the implementation.


I'm

not sure I fully understand the difference between compiled (.pyc)
code and exec-ed code. Is the exec-ed code turned to bytecode too,
i.e. it will be as efficient as compile-d code ?

CPython always compiles to bytecode before executing. There is no
alternative execution path.

 

[Maric Michaud]

Le Tuesday 24 June 2008 07:18:47 eliben, vous avez écrit :

And while we're on the topic of what compilation means in Python, I'm
not sure I fully understand the difference between compiled (.pyc)
code and exec-ed code. Is the exec-ed code turned to bytecode too,
i.e. it will be as efficient as compile-d code ?

Yes, exactly the same, cpython always interprets compiled code, when a script
is executed for example, it is parsed/compiled to bytecode by the interpreter
before any execution. The .pyc/pyo files are just a cache created at import
time to avoid the rather time consuming parsing stage.

 

[jhermann]

Since nobody mentioned textwrap.dedent yet as an alternative to the
old "if 1:" trick, I thought I should do so.

 

[eliben]

Thanks for all the replies in this post. Just to conclude, I want to
post a piece of code I wrote to encapsulate function creation in this
way:

def create_function(code):
""" Create and return the function defined in code.
"""
m = re.match('\s*def\s+([a-zA-Z_]\w*)\s*\(', code)
if m:
func_name = m.group(1)
else:
return None

d = {}
exec code.strip() in globals(), d
return d[func_name]

Actually this won't work, because globals() returns the scope in which
create_function is defined, not called. So if I want to place
create_function in some utils file, the code it generates won't be
able to use auxiliary functions from the file that uses
create_function.

Eli