Getting in to metaprogramming

[Rafe]

Hi,

In the name of self-education can anyone share some pointers, links,
modules, etc that I might use to begin learning how to do some
"metaprogramming". That is, using code to write code (right?)

Cheers,

- Rafe

 

[Michele Simionato]

Hi,

In the name of self-education can anyone share some pointers, links,
modules, etc that I might use to begin learning how to do some
"metaprogramming". That is, using code to write code (right?)

Cheers,

- Rafe

The word "metaprogramming" has many meanings. It covers code
generation, as you say,
but also a lots of other techniques; for instance you do
metaprogramming with decorators
when write functions operating over functions, or with metaclasses
when you write classes
operating over classes. You should clarify what you are interested in.
For instance, are
you interested in metaprogramming in Python only or you want a more
general introduction?
Have you read the Wikipedia entry?
http://en.wikipedia.org/wiki/Metaprogramming

 

[Aaron Brady]

Hi,

In the name of self-education can anyone share some pointers, links,
modules, etc that I might use to begin learning how to do some
"metaprogramming". That is, using code to write code (right?)

Cheers,

- Rafe

Python programs can generate code for themselves.
.... d= { 'cls': i }
.... s="""
.... class Cls%(cls)s:
.... def meth%(cls)s( self, arg ):
.... print 'in meth%(cls)s, arg:', arg
.... """% d
.... exec( s )
.... s= """
.... inst%(cls)s= Cls%(cls)s()
.... """% d
.... exec( s )
....in meth2, arg: arg

The 'Cls0', 'Cls1', 'Cls2' repetitiveness is taken care of with a for-
loop.

 

[Rafe]

Python programs can generate code for themselves.


...   d= { 'cls': i }
...   s="""
... class Cls%(cls)s:
...   def meth%(cls)s( self, arg ):
...     print 'in meth%(cls)s, arg:', arg
... """% d
...   exec( s )
...   s= """
... inst%(cls)s= Cls%(cls)s()
... """% d
...   exec( s )
...>>> inst0.meth0( "arg" )
in meth0, arg: arg

in meth2, arg: arg

The 'Cls0', 'Cls1', 'Cls2' repetitiveness is taken care of with a for-
loop.

Michele, I am thinking about python which writes python. Which makes
Aaron's post accurate to my needs. More specifically, I am considering
what it might be like to use python to build a script file which can
be executed later. Until now I planned to store info to XML and then
parse it to run later. There are good reasons that generating a script
file would be more useful for me.

Aaron, Is it really as simple as gathering strings of code? Sort of
like generating HTML or XML directly? Is there any other framework or
pattern set that is worth looking in to?

Thanks for helping me explore this.

- Rafe

 

[Michele Simionato]

is it really as simple as gathering strings of code?
Yes.

Sort of like generating HTML or XML directly? Is there any other framework or
pattern set that is worth looking in to?

Yes, the compiler module and the ast module in the standard library.
You may also
look at how templating languages that compile to Python code work (I
mean mako
or PTL/qpy). You may find interesting how the import hooks work too.

 

[Stef Mientki]

Michele, I am thinking about python which writes python. Which makes
Aaron's post accurate to my needs. More specifically, I am considering
what it might be like to use python to build a script file which can
be executed later. Until now I planned to store info to XML and then
parse it to run later. There are good reasons that generating a script
file would be more useful for me.

I use Python's own triple quoted string as a generator for the GUI, like
this:
GUI = """
self.NB ,wx.Notebook ,style = wx.NO_BORDER
self.Grid ,Base_Table_Grid ,None, data_values, data_types, data_defs
Panel2 ,PanelVer, 11 ,name = "Page2"
list1 ,wx.ListCtrl ,style = wx.LC_REPORT
"""
cheers,
Stef

 

[Aaron Brady]

Yes.


Yes, the compiler module and the ast module in the standard library.
You may also
look at how templating languages that compile to Python code work (I
mean mako
or PTL/qpy). You may find interesting how the import hooks work too.

You could have Python generate HTML, Python generate COBOL, Python
generate Python, Perl generate Python, sure. Not that I know COBOL.

If you're generating Python with Python, you only need one file to do
it, not two separate programs, as you would for the other
combinations; it was my point. It could be really helpful or
unnecessary, depending on your application.

There are some templating modules, possibly for Python code. If
anyone knows them, they can point them out.

I don't know a clean, reliable way to structure a metaprogram though.
Mine always turn out messy.

 

[Arnaud Delobelle]

I don't know a clean, reliable way to structure a metaprogram though.
Mine always turn out messy.

I don't think Python is designed for large scale metaprogramming. Lisp
is the only naturally suited language that I know.

 

[Michele Simionato]

Then another thing - it strikes me that any problem that can be solved
by metaprogramming, can be solved by putting similar code into a class
and instanciating an instance.

Does anybody know if this is true?

If it is, it limits the usefulness of metaprogramming to the creation
of "stored procedures" for "later execution".

- Hendrik

Many times (probably most times) you can avoid code generation and use
classes or
higher order functions instead. Actually I don't like code generation
techniques
in Python and other languages. I would make an exception for Lisp-like
languages,
however, since there you a pretty powerful macro mechanism that in
some cases can be
better than using higher order functions, especially if performance is
important.
I say something more in the latest episode of my "Adventures of a
Pythonista in Schemeland"
which some of you may find interesting: http://www.artima.com/weblogs/viewpost.jsp?thread=240836


M. Simionato

 

[Steven D'Aprano]

Yes.

Then another thing - it strikes me that any problem that can be solved
by metaprogramming, can be solved by putting similar code into a class
and instanciating an instance.

Does anybody know if this is true?

Well, I don't know about "any problem". And it's not so much about
whether metaprograms can solve problems that can't be solved by anything
else, as whether metaprograms can solve problems more effectively than
other techniques.

If you include factory functions, class factories, the builder design
pattern, metaclasses, etc. as "metaprogramming", then I use it all the
time, and find it an excellent technique to use.

But if you mean using a Python program to generate Python source code,
then I can't think of any time I used it. Which doesn't mean that others
don't find it helpful, only that I haven't yet.

Thinking further back, when I was young and programming in Apple's
Hypercard 4GL, I used to frequently use Hypercard scripts to generate new
Hypercard scripts. That was to work around the limitations of the
scripting language.

I don't think metaprogramming in the limited sense (programs to output
source code) is a bad idea, but I do think that there are probably better
alternatives in a language like Python.

 

[Kay Schluehr]

Hi,

In the name of self-education can anyone share some pointers, links,
modules, etc that I might use to begin learning how to do some
"metaprogramming". That is, using code to write code (right?)

Cheers,

- Rafe

http://www.letmegooglethatforyou.com/?q=python+metaprogramming

 

[Kay Schluehr]

Given that, can anybody think of an example that you could not do with
a class? (excepting the "stored procedure" aspect)

I just noticed that corepy 1.0 [1] has been released. Corepy is an
embedded DSL for synthesizing machine code from chaining Python
commands. This means it provides objects and exploits control
structures used to create machine code that can finally be executed
interactively.

Let's say you have an ordinary Python function that computes a CRC 32.
Now you could attempt to translate the function into other Python code
that expresses a corepy routine. You could create a decorator that
works as follows

1) reads the source of the decorated function
2) transforms the source into corepy source and compiles it or
3) if 2) fails it just returns the passed code object.

Kay

[1] http://www.corepy.org/

 

[Terry Reedy]

I am using the term in the restricted sense of Python writing Python source.

Given that, can anybody think of an example that you could not do with
a class? (excepting the "stored procedure" aspect)

I am not sure I understand your question.

def iterize(recursive_function_text):
<code to parse input and fill a template>
return equivalent_iterative_function_text

where input and output are both Python code. If one were to implement
this by compiling the input to AST form and then walking the tree, the
AST node classes would be involved, but I would scarely say the
translation was done by the classes, as opposed to functions which might
or might not be attacked to a class as methods.

tjr

 

[Hendrik van Rooyen]

I don't know a clean, reliable way to structure a metaprogram though.
Mine always turn out messy.

Yes.

Then another thing - it strikes me that any problem that can be solved
by metaprogramming, can be solved by putting similar code into a class
and instanciating an instance.

Does anybody know if this is true?

If it is, it limits the usefulness of metaprogramming to the creation
of "stored procedures" for "later execution".

- Hendrik

 

[Aaron Brady]

 "Steven D'Aprano" <steau> wrote:






I am using the term in the restricted sense of Python writing Python source.

Given that, can anybody think of an example that you could not do with
a class?  (excepting the "stored procedure" aspect)

Or can I claim a new a new meta - rule - I would call it van Rooyen's folly...

The example I think of is the Visitor Pattern of Gamma and all. One
class's method calls another's method with its own class's name in the
name.

class Visitor:
def visit_A( self, arg ):...
def visit_B( self, arg ):...

class A:
def visit( self, vis ):
vis.visit_A( self )

class B:
def visit( self, vis ):
vis.visit_B( self )

As you can see, the 'visit' method is mechanical for classes A and B.
One might want to autogenerate those in some languages, but Python has
introspection:

class BaseAB:
def visit( self, vis ):
getattr( vis, 'visit_%s'% self.__class__.__name__ )( self )

And it's easier to modify the default behavior this way than in
autogenerated code too.

 

[Steven D'Aprano]

I am using the term in the restricted sense of Python writing Python
source.

Given that, can anybody think of an example that you could not do with a
class? (excepting the "stored procedure" aspect)

GUI designer. You write a program to let the user create code by clicking
buttons, dragging objects, drawing lines, etc. The GUI designer may use
classes, but the purpose of those classes is to generate source code.

Testing code speed... you might have some functions with a loop, and you
want to unroll the loop as an optimization. If you have one function, you
can unroll it yourself. If you have a hundred such functions, you might
want to write a program to do it. (Yes, I'm stretching...)

Don't like that Python doesn't optimize tail-recursion? Then write a
source-code analyzer that detects tail-recursion and re-writes the
function using a while loop.

What sort of stuff did you do, and would having had simple OO available
have rendered it unnecessary?

It's been 20-odd years, and the examples were pretty trivial... I don't
really recall exactly, but it would have been something like this:

* design a GUI involving lots of buttons on screen, each one with quite
similar but not identical code;

* since Hypercard didn't have a layout manager, write a script to
generate each button, place it where needed, and set the button's code.

Hypercard did have a message passing hierarchy (like inheritance for
objects), so often you could take the button's code and place it in a
higher level of the hierarchy (the card, the background, the stack), but
there were odd cases where that wasn't enough.

Another example: Hypercard had a very limited number of GUI elements
(text fields and buttons, basically) but I designed a slider control
using a few buttons, each button with a custom script. To avoid needing
to create and place the buttons by hand each time I wanted a slider, I
had a script that did it for me. The script not only created the buttons,
but it created the scripts used by the buttons. This wasn't as difficult
as it sounds -- it was basically taking a template and doing some text
replacements, then telling the button to use it as a script.

 

[Aaron Brady]

GUI designer. You write a program to let the user create code by clicking
buttons, dragging objects, drawing lines, etc. The GUI designer may use
classes, but the purpose of those classes is to generate source code.

I want to disagree with this example. I hold that the proper output
of a GUI designer is in XML, or another data format.

Testing code speed... you might have some functions with a loop, and you
want to unroll the loop as an optimization. If you have one function, you
can unroll it yourself. If you have a hundred such functions, you might
want to write a program to do it. (Yes, I'm stretching...)

Don't like that Python doesn't optimize tail-recursion? Then write a
source-code analyzer that detects tail-recursion and re-writes the
function using a while loop.

I've seen a preprocessor come up a few times. The input is a
program's source, and the output is source. Of course you can do it
with quotes and 'exec' at run-time, but if your whole file is quoted,
it may be a better option.

(Part of my rationale was I wanted syntax-coloring.)

It's been 20-odd years, and the examples were pretty trivial... I don't
really recall exactly, but it would have been something like this:

* design a GUI involving lots of buttons on screen, each one with quite
similar but not identical code;

* since Hypercard didn't have a layout manager, write a script to
generate each button, place it where needed, and set the button's code.

snip

(Above.) I'm not sure if you'd count something like 'perlmodule':

import perl

# Simple arithmetics
six = perl.eval("3+3")

# Eval can also return functions
sum = perl.eval("sub { my $s = shift; $s += shift while @_; $s }")
print sum(1,2,3)

http://aspn.activestate.com/ASPN/CodeDoc/pyperl/perlmodule.html

but speaking of quoted code and all...

 

[Hendrik van Rooyen]

Well, I don't know about "any problem". And it's not so much about
whether metaprograms can solve problems that can't be solved by anything
else, as whether metaprograms can solve problems more effectively than
other techniques.

If you include factory functions, class factories, the builder design
pattern, metaclasses, etc. as "metaprogramming", then I use it all the
time, and find it an excellent technique to use.

But if you mean using a Python program to generate Python source code,
then I can't think of any time I used it. Which doesn't mean that others
don't find it helpful, only that I haven't yet.

I am using the term in the restricted sense of Python writing Python source.

Given that, can anybody think of an example that you could not do with
a class? (excepting the "stored procedure" aspect)

Or can I claim a new a new meta - rule - I would call it van Rooyen's folly...

Thinking further back, when I was young and programming in Apple's
Hypercard 4GL, I used to frequently use Hypercard scripts to generate new
Hypercard scripts. That was to work around the limitations of the
scripting language.

What sort of stuff did you do, and would having had simple OO available
have rendered it unnecessary?

I don't think metaprogramming in the limited sense (programs to output
source code) is a bad idea, but I do think that there are probably better
alternatives in a language like Python.

True. No argument here - I was just wondering if the relationship holds.

- Hendrik

 

[Rafe]

 "Steven D'Aprano" <steau> wrote:






I am using the term in the restricted sense of Python writing Python source.

Given that, can anybody think of an example that you could not do with
a class?  (excepting the "stored procedure" aspect)

Or can I claim a new a new meta - rule - I would call it van Rooyen's folly...




What sort of stuff did you do, and would having had simple OO available
have rendered it unnecessary?




True. No argument here - I was just wondering if the relationship holds.

- Hendrik

"Given that, can anybody think of an example that you could not do
with a class?"

Generating a template for a specific script application. For example,
a script with pre-defined callbacks that only require the addition of
the contents.

I was really interested in exploring the idea of using python output,
instead of XML, to record something a user did in a GUI. I have seen
it done and it is really advantageous in the 3D industry because it
means the script files can be edited directly, in a pinch, to generate
something slightly different.

For example, say we have code which generates a cube by plotting it's
points. A user then changes a point position in the GUI. The change is
saved by outputting the function call to a file with new arguments
(the new point location). If I wanted to make 100s of copies of the
cube, but with a slightly different point position, I could edit the
custom cube's python code and hand it back for creation without using
the GUI. I could do this with XML, but it can be harder to work with
in a text editor (though I have seen some XML editors that make it a
bit easier.) In fact, in most 3D applications, the app prints
everything the user does to a log. Sometimes in a choice of languages,
so I guess I am looking to do the same thing with my own custom tools.

In a real situation the generated code file can build some pretty
complex 3D object hierarchies. It moves beyond simple storage of data
and becomes a real script that can be hacked as necessary.

It is nice to have everything as python scripts because we always have
a blend of GUI users and developers to get a job done.

- Rafe

 

[Rafe]

"Given that, can anybody think of an example that you could not do
with a class?"

Generating a template for a specific script application. For example,
a script with pre-defined callbacks that only require the addition of
the contents.

I was really interested in exploring the idea of using python output,
instead of XML, to record something a user did in a GUI. I have seen
it done and it is really advantageous in the 3D industry because it
means the script files can be edited directly, in a pinch, to generate
something slightly different.

For example, say we have code which generates a cube by plotting it's
points. A user then changes a point position in the GUI. The change is
saved by outputting the function call to a file with new arguments
(the new point location). If I wanted to make 100s of copies of the
cube, but with a slightly different point position, I could edit the
custom cube's python code and hand it back for creation without using
the GUI. I could do this with XML, but it can be harder to work with
in a text editor (though I have seen some XML editors that make it a
bit easier.) In fact, in most 3D applications, the app prints
everything the user does to a log. Sometimes in a choice of languages,
so I guess I am looking to do the same thing with my own custom tools.

In a real situation the generated code file can build some pretty
complex 3D object hierarchies. It moves beyond simple storage of data
and becomes a real script that can be hacked as necessary.

It is nice to have everything as python scripts because we always have
a blend of GUI users and developers to get a job done.

-Rafe

I was just thinking (hopefully i get some time to try this soon) that
it wouldn't be difficult to decorate a function so that when called, a
line of code is output. as long as the arguments can be stored as a
string (strings, numbers, lists, etc. but no 'object' instances) it
should be able to be executed to get the same result. I think it would
just have to:

1) Dynamically write the name with a '('

2) Gather all the args in a list and ", ".join(args)

3) Gather kwargs as a list of ['%s = %s' % key, value] and then and ",
".join(kwlist)

4) Add ')\n'


- Rafe