Limits of Metaprogramming

[Wilson]

Hi all,

I have an interesting problem that I'm hoping can be solved with
metaprogramming, but I don't know how far Python supports code
generation (and I don't know if I'm taking the correct approach
either... hence why I'm asking on this group):

I'd like to write a program that writes/manipulates a statemachine. My
idea was that I would define states in a .py file along with their
transitions and other metadata. I could then write a suite of programs
that would manipulate these states in interesting ways, such as
generate diagrams and documentation, test suites, do verifications and
also edit them. These programs would import the .py file and use
introspection upon the module's classes (states would be modelled as
classes). For example, a documentation tool could use graphviz to draw
the statediagram with the transition table and docstring...

My problem is that I don't know if it's possible to edit these states
and then write them back to .py. Firstly, if my editing tool was to
create a new state, I would want to create the class (using type) and
attach it to the imported state module somehow. I can't find
information on whether this is possible. Secondly, I'd like to
manipulate the class and then use the inspect module to get the source
and write it back to file. Now I'm pretty sure the inspect module will
only write the source code as found in the original .py file. Is there
a way to generate the code for a class or other python objects in a
generic way? Or will I have to bite the bullet and write to custom
file and generate code from that (like glade with XML)? Or is there
another solution with templates that I'm over looking?

Having the classes in memory, editing their attributes, adding methods
and then serialising them to code has an elegance to it in my mind and
would hopefully simplify things. The result would be directly
executable by the interpreter itself and editable by python
developers. But I fear that I may need an intermediary notation/format
for storage. Any advice/hints would be very useful.

Thanks,
Paul

 

[Jeff]

You could write a class composed of states and then use the pickle
module to serialize it to disk.

 

[Wilson]

You could write a class composed of states and then use the pickle
module to serialize it to disk.

Thanks Jeff.

I guess this is my intermediary format!

 

[Tomasz Rola]

Hi all,

Howdy,

I am not sure if my remarks will be of any use for you, but here it goes.

I have an interesting problem that I'm hoping can be solved with
metaprogramming, but I don't know how far Python supports code
generation (and I don't know if I'm taking the correct approach
either... hence why I'm asking on this group):

I'd like to write a program that writes/manipulates a statemachine. My
idea was that I would define states in a .py file along with their
transitions and other metadata. I could then write a suite of programs
that would manipulate these states in interesting ways, such as
generate diagrams and documentation, test suites, do verifications and
also edit them. These programs would import the .py file and use
introspection upon the module's classes (states would be modelled as
classes). For example, a documentation tool could use graphviz to draw
the statediagram with the transition table and docstring...

Maybe you have actually two problems.

One problem is processing source code into something useful, but
not executable. So it does not necessarily need metaprogramming to be
solved, just some "code comprehension" (like get a python grammar,
lex/yacc and read a lot of docs how to use them).

My problem is that I don't know if it's possible to edit these states
and then write them back to .py. Firstly, if my editing tool was to
create a new state, I would want to create the class (using type) and
attach it to the imported state module somehow. I can't find
information on whether this is possible. Secondly, I'd like to
manipulate the class and then use the inspect module to get the source
and write it back to file. Now I'm pretty sure the inspect module will
only write the source code as found in the original .py file. Is there
a way to generate the code for a class or other python objects in a
generic way? Or will I have to bite the bullet and write to custom
file and generate code from that (like glade with XML)? Or is there
another solution with templates that I'm over looking?

Well, "everything" is doable in a Turing-complete language . However,
this one problem - code generation and using it on the fly - may be better
solved with some other language, perhaps?

Not that I want to dissuade you (or anyone else) from using Python, but
last time I checked the state of things (was a good couple of years ago, I
wanted to solve some math-related problem with metaprogramming in Python)
I decided I would do better if I learned Scheme. At least in Scheme,
program structure is so simple, it is just a list (or a sequence of them),
so manipulating code is IMHO much easier.

I mean, even if in theory every language in Turing-complete class is equal
to each other (one can do the same kind of stuff in one as in the other),
in practice they are not equal at all. So doing everything in one ultimate
language of choice is not very wise from my point of view.

It does not mean you should immediately go to

http://en.wikipedia.org/wiki/Scheme_(programming_language)

but you may do so, and then make an informed choice by yourself. To be
better informed, you may also google for more material - introductions,
opinions, comparisons and some first-hand experience.

On the other hand, writing "metafiles" and interpreters for them - ugh...
. I have read somewhere, that some programmers resist using Scheme
(or some other Lisp) for solving their problem and end up with writing
their own Lisp-like (language|file format) and interpreter(s) for it. So
you better watch yourself. If this effort is going to be something more
permament than "done and go", then you will probably find yourself in the
situation of reinventing the wheel, as your project goes more and more
complex. If this is going to happen, you might use existing wheels and
save some time for thinking on something better.

Having the classes in memory, editing their attributes, adding methods
and then serialising them to code has an elegance to it in my mind and
would hopefully simplify things. The result would be directly
executable by the interpreter itself and editable by python
developers. But I fear that I may need an intermediary notation/format
for storage. Any advice/hints would be very useful.

Yes, there is something elegant in this way of thinking, at least for me.

As I (indirectly) said above, I am not up to date with recent Python
abilities (I still use Python for a lot of things, I just do not overuse
it for things, that it seems to be not so good at). I hope someone knows
better and steps in to prove me wrong - it is always good to learn
something new. Also, if you have just one problem, then learning a new
language because of it is a bit too much (or maybe not, if you enjoy it
). But if you plan doing similar or more difficult things in the
future, then knowing Scheme may be good for you. Even if you choose to not
use it, you will be aware of what can be done with it so this knowledge
can provide some ready to use templates.

And yes, there are also some other languages in Lisp family, but I think
Scheme is best choice if you don't know any of them already. It is quite
small, it is well defined (I mean, it has some real specification,
instead of being "specified by its implementation") and there is a lot of
info about it on the net that could be easily understood. And, last but
not least, it has metaprogramming included with nice bunch of additional
stuff. For a start, PLT's DrScheme looks nice (IMHO - yes, there are
other nice looking Scheme implementations but this one is probably best
fitted for a beginner):

http://www.plt-scheme.org/

Regards,
Tomasz Rola

--
** A C programmer asked whether computer had Buddha's nature. **
** As the answer, master did "rm -rif" on the programmer's home **
** directory. And then the C programmer became enlightened... **
** **
** Tomasz Rola mailto:[email protected] **

 

[Wilson]

Howdy,

I am not sure if my remarks will be of any use for you, but here it goes.



Maybe you have actually two problems.

One problem is processing source code into something useful, but
not executable. So it does not necessarily need metaprogramming to be
solved, just some "code comprehension" (like get a python grammar,
lex/yacc and read a lot of docs how to use them).

I do want my resultant code to be executable; I want to be able to use
"import" to get programmatical access to the classes and structures it
contains, such that they can be manipulated, added to, exercised and
drawn (using graphviz) via introspection... After any manipulation,
i'd like to be able to flatten them back into a .py file (as source
such that they are readable as Python code).

Well, "everything" is doable in a Turing-complete language . However,
this one problem - code generation and using it on the fly - may be better
solved with some other language, perhaps?

Not that I want to dissuade you (or anyone else) from using Python, but
last time I checked the state of things (was a good couple of years ago, I
wanted to solve some math-related problem with metaprogramming in Python)
I decided I would do better if I learned Scheme. At least in Scheme,
program structure is so simple, it is just a list (or a sequence of them),
so manipulating code is IMHO much easier.

I mean, even if in theory every language in Turing-complete class is equal
to each other (one can do the same kind of stuff in one as in the other),
in practice they are not equal at all. So doing everything in one ultimate
language of choice is not very wise from my point of view.

It does not mean you should immediately go to

http://en.wikipedia.org/wiki/Scheme_(programming_language)

but you may do so, and then make an informed choice by yourself. To be
better informed, you may also google for more material - introductions,
opinions, comparisons and some first-hand experience.

On the other hand, writing "metafiles" and interpreters for them - ugh...
. I have read somewhere, that some programmers resist using Scheme
(or some other Lisp) for solving their problem and end up with writing
their own Lisp-like (language|file format) and interpreter(s) for it. So
you better watch yourself. If this effort is going to be something more
permament than "done and go", then you will probably find yourself in the
situation of reinventing the wheel, as your project goes more and more
complex. If this is going to happen, you might use existing wheels and
save some time for thinking on something better.

I appreciate these comments, and I know the quote that you are
refering to although I can't find it now... Something like:
" Every sufficiently large application has a poor/incomplete
implementation of LISP embedded within it ". I've looked at LISP
before and do appreciate its elegance, but Python has a beauty of its
own in its pragmatism, standard libraries and community. So I'll
choose to stick with it.

Yes, there is something elegant in this way of thinking, at least for me.

As I (indirectly) said above, I am not up to date with recent Python
abilities (I still use Python for a lot of things, I just do not overuse
it for things, that it seems to be not so good at). I hope someone knows
better and steps in to prove me wrong - it is always good to learn
something new. Also, if you have just one problem, then learning a new
language because of it is a bit too much (or maybe not, if you enjoy it
). But if you plan doing similar or more difficult things in the
future, then knowing Scheme may be good for you. Even if you choose to not
use it, you will be aware of what can be done with it so this knowledge
can provide some ready to use templates.

I've been through quite a few of the SICP lectures and again, do
appreciate its elegance and they have changed the way I program.

And yes, there are also some other languages in Lisp family, but I think
Scheme is best choice if you don't know any of them already. It is quite
small, it is well defined (I mean, it has some real specification,
instead of being "specified by its implementation") and there is a lot of
info about it on the net that could be easily understood. And, last but
not least, it has metaprogramming included with nice bunch of additional
stuff. For a start, PLT's DrScheme looks nice (IMHO - yes, there are
other nice looking Scheme implementations but this one is probably best
fitted for a beginner):

http://www.plt-scheme.org/

Regards,
Tomasz Rola

Thanks for your comments. But I believe my solution may lie in using a
template language such as cheetah. Too much is already invested in
Python!

Best Regards,
Paul

 

[John Nagle]

Hi all,

I have an interesting problem that I'm hoping can be solved with
metaprogramming, but I don't know how far Python supports code
generation (and I don't know if I'm taking the correct approach
either... hence why I'm asking on this group):

I'd like to write a program that writes/manipulates a statemachine. My
idea was that I would define states in a .py file along with their
transitions and other metadata.

You're probably better off coming up with a representation for a
state machine (which might be a pickled collection of Python objects)
then providing a "compile" operation which cranks out a .py file
with code to implement the state machine.

John Nagle

 

[Tomasz Rola]

" Every sufficiently large application has a poor/incomplete
implementation of LISP embedded within it ".

Yep, this is either exact or very close copy of what I have read.

I've looked at LISP
before and do appreciate its elegance, but Python has a beauty of its
own in its pragmatism, standard libraries and community. So I'll
choose to stick with it. [...]
I've been through quite a few of the SICP lectures and again, do
appreciate its elegance and they have changed the way I program. [...]
Thanks for your comments. But I believe my solution may lie in using a
template language such as cheetah. Too much is already invested in
Python!

Best Regards,
Paul

Ok, so you do know something about Lisp - that is good . Of course,
since you have already existing Python code, converting is not for you.
Just to make sure you stay informed, there is CLPython:

http://common-lisp.net/project/clpython/

I did not try this thing, but I am very pleased it exists. I have some
Python code myself and knowing there is a way to reuse it if I choose some
other way makes me feel better. Or rather, knowing I can easily merge the
two (or more) ways.

Regards,
Tomasz Rola

--
** A C programmer asked whether computer had Buddha's nature. **
** As the answer, master did "rm -rif" on the programmer's home **
** directory. And then the C programmer became enlightened... **
** **
** Tomasz Rola mailto:[email protected] **

 

[Wilson]

    You're probably better off coming up with a representation for a
state machine (which might be a pickled collection of Python objects)
then providing a "compile" operation which cranks out a .py file
with code to implement the state machine.

                                John Nagle

I agree. I guess my 'representation' is the big question!

Thanks!

 

[castironpi]

Hi all,

My problem is that I don't know if it's possible to edit these states
and then write them back to .py. Firstly, if my editing tool was to
create a new state, I would want to create the class (using type) and
attach it to the imported state module somehow. I can't find

Thanks,
Paul

Yes it's possible, using type, but you can't attach it to the module
in any persistent way. The next time you load the module, it's back
to the way it was.

I think this may be the hang-up in your design: if you think about
generating lines of code directly from the user's actions, you might
stay on the right track. At that point, you can place them in the .py
file. Then you can reimport it, or just execute the new lines
directly.

If this is your class,

class Auto54701:
def next( self, input ):
if( something ):
return value1
return value2

Here's how to work with it at the interactive prompt:
... def next( self, input ):
... if( something ):
... return value1
... return value2
... """'class Auto54701:\n def next( self, input ):\n if( something ):
\n ret
urn value1\n return value2\n'
And by the way, you can't use pickle to store types. There needs to
be some fleshed-out code to back them up.

Two more alternatives, which I can discuss more later, one where you
implement code as data, but it ends up being a lot like Python
anyway. It only works if your classes' methods are going to be
simple, probably sequential if-s and assignments at most.

class Auto54701:
def next( self, input ):
if( something1 ):
return value1
elif( something2 ):
return value2
elif( something3 ):
return value3
elif( something4 ):
return value4
return value5

can be stored in data as

[ ( something1, value1 ), ( something2, value2 ), ... ]

, and just checked in a loop, that you only have in code once, rather
than many times.

class GenericAuto:
def next( self, input ):
for something, value in conditionpairs:
if( something ):
return value
return valuen

Two, if all your methods will have uniform signatures and closures,
you can store class methods as only their co_code objects:
'd\x00\x00S'

And fabricate them dynamically into full live types as needed.

 

[Wilson]

Hi all,

My problem is that I don't know if it's possible to edit these states
and then write them back to .py. Firstly, if my editing tool was to
create a new state, I would want to create the class (using type) and
attach it to the imported state module somehow. I can't find

Thanks,
Paul

Yes it's possible, using type, but you can't attach it to the module
in any persistent way.  The next time you load the module, it's back
to the way it was.

I think this may be the hang-up in your design: if you think about
generating lines of code directly from the user's actions, you might
stay on the right track.  At that point, you can place them in the .py
file.  Then you can reimport it, or just execute the new lines
directly.

If this is your class,

class Auto54701:
  def next( self, input ):
     if( something ):
       return value1
     return value2

Here's how to work with it at the interactive prompt:

...   def next( self, input ):
...      if( something ):
...        return value1
...      return value2
... """>>> s

'class Auto54701:\n  def next( self, input ):\n     if( something ):
\n       ret
urn value1\n     return value2\n'>>> exec( s )
<class __main__.Auto54701 at 0x00A80C60>



And by the way, you can't use pickle to store types.  There needs to
be some fleshed-out code to back them up.

Two more alternatives, which I can discuss more later, one where you
implement code as data, but it ends up being a lot like Python
anyway.  It only works if your classes' methods are going to be
simple, probably sequential if-s and assignments at most.

class Auto54701:
  def next( self, input ):
     if( something1 ):
       return value1
     elif( something2 ):
       return value2
     elif( something3 ):
       return value3
     elif( something4 ):
       return value4
    return value5

can be stored in data as

  [ ( something1, value1 ), ( something2, value2 ), ... ]

, and just checked in a loop, that you only have in code once, rather
than many times.

class GenericAuto:
  def next( self, input ):
    for something, value in conditionpairs:
      if( something ):
        return value
    return valuen

Two, if all your methods will have uniform signatures and closures,
you can store class methods as only their co_code objects:

'd\x00\x00S'

And fabricate them dynamically into full live types as needed.

Thanks for your comments and advice. This second option intrigues me;
could you elaborate further, I don't follow you...

Thanks Paul

 

[castironpi]

Thanks for your comments and advice. This second option intrigues me;
could you elaborate further, I don't follow you...

Thanks Paul

Depending on the complexity of the functions, a code string could be
all you need to store to determine (redetermine) a function's
behavior. For something moderately simple,

def trans1( self, prev, trans ):
if prev== 0 and trans== 'a':
return 1
if prev== 1 and trans== 'b':
return 0
return prev

I found you need to store code.co_nlocals, code.co_code, and
code.co_consts, to distinguish from a blank stub. With extra
variables, I needed code.co_names and code.co_varnames too. To
recreate a code object completely, you need 12 variables (14 to
include closures), some of which are composite objects and would need
to be pickled to be stored.

Then you can build a new code object, then a new function object, then
a new method object, then you can call it. Instead of a module of
code, perhaps you could have a datafile containing only these values,
up to twelve per record, one record for each different function you
have.

Here is the benefit:

newcode= dupecode( oldcode, codet1 )
newfun= FunctionType( newcode, {} )
stub.stub= MethodType( newfun, stub )
prev= stub.stub( prev, trans )
print prev

You can loop over these five lines, re-loading function data with
'dupecode', executing it, then reloading the next one, and you have a
different function. Additions to your database of functions would
start in source first (unless you construct each parameter, in
particular co_code, by hand, which you may want), then get compiled,
then go in.

Here is the complete constructor for a code object:
Help on code object:

class code(object)
| code(argcount, nlocals, stacksize, flags, codestring, constants,
names,
| varnames, filename, name, firstlineno, lnotab[, freevars[,
cellvars]])

Here's the constructor in Python 3.0:

class code(object)
| code(argcount, kwonlyargcount nlocals, stacksize, flags,
codestring,
| constants, names, varnames, filename, name, firstlineno,
| lnotab[, freevars[, cellvars]])

I defined Stub.stub like this:

class Stub:
def stub( self, prev, trans ):
return prev
stub= Stub( )

You need imports from the types module:

from types import MethodType, FunctionType, CodeType

And here is 'dupecode', which currently only replaces five of the old
function's members with new ones:

def dupecode( old, new ):

newcode= CodeType( old.co_argcount, new.co_nlocals, old.co_stacksize,
old.co_flags,
new.co_code,
new.co_consts, new.co_names,
new.co_varnames, old.co_filename, old.co_name, old.co_firstlineno,
old.co_lnotab )

return newcode

 

[Wilson]

Thanks for your comments and advice. This second option intrigues me;
could you elaborate further, I don't follow you...

Thanks Paul

Depending on the complexity of the functions, a code string could be
all you need to store to determine (redetermine) a function's
behavior.  For something moderately simple,

def trans1( self, prev, trans ):
        if prev== 0 and trans== 'a':
                return 1
        if prev== 1 and trans== 'b':
                return 0
        return prev

I found you need to store code.co_nlocals, code.co_code, and
code.co_consts, to distinguish from a blank stub.  With extra
variables, I needed code.co_names and code.co_varnames too.  To
recreate a code object completely, you need 12 variables (14 to
include closures), some of which are composite objects and would need
to be pickled to be stored.

Then you can build a new code object, then a new function object, then
a new method object, then you can call it.  Instead of a module of
code, perhaps you could have a datafile containing only these values,
up to twelve per record, one record for each different function you
have.

Here is the benefit:

newcode= dupecode( oldcode, codet1 )
newfun= FunctionType( newcode, {} )
stub.stub= MethodType( newfun, stub )
prev= stub.stub( prev, trans )
print prev

You can loop over these five lines, re-loading function data with
'dupecode', executing it, then reloading the next one, and you have a
different function.  Additions to your database of functions would
start in source first (unless you construct each parameter, in
particular co_code, by hand, which you may want), then get compiled,
then go in.

Here is the complete constructor for a code object:

Help on code object:

class code(object)
 |  code(argcount, nlocals, stacksize, flags, codestring, constants,
names,
 |        varnames, filename, name, firstlineno, lnotab[, freevars[,
cellvars]])

Here's the constructor in Python 3.0:

class code(object)
 |  code(argcount, kwonlyargcount nlocals, stacksize, flags,
codestring,
 |        constants, names, varnames, filename, name, firstlineno,
 |        lnotab[, freevars[, cellvars]])

I defined Stub.stub like this:

class Stub:
        def stub( self, prev, trans ):
                return prev
stub= Stub( )

You need imports from the types module:

from types import MethodType, FunctionType, CodeType

And here is 'dupecode', which currently only replaces five of the old
function's members with new ones:

def dupecode( old, new ):

        newcode= CodeType( old.co_argcount, new.co_nlocals, old..co_stacksize,
                old.co_flags,
                new.co_code,
                new.co_consts, new.co_names,
                new.co_varnames, old.co_filename, old.co_name, old.co_firstlineno,
                old.co_lnotab )

        return newcode

Still don't really understand this so I'm going to admit defeat.
Thanks all for your advice... Very much appreciated!

 

[castironpi]

Still don't really understand this so I'm going to admit defeat.
Thanks all for your advice... Very much appreciated!

I was describing an alternative to storing functions in a way that
wasn't in serial in plain text. It was off-topic from state-machine
transitions.

Can you start with this?

# state, input, next state
transitions= [
( 0, 'a', 1 ),
( 1, 'a', 2 ),
( 2, 'a', 0 ),
( 0, 'b', 0 ),
( 1, 'b', 0 ),
( 2, 'b', 2 )
]

What further?

 

[Wilson]

Still don't really understand this so I'm going to admit defeat.
Thanks all for your advice... Very much appreciated!

I was describing an alternative to storing functions in a way that
wasn't in serial in plain text.  It was off-topic from state-machine
transitions.

Can you start with this?

# state, input, next state
transitions= [
  ( 0, 'a', 1 ),
  ( 1, 'a', 2 ),
  ( 2, 'a', 0 ),
  ( 0, 'b', 0 ),
  ( 1, 'b', 0 ),
  ( 2, 'b', 2 )
]

What further?

My problem relates to the states really. I'd like to import a file
containing the state classes into a "state editor" that is just a
class editor really. Then, at *runtime* I'd like to add and remove
methods from this class and change its attributes (__dict__). I'd also
like to be able to add and remove classes from the module. When done,
I was hoping to do introspection upon which classes were associated
with this module and write back the modified class definitions to the
same module. Next time the module is imported, the changes would
remain intact. Unfortunately, inspect.getsource() reads the original
file and does not magically return source from the runtime objects.

Hope that makes sense!
Paul

 

[castironpi]

I was describing an alternative to storing functions in a way that
wasn't in serial in plain text.  It was off-topic from state-machine
transitions.

Can you start with this?

# state, input, next state
transitions= [
  ( 0, 'a', 1 ),
  ( 1, 'a', 2 ),
  ( 2, 'a', 0 ),
  ( 0, 'b', 0 ),
  ( 1, 'b', 0 ),
  ( 2, 'b', 2 )
]

What further?

My problem relates to the states really. I'd like to import a file
containing the state classes into a "state editor" that is just a
class editor really. Then, at *runtime* I'd like to add and remove
methods from this class and change its attributes (__dict__). I'd also
like to be able to add and remove classes from the module. When done,
I was hoping to do introspection upon which classes were associated
with this module and write back the modified class definitions to the
same module. Next time the module is imported, the changes would
remain intact. Unfortunately, inspect.getsource() reads the original
file and does not magically return source from the runtime objects.

Hope that makes sense!
Paul

I see. You want:

import states

def funa( ):
something( )

states.funa= funa

And have those changes appear in the actual text of 'states.py'.

Are you certain that classes are your best choice for representing
states? I am interpreting that you want one class per state, and one
method per transition out of that state. How close is that?

If you are seeking to generate a .py file from a data structure, why
not just edit the data structure directly? If you do need a .py file,
can you just regenerate it?

 

[Iain King]

Yep, this is either exact or very close copy of what I have read.

It's Greenspun's Tenth Rule of Programming:

"Any sufficiently complicated C or Fortran program contains an ad-hoc,
informally-specified bug-ridden slow implementation of half of Common
Lisp."

Iain

 

[Wilson]

It's Greenspun's Tenth Rule of Programming:

"Any sufficiently complicated C or Fortran program contains an ad-hoc,
informally-specified bug-ridden slow implementation of half of Common
Lisp."

Iain

Thanks for that. Makes my attempt look pathetic!