Metaprogramming Example

[andrew cooke]

Hi,

Thanks for the help a couple of days ago. I completed what I was
doing and wrote a summary which I've posted at http://acooke.org/cute/PythonMeta0.html
(it's kind of long to post here). I hope it might be useful to
someone else - it's complete code for a simple metaprogramming task
that uses metaclasses and descriptors.

I'd also appreciate further feedback if I've done anything stupid or
if there's some interesting approach I've missed that might work
better.

Thanks again,
Andrew

 

[bruno.desthuilliers]

Hi,

Thanks for the help a couple of days ago. I completed what I was
doing and wrote a summary which I've posted athttp://acooke.org/cute/PythonMeta0.html
(it's kind of long to post here). I hope it might be useful to
someone else - it's complete code for a simple metaprogramming task
that uses metaclasses and descriptors.

I'd also appreciate further feedback if I've done anything stupid or
if there's some interesting approach I've missed that might work
better.

1/ Not about metaprogramming, but ORMs. You write:
"""
I could use an existing ORM package, but my experience with
them hasn't been that positive (in particular, I find SQL to be very
useful and want to use it more than is normally possible with standard
ORM).
"""

Makes me wonder if you really took a close-enough look at
SQLAlchemy... (please ignore this if you did)


2/ about your code:

if (self.ignore_identical is False or
new_value is not obj._auto_write_dict[self.name]):

Take care, 'is' is the identity test operator, not the equality test
operator. I think you *really* want the equality operator here. If you
don't understand why, here's a sample run that's worth a longer
explanation:

Also, the parens are not needed, and boolean tests don't need to be
that explicit (ie, you don't necessarily need to test a boolean
expression against True or False). And, last point, double-negations
can be harder to understand.

May I suggest:

if not (self.ignore_identical and new_value ==
obj._auto_write_dict[self.name]):


3/ code again:

attr = getattr(obj.__class__,
"%s%s" % (self.prefix, self.name), None)
obj._auto_write_dict[self.name] = attr(obj, new_value)

Three points here:
- build the setter name once and "cache" it. It'll make your code more
readable (and save a couple cycles)

def __init__(self, name, ignore_identical=True, prefix='_set_'):
self.name = name
self.ignore_identical = ignore_identical
self.prefix = prefix
self.settername = "%s%s" % (self.prefix, self.name)

then use self.settername in __set__

- retrieving the setter on the object (instead of it's class) would be
more pythonic, and way simpler to read.

- None is not callable, so you code will crash if the object doesn't
define a setter. You can either test the return value of getattr
against None, put a try/except around the call, or use an identity
function as default value for getattr (which is what I'd do here)

Fix:

setter = getattr(obj, self.settername, lambda x : x)
obj._auto_write_dict[self.name] = setter(new_value)



About the article itself, I'd say it makes a good introduction to a
common metaprogramming pattern, but you'd have to get the opinion of
someone that has no experience with these topics. Ho, and yes : one
day, you'll get why it's such a good thing to unite functions and
methods !-)

My 2 cents

 

[andrew cooke]

On Apr 17, 7:12 am, "(e-mail address removed)"
[...]

Thanks very much!

These are useful pointers. I'll update my code accordingly.

At one point you pointed out I didn't need parentheses and I agree - I
was using them to avoid having a line continuation backslash (I think
I read to do that in a style guide recently).

One other question. I had "foo is False" and you said I need
equality, which is a good point. However, in any other language "not
foo" would be preferable. I was surprised you didn't suggest that
(and I'm unsure now why I didn't write it that way myself). Is there
some common Python standard that prefers "foo == False" to "not foo"?

Thanks,
Andrew

PS Is there anywhere that explains why Decorators (in the context of
functions/methods) are so good? I've read lots of things saying they
are good, but no real justification of why. To me it looks more like
"re-arranging deck chairs on the Titanic" - you're just moving where
the hack happens from one place to another. Is the point that now the
hack is more visible and hence modifiable?

 

[Paul McGuire]

One other question.  I had "foo is False" and you said I need
equality, which is a good point.  However, in any other language "not
foo" would be preferable.  I was surprised you didn't suggest that
(and I'm unsure now why I didn't write it that way myself).  Is there
some common Python standard that prefers "foo == False" to "not foo"?

In addition to the problematic "foo is False" test, Bruno was also
saying that assertions of True tend to be more readable than negated
assertions of False.

In your original, you were testing for not P or not Q, Bruno recoded
this to the equivalent not(P and Q). That is, the direct way to
change the 'is' identity testing to equality testing would have been:

if (not self.ignore_identical or
new_value != obj._auto_write_dict[self.name]):

But Bruno further changed this to:

if not (self.ignore_identical and
new_value == obj._auto_write_dict[self.name]):


-- Paul

 

[bruno.desthuilliers]

[...]

Thanks very much!

These are useful pointers. I'll update my code accordingly.

At one point you pointed out I didn't need parentheses and I agree - I
was using them to avoid having a line continuation backslash (I think
I read to do that in a style guide recently).

Ok. I personnaly prefer \ continuations, but that's another
troll^M^discussion !-)

One other question. I had "foo is False" and you said I need
equality, which is a good point. However, in any other language "not
foo" would be preferable.

And it's indeed the case in Python.

I was surprised you didn't suggest that

I did - even if somewhat implicitly -, as Paul brillantly explained.

(and I'm unsure now why I didn't write it that way myself). Is there
some common Python standard that prefers "foo == False" to "not foo"?

Definitively not.

FWIW, True and False are later additions to Python, which has a much
more generic concept of what's true or false (notice the lowercase) in
the context of a boolean expression, that is:
- False, None, numeric zero are false
- an object that has a __nonzero__ method has the truth value of the
result of calling this method
- a "sizeable" object (one which defines a __len__ method) is true if
len(obj) > 0, else it's false (which implies that empty dicts, lists,
tuples and strings are false)
- anything else is true

(please someone correct me if I forgot something here).

PS Is there anywhere that explains why Decorators (in the context of
functions/methods) are so good? I've read lots of things saying they
are good, but no real justification of why. To me it looks more like
"re-arranging deck chairs on the Titanic" - you're just moving where
the hack happens from one place to another. Is the point that now the
hack is more visible and hence modifiable?

I wouldn't call function decorators "a hack" - it's just a pretty
common use of HOFs. Now wrt/ the @decorator syntax: yes, you're plain
right, the GoodThing(tm) is that it makes the use of the HOF clearly
visible at the top of the function definition so you just can't miss
it.

 

[andrew cooke]

bruno:

Ho, and yes : one day, you'll get why it's such a good thing to unite
functions and methods !-)
me:
PS Is there anywhere that explains why Decorators (in the context of
functions/methods) are so good? I've read lots of things saying they
are good, but no real justification of why.

in the text above i wrote "Decorators" rather than "Descriptors". it
was in response to bruno's comment, also above. sorry for the
confusion.
also, sorry for the inflammatory language - by referring to the
titanic
i didn't mean that python was a disaster, rather that the "iceberg" is
still there (i am not 100% sure what the iceberg is, but it's
something
to do with making namespaces explicit in some places and not others).

anyway, it was only a small point. thanks for all the replies.
didn't
respond earlier as i was at work. now i can go back and polish that
code...

andrew

 

[Bruno Desthuilliers]

andrew cooke a écrit :

bruno:

in the text above i wrote "Decorators" rather than "Descriptors". it
was in response to bruno's comment, also above. sorry for the
confusion.

Ho, you meant Descriptors ?

Well, a first point is that the support for methods is built on a
combination of two "general purpose" constructs - callable objects and
the descriptor protocol - instead of being a special-case construct by
itself. IOW, this is build *with* Python itself, instead of being built
*into* Python.

Practically, this means that (amongst other niceties) :
- you can define functions outside classes and use them as instance or
class methods
- you can add/replaces methods dynamically on a per-class or
per-instance basis
- you can access the function object of a method and use it as a function
- you can define your own callable types, that - if you implement the
appropriate support for the descriptor protocol - will be usable as
methods too

also, sorry for the inflammatory language

Which one ???

by referring to the titanic
i didn't mean that python was a disaster, rather that the "iceberg" is
still there (i am not 100% sure what the iceberg is, but it's
something
to do with making namespaces explicit in some places and not others).

I guess you're thinking of the self argument, declared in the function's
signature but not "explicitly passed" when calling the method ?

If so, the fact is you *do* pass it explicitly - by calling the function
*as a method of an objet*. Given the following definitions:

def func(obj, data):
print "obj : %s - data : %s" % (obj, data)

class Foo(object):
meth = func

f = Foo()

What the difference between:

func(f, 42)

and

f.meth(42)

In both cases, f is directly and explicitly involved as one of the
"targets" of the call.

My 2 cents...

 

[andrew cooke]

On Apr 18, 4:48 am, Bruno Desthuilliers <bruno.
(e-mail address removed)> wrote:
[...]

Practically, this means that (amongst other niceties) :
- you can define functions outside classes and use them as instance or
class methods
- you can add/replaces methods dynamically on a per-class or
per-instance basis
- you can access the function object of a method and use it as a function
- you can define your own callable types, that - if you implement the
appropriate support for the descriptor protocol - will be usable as
methods too

ok, that's convincing (i had thought the majority of these were
already
possible, albeit with some kind of hard-coded "magic" behind the
scenes).

[...]

I guess you're thinking of the self argument, declared in the function's
signature but not "explicitly passed" when calling the method ?

not really. more to do with when namespaces (i am not sure i have the
right term - the dictionary that provides the mapping from name to
object)
are explicit or implicit. for example, python has closures (implicit
lookup) and "self" (explicit lookup).

but as i said, i don't have a clear argument - something just feels
"odd".
at the same time, i know that language design is a practical business
and so this is probably not important.

finally, thank you for pointing me to sql alchemy (i think it was
you?).
it really is excellent.

andrew

 

[Kay Schluehr]

PS Is there anywhere that explains why Decorators (in the context of
functions/methods) are so good?

We had kind of an inverse discussion a while ago when someone asked
about the fate of aspect oriented programming (AOP) in Python. My
answer was that technically "aspect weaving" using code generators is
dead in application programming [1] and decorators are handy,
lightweight, local and controllable while serving very similar
purposes.

[1] There are occasions where source code weaving might still has its
place. Profiling for example or code coverage. Just examine this
interesting blog article:

http://nedbatchelder.com/blog/200804/wicked_hack_python_bytecode_tracing.html

and the subsequent discussion.

 

[bruno.desthuilliers]

[...]

Practically, this means that (amongst other niceties) :
- you can define functions outside classes and use them as instance or
class methods
- you can add/replaces methods dynamically on a per-class or
per-instance basis
- you can access the function object of a method and use it as a function
- you can define your own callable types, that - if you implement the
appropriate support for the descriptor protocol - will be usable as
methods too

ok, that's convincing (i had thought the majority of these were
already
possible, albeit with some kind of hard-coded "magic" behind the
scenes).

Yep, the whole point is that it's not that hard-coded anymore. Python
exposes most of it's inner mechanisms, so that you can taylor quite a
lot of things to your needs. This is quite useful for writing clean
frameworks needing very few boilerplate in the user code.

[...]

I guess you're thinking of the self argument, declared in the function's
signature but not "explicitly passed" when calling the method ?

not really. more to do with when namespaces (i am not sure i have the
right term - the dictionary that provides the mapping from name to
object)
are explicit or implicit. for example, python has closures (implicit
lookup) and "self" (explicit lookup).

but as i said, i don't have a clear argument - something just feels
"odd".
at the same time, i know that language design is a practical business
and so this is probably not important.

The fact is that everything you do with closures can be done with
objects. OTHO, there are quite a lot of cases where defining a
specific class would be just way too heavy, and a closure is much more
lightweight. So yes, it's a matter of "practicality beats purity".
While trying to remain as clean as possible, Python is definitively a
practical language.

finally, thank you for pointing me to sql alchemy (i think it was
you?).

Seems so.

it really is excellent.

Indeed !-)