metaclass and customization with parameters

[zipher]

After searching through comp.lang.python and the web regarding
metaclasses, I could not find an example for customing classes using
metaclass parameters.

I want to be able to create a class at runtime by calling some
function or 'meta-constructor' which returns a customized class and
sets a class attribute according a given parameter.

Ideally, I'd be able to do something like:
new class

(Aside: the issue of magically knowing the variable name (i.e. 'Bag')
in that first assignment statement and setting the class name
accordingly would be ideal, but I could live with passing the name as
a parameter to Metadict if necessary.)

There would also be a number of new and re-defined methods which would
be part of all classes constructed and returned by calls to Metadict.
For example, somewhere I need to re-define dict's __setitem__ method
to something like:
.... self[key] = self.filter(value) #coerce/validate values before
adding to dict

Ideally, all such methods would be in their own class definition since
they will be common to all classes created by Metadict (perhaps they
would need be defined within Metadict itself).


This seems like a simple thing to want, but unusally convoluted to
actually implement. I'm not even sure where to start.

Any pointers?

Thank you,

zipher

 

[Alex Martelli]

After searching through comp.lang.python and the web regarding
metaclasses, I could not find an example for customing classes using
metaclass parameters.

When a class statement executes, Python determines the metaclass (e.g.
by finding a '__metaclass__' attribute in the class body) and then calls
it with three parameters -- never more, never fewer:

class name(bases):
__metaclass__ = metaboo
...rest of class body snipped...

is exactly equivalent to:

name = metaboo('name', bases, d)

where 'd' is the dict that remains after execution of the class body
(all barenames bound in the body -- be that by def, class, assignment,
import, whatever -- are left as entries in that 'd').

I want to be able to create a class at runtime by calling some

All classes are created at runtime, normally by executing (at runtime:
there is no other 'time'...;-) a 'class' statement.

function or 'meta-constructor' which returns a customized class and
sets a class attribute according a given parameter.

Ideally, I'd be able to do something like:

'Bag'

The normal Python syntax for that purpose might rather be:

class Bag:
__metaclass__ = Metadict
filter = int

and for that use, Medadict would be rather simple to write... but
your stated purpose (to this point) is even better accomplished by:

class Bag(dict):
filter = int

new class

(Aside: the issue of magically knowing the variable name (i.e. 'Bag')
in that first assignment statement and setting the class name
accordingly would be ideal, but I could live with passing the name as
a parameter to Metadict if necessary.)

If you want to explicitly call Metadict as a factory, rather than use a
'class' statement, yep, there is then no way in which Metadict can know
what name its result will be bound to, so you'd have to pass it. But
why isn't a class statement even better?

There would also be a number of new and re-defined methods which would
be part of all classes constructed and returned by calls to Metadict.
For example, somewhere I need to re-define dict's __setitem__ method
to something like:
... self[key] = self.filter(value) #coerce/validate values before
adding to dict

Ideally, all such methods would be in their own class definition since
they will be common to all classes created by Metadict (perhaps they
would need be defined within Metadict itself).

No, this is best done by inheritance:

class Bag(filtered_dict):
filter = int

with, e.g:

class filtered_dict(dict):
def __setitem__(self, key, value):
dict.__setitem__(self, key, self.filter(value))

This seems like a simple thing to want, but unusally convoluted to
actually implement. I'm not even sure where to start.

Any pointers?

I would start with inheritance, because I don't see what value a custom
metaclass would add; indeed I see only downsides, such as needing to
pass the intended class name, if you want to use explicit-call syntax.

If it does turn out that subclasses of filtered_dict all DO need a
custom metaclass, for reasons connected to specs you haven't yet
mentioned, just add the metaclass (with a __metaclass__ in the body) to
filtered_dict, and every subclass thereof will inherit it. That's all
there is to it... class statement, and inheritance, appear to give you
all that you have required/explained so far, except the kind of syntax
you deem ideal (but is really second best, due to the need to pass a
class name...).


Alex

 

[zipher]

All classes are created at runtime, normally by executing (at runtime:
there is no other 'time'...;-) a 'class' statement.

Yes, I should have put that in quotes. What I meant to convey was
that I want to be able to create a class 'on the fly' without having
to define it first.

The normal Python syntax for that purpose might rather be:

class Bag:
__metaclass__ = Metadict
filter = int

and for that use, Medadict would be rather simple to write... but
your stated purpose (to this point) is even better accomplished by:

class Bag(dict):
filter = int

Yes, your suggestions were my second, and first implemenations
respectively. However, both require defining a class by the 'user',
so to speak, as opposed to letting the user simply call a 'function'
that returns the customized class with the filter class attribute
already set.

Another reason it would be helpful to be able to wrap this in an
expression like "Metadict(filter=int)" is because I can then use this
as a parameter passed to other parts of the application. (Although,
as you suggested, I'd have to pass the name as well.)

For example, in keeping with the aforementioned example, consider
creating a basic integer-weighted Graph class starting with the
following:

#A Graph is a dict whose values are Vertex.
#A Vertex is a dict whose values are int.

Graph = Metadict(name='Graph', filter=Metadict(name='Vertex', filter=int))
g = Graph() # create outgoing links on graph vertex 1 to 3,4,5:
g[1] = {3: 1, 4: 2.0, 5: '3'}

#this raw dict above will be converted to Vertex because
# of the Graph.filter function and assigned to g[1]
#Vertex.filter automatically coerced dict values to ints:

g[1]

Vertex({3: 1, 4: 2, 5: 3})

But perhaps a simpler way to illustrate my intent is to just imagine a
series of dictionary classes which restrict all their values to a
certain type (through the filter function) before insertion into the
dictionary. Dictionaries of ints, lists, and strings along with
custom filters that, say, validate the values for correct spelling, or
whatever.

Now you ("I" ;^) don't want to require the user to create a new class
for every possible filtered dict variety. And, yes, you could avoid
this by just subclassing dict and passing the filter function to
__init__ and setting it as an instance attribute, but then you have an
extra attribute stored for every instance of your filtered dictionary
and the (albeit) small cost of assigning this attribute for every
instance created. (Consider again the Graph example, where there are
going to be many instances of the Vertex class within the Graph and
you don't want a unique filter function for each Vertex instance
within the Graph.)

This would seem like an ideal application for a metaclass (in the
abstract sense, not necessarily in the way Python implements the
concept): I want something that will create and return a specialized
class based on the parameters I pass to it.

Alex, thanks for your response.

zipher

 

[Mike C. Fletcher]

zipher wrote:
....

Yes, your suggestions were my second, and first implemenations
respectively. However, both require defining a class by the 'user',
so to speak, as opposed to letting the user simply call a 'function'
that returns the customized class with the filter class attribute
already set.

Another reason it would be helpful to be able to wrap this in an
expression like "Metadict(filter=int)" is because I can then use this
as a parameter passed to other parts of the application. (Although,
as you suggested, I'd have to pass the name as well.)

I could *swear* I sent you an answer which did exactly what you wanted
just a few minutes after your first posting (at least Mozilla claims I did):

def Metadict( name, **args ):
return type( name, (baseClassWithCommonCode,), args )

That is, a function that you call with a name for the new class, and a
set of parameters which will be passed into the namespace of the newly
created class, (where the newly created class is to be a sub-class of
some specified base class), which returns the new class. This does,
indeed, use a meta-class (type), btw, it just doesn't use the metaclass
hook, which is what's bothering you elsewhere.

i.e.

fred = Metadict( 'fred', filter=int )

should work fine with that function, and it seems to meet all of your
other criteria modulo magically knowing the name to which it's being
assigned.

Enjoy,
Mike

________________________________________________
Mike C. Fletcher
Designer, VR Plumber, Coder
http://www.vrplumber.com
http://blog.vrplumber.com

 

[Carlos Ribeiro]

After searching through comp.lang.python and the web regarding
metaclasses, I could not find an example for customing classes using
metaclass parameters.

I want to be able to create a class at runtime by calling some
function or 'meta-constructor' which returns a customized class and
sets a class attribute according a given parameter.

I'm not a metaclass expert, but I think I can share a few useful tips
with you. I've been studying metaclasses lately, and I've fallen on
some traps that seem to be common. I apologize if my tone sound
pretentious at times -- my intention is probably better than my words.

Let's start from the very beginning. In opposition to what happens at
other widely known languages -- C, for example -- class definitions in
Python are actually a two-step process. First the class code is
*executed*; when it finishes executing, the class is created by
calling its *metaclass*.

Step 1 - Executing class declaration code
------------------------------------------------
For simple classes -- the ones with just a few def's -- the execution
step seems to be the same as with class declarations in other
languages. That happens because def's are not executed at this time -
def's are only compiled. But any code inside a class definition that
is *not* shielded inside a method is executed, and can have side
effects on the way the class is created. One can do funny things with
it. Alex Martelli has show some examples on a previous thread -- for
example a metaclass doesn't need to be a class, any callable will do,
and it's possible to do weird things with it. The following code is
such a sample:
.... for i in range(10):
.... def newmethod(self,i=i):
.... print "I'm method #%d" % i
.... newname = "method%d" % i
.... locals()[newname] = newmethod
.... I'm method #1

Although it doesn't involve metaclasses at all, it shows how dynamic
the class creation process is, and how much control one can have with
it. There are two worthy comments to make:

1) the locals() dict is the key at this step of class creation. The
local dict is one of the parameters passed to the metaclass. If you
fill the locals() with values -- either manually, as in the example
above, or by assigning values to local vars -- all the values will be
part of the class definition.

2) the signature of "def newmethod(self,i=i)" shows another
interesting side effect. The value if <i> (defined in the for loop) is
passed as a keyword paramenter, and is evaluated whenever the 'def'
statement is found. Because it changes, the def is re-run at every
loop. In the lack of this hack, the def would run only at the first
time, and all methods would in fact be the same.

Step 2 - Creating the new class
-----------------------------------------
The second step is the crucial one. A class is created from three
parameters: it's name, a list of base classes, and it's dict. The
following sample shows how to create a new class, pretty much as
you've asked:
1

By calling type() with carefully constructed parameters, it's possible
to create the class the way you want it -- including all the methods
and attributes you can possibly need. You can also supply the base
class, and it can be a dict, as you wish:

T = type('newdict', (types.DictType,), {'x':1, 'y':2})
t = T()
t.x 1
t['teste'] = 4
t {'teste': 4}
t.items() [('teste', 4)]

Things that you can't do
---------------------------
Unfortunately, and even with all the flexibility that the process
have, there are a few things that are still difficult (or impossible)
to do. For example, there is no (easy) way to intercept the creation
of new names during the 'step 1', which means that some hacks need to
be done only after this step is completed. One such example is the
recently requested method overload' recipe.

It's also possible to rename a class that is being defined (by
changing its name inside the metaclass); however, it will still be
bound to the name stated in the source code. For example:

class NewClass:
def __metaclass__(name, bases, dct):
return type('__%s__' % name, bases, dct)
<class '__main__.__NewClass__'>



--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Alex Martelli]

(e-mail address removed) (Alex Martelli) wrote in message

Yes, I should have put that in quotes. What I meant to convey was
that I want to be able to create a class 'on the fly' without having
to define it first.

That's exactly what a class statement does: it creates a class on the
fly. There is no concept of "having to define it first" in Python; you
may be thinking of other and very different languages.

Yes, your suggestions were my second, and first implemenations
respectively. However, both require defining a class by the 'user',
so to speak, as opposed to letting the user simply call a 'function'
that returns the customized class with the filter class attribute
already set.

All they require is using the statement that is normally used to create
classes, namely 'class', rather than a statement which is rarely used,
and less powerful and convenient, for the same purpose, namely an
assignment statement. The "filter attribute already set" issue appears
to me to be a red herring: why should it make any difference for the
user if the code string
filter=int
appears within parentheses (in the call to Metadict that you want) or
without them (in the 'class' statement that appears to me to be the one
obvious way to accomplish the identical purpose)?

Another reason it would be helpful to be able to wrap this in an
expression like "Metadict(filter=int)" is because I can then use this
as a parameter passed to other parts of the application. (Although,
as you suggested, I'd have to pass the name as well.)

Yep, if you want to supply a name you have to code it SOMEwhere, and why
shouldn't one code it in the normal way, right after the 'class'
keyword?

For example, in keeping with the aforementioned example, consider
creating a basic integer-weighted Graph class starting with the
following:

#A Graph is a dict whose values are Vertex.
#A Vertex is a dict whose values are int.
filter=int))

versus:

class Graph(filtered_dict):
class Vertex(filtered_dict):
filter = int
filter = Vertex

all the rest that you write in the following:

g = Graph() # create outgoing links on graph vertex 1 to 3,4,5:
g[1] = {3: 1, 4: 2.0, 5: '3'}

#this raw dict above will be converted to Vertex because
# of the Graph.filter function and assigned to g[1]
#Vertex.filter automatically coerced dict values to ints:

g[1]

Vertex({3: 1, 4: 2, 5: 3})

....just happens the same way, it makes no difference that you've used
the normal and usual way (the class statement) rather than the strange
way you want to define classes Graph and Vertex.

But perhaps a simpler way to illustrate my intent is to just imagine a
series of dictionary classes which restrict all their values to a
certain type (through the filter function) before insertion into the
dictionary. Dictionaries of ints, lists, and strings along with
custom filters that, say, validate the values for correct spelling, or
whatever.

Now you ("I" ;^) don't want to require the user to create a new class
for every possible filtered dict variety.

I could care less. You asked for some way to do exactly this: have the
user create a new class object for every filtered dict variety the user
needs. I'm not getting into whether this is a good or bad idea. I'm
just saying: the normal way to have the user create new class objects is
by having the user execute 'class' statements; I see no advantage to
your preference of having the user execute function calls instead.

And, yes, you could avoid
this by just subclassing dict and passing the filter function to
__init__ and setting it as an instance attribute, but then you have an
extra attribute stored for every instance of your filtered dictionary
and the (albeit) small cost of assigning this attribute for every
instance created. (Consider again the Graph example, where there are
going to be many instances of the Vertex class within the Graph and
you don't want a unique filter function for each Vertex instance
within the Graph.)

As I said, I'm not examining _at all_ the issue of making new classes
for each value of filter, vs storing such values in instances instead,
etc. _ALL_ I'm addressing is: if you want to make new classes, the
normal way is the 'class' statement, rather than a function call, and I
don't see why you insist on the class object creation having to happen
via a function call instead.

This would seem like an ideal application for a metaclass (in the
abstract sense, not necessarily in the way Python implements the
concept): I want something that will create and return a specialized
class based on the parameters I pass to it.

You have it: it's the 'class' statement. The only metaclass in play is
'type' (the normal metaclass of all new-style classes), of course.

Still, having hopefully communicated that your syntax preference for
function calls rather than perfectly good 'class' statements is really
very troublesome to me, and that I believe your software will be much
better if you let the user use 'class' statements to create classes
unless you have strong reasons to do otherwise (and I really don't see
what reason you have in this case), here's a couple of alternatives:

def but_WHY_ever(name, filter):
class anon(filtered_dict): filter=filter
anon.__name__ = name
return anon

or

def I_really_dont_see_WHY(name, filter):
return type(name, (filtered_dict,), dict(filter=filter))

so, now the following three constructs have become equivalent:

class Graph(filtered_dict): filter=int

Graph = but_WHY_ever('Graph', filter=int)

Graph = I_really_dont_see_WHY('Graph', filter=int)

to me, the first one seems by far the best. But if you want to confuse
the issues by using the others, go right ahead: never be it said that
Python doesn't offer you enough rope to shoot yourself in the foot!-)


Alex

 

[Alex Martelli]

...
Mostly interesting comments, snipped but hopefully useful to clear up
some confusions from readers, but one I need to comment on...:

languages. That happens because def's are not executed at this time -
def's are only compiled. But any code inside a class definition that

Each def statement IS indeed executed; I think what you mean to say is
that a def statement does not execute the body of the function it's
defining, it just stashes that body away (in a compiled form) for future
execution (each time that function is called).


Alex

 

[Carlos Ribeiro]

...
Mostly interesting comments, snipped but hopefully useful to clear up
some confusions from readers, but one I need to comment on...:


Each def statement IS indeed executed; I think what you mean to say is
that a def statement does not execute the body of the function it's
defining, it just stashes that body away (in a compiled form) for future
execution (each time that function is called).

Correct. Thanks for pointing out the difference between the def
statement (which is executed) and the body of the function. This
difference is really important, and once you *really* understand it --
as far as classes, functions and modules declarations work at runtime
-- everything just fits in perfectly.

BTW, once I finish my deep imersion into new style classes, which
includes metaclasses and descriptors, I intend to write a tutorial and
post it somewhere. It's a fascinating topic, but the documentation
seems to assume that you already know what is this stuff useful for
A tutorial would help to clarify stuff greatly.
--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Mike C. Fletcher]

Carlos Ribeiro wrote:
....

BTW, once I finish my deep imersion into new style classes, which
includes metaclasses and descriptors, I intend to write a tutorial and
post it somewhere. It's a fascinating topic, but the documentation
seems to assume that you already know what is this stuff useful for
A tutorial would help to clarify stuff greatly.

My PyCon presentation (available here
http://www.vrplumber.com/programming/ ) was explicitly targeted at
answering that question (what they are good for/when and why you would
use them).

Have fun,
Mike

________________________________________________
Mike C. Fletcher
Designer, VR Plumber, Coder
http://www.vrplumber.com
http://blog.vrplumber.com

 

[Alex Martelli]

Correct. Thanks for pointing out the difference between the def
statement (which is executed) and the body of the function. This

You're welcome!

difference is really important, and once you *really* understand it --
as far as classes, functions and modules declarations work at runtime

If you avoid using the word 'declaration' in this context, and keep
talking about these as _executable statements_ (because that's what they
are!), I think you may help your readers' understanding even more.


Alex

 

[Michele Simionato]

BTW, once I finish my deep imersion into new style classes, which
includes metaclasses and descriptors, I intend to write a tutorial and
post it somewhere. It's a fascinating topic, but the documentation
seems to assume that you already know what is this stuff useful for
A tutorial would help to clarify stuff greatly.

Notice that there already valuable resources on this stuff.
Unfortunately, after several minutes of searching on www.python.org
I didn't find them (which may denote that something is wrong with me
or with python.org). Search for new style classes,
Shalab <last name which spelling I cannot remember> and may be
even Dave Kuhlman <where the spelling is probably wrong>.

Michele Simionato

 

[Michele Simionato]

<following myself>

just found this on my bookmarks:

http://www.python.org/cgi-bin/moinmoin/MetaClasses


M.S.

 

[Carlos Ribeiro]

Notice that there already valuable resources on this stuff.
Unfortunately, after several minutes of searching on www.python.org
I didn't find them (which may denote that something is wrong with me
or with python.org). Search for new style classes,
Shalab <last name which spelling I cannot remember> and may be
even Dave Kuhlman <where the spelling is probably wrong>.

I've read all the tutorials and resources pointed there from the Wiki
as a starting point. All do a good job (some better than others) at
explaining a lot of the technical issues. In fact, I've read both
articles you and David Mertz have written, and found them great. The
standard documentation is unfortunately up to this standard -- I've
tried to find comprehensive references for it inside Python's
reference manuals to no avail. This is something that really needs to
be done there.

But the problem, in my particular case, was not with metaclasses per
se, but with understanding the 'big picture'. With all respect for
you, David Mertz, Alex Martelli, and all the others who helped to
guide me through this (steep) learning curve -- you've done a great
job, and I have to thank you all -- but most of the documentation
seems to start 'from the inside'; in other words, from the perspective
of someone that thoroughly understands how Python works, specially
regarding the handling of declarations (classes, modules and
functions), and that also knows what metaclasses are good for. For
people that come from a different background (such as me), some of the
explanations fail to 'ring a bell', which usually only happens after a
lot of practical attempts and failures.

This very thread, including the last comments from myself and Alex
Martelli, is specially illuminating in this respect. It took me a
while to figure out the obvious -- that there are not 'real' class
declarations in Python, at least as far as they do exist in C++ or
Delphi (the languages that I worked with before Python). (btw, I
really don't know Java, but I feel confident to say that Java is
probably in the C++ camp here also).

Figuring out the obvious tend to be a big barrier to the learning
process. For experienced writers, it tends to be a problem also,
because it's so obvious that one feels compelled to leave it out of
the explanation. But that's the issue here -- the fact that what I'm
freely calling 'class declarations' in Python are not really
declarations (in the same sense of C++, for example), are just
executable code like anything else; and that only as a cleverly
designed side effect (*) a new class (or module) is built out from the
results of the code that was executed.

(*) I beg your pardon, that's just a figure of mine that shows how do
people with different mental models understand it when they finally
'get' it.

I have developed my own mental model on this, and it tells me that
classes in Python are declared in a two-step process: code execution,
and class creation. As I said, this may be obvious to Python insiders
(obvious to the point where it's natural and doesn't deserve any
special explanation), but I assume that it isn't, given:

a) my own trouble to understand the full implications of it;
b) the type of questions that pop up in this group;
c) the surprised look that even old-time pythoneers do when they see
code such as the 'def __metaclass__(...):' inside a class, or similar
hacks.

It may be the case that, as I work more with metaclasses, I will tend
towards a more natural vision of the process, and some of the nuances
that are now relevant to me may become a thing of the past. But I
think that I'm not alone on this, and that is the real problem faced
by many newcomers to Python.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Anna Martelli Ravenscroft]

I've read all the tutorials and resources pointed there from the Wiki
as a starting point. All do a good job (some better than others) at
explaining a lot of the technical issues. In fact, I've read both
articles you and David Mertz have written, and found them great. The
standard documentation is unfortunately up to this standard -- I've
tried to find comprehensive references for it inside Python's
reference manuals to no avail. This is something that really needs to
be done there.

But the problem, in my particular case, was not with metaclasses per
se, but with understanding the 'big picture'. With all respect for
you, David Mertz, Alex Martelli, and all the others who helped to
guide me through this (steep) learning curve -- you've done a great
job, and I have to thank you all -- but most of the documentation
seems to start 'from the inside'; in other words, from the perspective
of someone that thoroughly understands how Python works, specially
regarding the handling of declarations (classes, modules and
functions), and that also knows what metaclasses are good for. For
people that come from a different background (such as me), some of the
explanations fail to 'ring a bell', which usually only happens after a
lot of practical attempts and failures.

It may be the case that, as I work more with metaclasses, I will tend
towards a more natural vision of the process, and some of the nuances
that are now relevant to me may become a thing of the past. But I
think that I'm not alone on this, and that is the real problem faced
by many newcomers to Python.

It's even harder for a person coming to Python as a first programming
language. Most technical documentation (not just with Python but with,
for example, Linux) assumes that you already understand the topic at
hand, just need a clarification of how-to. More explanation about
"why-to" and "what does it mean" and underlying "how do I usefully think
about this" is really helpful (in carefully measured doses...

It's unfortunately common that people who understand the topic
thoroughly (in any field, not just computers) don't think about the
details consciously anymore, so it isn't that they *choose* not to
mention it - it simply never occurs to them to mention it, because they
"just know". This is a common issue in Training - the subject matter
experts often are too "deep" in their subject to be able to explain it
well to newbies.

I give a lot of credit to many of the folks in Python - at least they
care whether they're getting across to newcomers. Some do better than
others at explaining, but all of them seem to care. They *want* to know
when something isn't clear and *why* so they can *fix* it! (Unlike my
initial experience with some computer areas many years ago, where anyone
who couldn't figure it out on their own was accused of being too stoopid
to waste time on...)

Anna

 

[Alex Martelli]

But the problem, in my particular case, was not with metaclasses per
se, but with understanding the 'big picture'. With all respect for

Tim Peters is on record as saying (I paraphrase by memory...) that
metaclasses are NOT something 99% of Python users should worry about,
anyway. The Big Picture, on the other hand, IS going to be useful to
just about any user of Python, even a casual one.

regarding the handling of declarations (classes, modules and
functions), and that also knows what metaclasses are good for. For

Well, that one IS easy: since there are no declarations (of classes,
modules and functions), the handling of declarations is: none at all.
(How would one handle something that doesn't exist, after all?).

Regarding "what metaclasses are good for" (in Python), that's something
we've mostly had to work out the hard way over the last few years --
perhaps harder for those with some understanding of Smalltalk, because
Python metaclasses are so different from Smalltalk ones that one had to
"unlearn" the Smalltalk-originated ideas first. Just as somebody who
comes from languages where there are declarations must sort of unlearn
those idea, before they can really get the NON-declaration
sort-of-equivalents in Python...

This very thread, including the last comments from myself and Alex
Martelli, is specially illuminating in this respect. It took me a
while to figure out the obvious -- that there are not 'real' class
declarations in Python, at least as far as they do exist in C++ or
Delphi (the languages that I worked with before Python). (btw, I
really don't know Java, but I feel confident to say that Java is
probably in the C++ camp here also).

Yep, Java has declarations, not quite like C++ but very close, while
Python doesn't (unless you consider the 'global' statement to be a
"declaration" -- it _is_ in any case a _wart_...;-).

Figuring out the obvious tend to be a big barrier to the learning
process. For experienced writers, it tends to be a problem also,
because it's so obvious that one feels compelled to leave it out of
the explanation. But that's the issue here -- the fact that what I'm
freely calling 'class declarations' in Python are not really
declarations (in the same sense of C++, for example), are just

Well, _this_ experienced writer keeps repeating this fact over and over
and over again in _his_ explanations (e.g., Nutshell p. 32: "Unlike
other languages, Python has no declarations"), but of course that
doesn't help all that much if one's explanations just aren't read.

Some simple google search such as:
<http://groups.google.com/groups?as_oq=declaration declarations&safe=i
mages&ie=UTF-8&as_ugroup=comp.lang.python&as_uauthors=martelli&lr=&num=2
0&hl=en>
can further show you that I _ceaselessly_ keep repeating this...
*hundreds* of times...: "def is not a declaration: it's an executable
statement", "Python has no declarations", "For that, you'd need some
kind of declaration. Python does not have declarations", "Right, since
we have no declarations." -- over and over and OVER again. And there
are just snippets readable in the short google summaries of the FIRST
page out of 351 hits.

So, for once, it seems to me that, on this specific subject at least, I
should be held blameless: far from feeling compelled to leave it out, I
am _obsessive_ about pointing it out again and again and AGAIN. I don't
think I can do much more to get the point across that "*Python has no
declarations, only executable statements*", except maybe personally seek
out every poster who speaks of "declarations in Python", grab them by
their lapels and shout the truth out in their faces. Sorry, I get too
many frequent flier miles as it, without adding all of these side trips
too. If (at least some) experienced writers are doing the best they
possibly can to get this crucial point across, and some readers still
just can't get it no matter what, perhaps it's not all that useful to
blame the writers -- and the READERS should perhaps take SOME
responsibility for listening to the unceasing repetitions in
question...?!

I have developed my own mental model on this, and it tells me that
classes in Python are declared in a two-step process: code execution,
and class creation. As I said, this may be obvious to Python insiders
(obvious to the point where it's natural and doesn't deserve any
special explanation), but I assume that it isn't, given:

It's a good model -- execution of the code in the class body, then usage
of the dictionary thus built for the purpose of creating the class
object; pretty close to how Python actually does it. I'm sorry you had
to develop your own mental model of this, since it IS just what I try to
present on p. 100-101 of the Nutshell (and sundry tutorials &c for which
you can find PDF files at www.strakt.com). "To execute a class
statement, Python first collects the base classes into a tuple t (an
empty one, if there are no base classes) and executes the class body in
a temporary dictionary d. Then, Python determines the metaclass M to
use" etc, etc.

a) my own trouble to understand the full implications of it;
b) the type of questions that pop up in this group;
c) the surprised look that even old-time pythoneers do when they see
code such as the 'def __metaclass__(...):' inside a class, or similar
hacks.

Maybe Tim Peters is right that 99% of Python users need not worry about
metaclasses? That applies to 99% of old-time pythoneers, too -- it's an
issue of whether you _need_ it, not of whether you could learn about it
if it WAS something you needed. Many Pythonistas are pragmatic guys and
quite understandably don't want to spend any time or energy learning
about stuff they don't need nor care about.

As for me, I'm leaning towards agreement with Guido, that a metaclass
that's not a type _is_ a hack that's best avoided. So, a 'def
__metaclass__' is already a code smell. A nested 'class
__metaclass__(type):' would of course be a completely different issue,
basically an "anonymous metaclass" so to speak, not particularly hackish
IMHO -- for the <1% (if Tim has it right) of Pythonistas who _DO_ need
to worry about metaclasses, of course.


Alex

 

[Anna Martelli Ravenscroft]

Well, _this_ experienced writer keeps repeating this fact over and over
and over again in _his_ explanations (e.g., Nutshell p. 32: "Unlike
other languages, Python has no declarations"), but of course that
doesn't help all that much if one's explanations just aren't read.

Some simple google search such as:
<http://groups.google.com/groups?as_oq=declaration declarations&safe=i
mages&ie=UTF-8&as_ugroup=comp.lang.python&as_uauthors=martelli&lr=&num=2
0&hl=en>
can further show you that I _ceaselessly_ keep repeating this...
*hundreds* of times...: "def is not a declaration: it's an executable
statement", "Python has no declarations", "For that, you'd need some
kind of declaration. Python does not have declarations", "Right, since
we have no declarations." -- over and over and OVER again. And there
are just snippets readable in the short google summaries of the FIRST
page out of 351 hits.

So, for once, it seems to me that, on this specific subject at least, I
should be held blameless: far from feeling compelled to leave it out, I
am _obsessive_ about pointing it out again and again and AGAIN. I don't
think I can do much more to get the point across that "*Python has no
declarations, only executable statements*", except maybe personally seek
out every poster who speaks of "declarations in Python", grab them by
their lapels and shout the truth out in their faces. Sorry, I get too
many frequent flier miles as it, without adding all of these side trips
too. If (at least some) experienced writers are doing the best they
possibly can to get this crucial point across, and some readers still
just can't get it no matter what, perhaps it's not all that useful to
blame the writers -- and the READERS should perhaps take SOME
responsibility for listening to the unceasing repetitions in
question...?!

Being completely unbiased, of course, I will say that you do a damn fine
job of mostly avoiding the "expert's error" of leaving out the "obvious"
stuff.

Anna

 

[Carlos Ribeiro]

Well, _this_ experienced writer keeps repeating this fact over and over
and over again in _his_ explanations (e.g., Nutshell p. 32: "Unlike
other languages, Python has no declarations"), but of course that
doesn't help all that much if one's explanations just aren't read.

I can only thank you for having pointed it out to me here at this
newsgroup. I'm not fortunate enough to have a copy your books here
with me But even so -- this topic _should_ be better explained,
and clarified, in the standard Python documentation. (as a side note,
it's not only declarations and metaclasses; I've read Raymond
Hettinger docs regarding descriptors, and I was left wondering why
weren't it included into the standard documentation, in some form or
other).

As for me, I'm leaning towards agreement with Guido, that a metaclass
that's not a type _is_ a hack that's best avoided. So, a 'def
__metaclass__' is already a code smell. A nested 'class
__metaclass__(type):' would of course be a completely different issue,
basically an "anonymous metaclass" so to speak, not particularly hackish
IMHO -- for the <1% (if Tim has it right) of Pythonistas who _DO_ need
to worry about metaclasses, of course.

During the learning process, more often than not I would try to solve
*all* problems with a metaclass, as if it was the only tool available.
Now that I'm starting to feel comfortable, things start to fall back
into the correct perspective once again. But of course, now I'm better
equiped with a powerful addition to my toolset Let's not abuse it
though.


--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Ville Vainio]

Carlos> During the learning process, more often than not I would
Carlos> try to solve *all* problems with a metaclass, as if it was
Carlos> the only tool available. Now that I'm starting to feel
Carlos> comfortable, things start to fall back into the correct
Carlos> perspective once again. But of course, now I'm better
Carlos> equiped with a powerful addition to my toolset Let's
Carlos> not abuse it though.

Good idea. If I was to review a piece of code that introduced a
metaclass (as opposed to just using one that came with a framework), I
would be prone to ask the author whether he really exhausted *all* the
other avenues before going the metaclass route. Not documenting
metaclasses in the standard, bundled documentation is a good thing.

 

[Anna Martelli Ravenscroft]

I can only thank you for having pointed it out to me here at this
newsgroup. I'm not fortunate enough to have a copy your books here
with me But even so -- this topic _should_ be better explained,
and clarified, in the standard Python documentation.

There's much about the "standard Python documentation" that sucks rocks.
OTOH - it doesn't suck as badly as most techie documentation, and
besides, if it didn't occasionally need clarification, we wouldn't need
to come posting here asking for help and meeting such wonderful people -
so I can't really complain. Particularly considering that asking him to
help me out on a (to me) obscure Python point (list.append returns None)
was how I reconnected with this Italian guy I knew from the internet,
which ended up with us getting married this summer... So, I REALLY can't
complain too loudly... ;-)

(as a side note,
it's not only declarations and metaclasses; I've read Raymond
Hettinger docs regarding descriptors, and I was left wondering why
weren't it included into the standard documentation, in some form or
other).

A lot of these things are pretty, erm, high-level issues - beginners
don't need them, so it makes some sort of sense that they not be part of
the standard "everybody needs it" documentation. After all, decorators
are pretty new - afaik.

Anna

 

[Carlos Ribeiro]

Carlos> During the learning process, more often than not I would
Carlos> try to solve *all* problems with a metaclass, as if it was
Carlos> the only tool available. Now that I'm starting to feel
Carlos> comfortable, things start to fall back into the correct
Carlos> perspective once again. But of course, now I'm better
Carlos> equiped with a powerful addition to my toolset Let's
Carlos> not abuse it though.

Good idea. If I was to review a piece of code that introduced a
metaclass (as opposed to just using one that came with a framework), I
would be prone to ask the author whether he really exhausted *all* the
other avenues before going the metaclass route. Not documenting
metaclasses in the standard, bundled documentation is a good thing.

While we agree on the use of metaclasses, I hav to strongly disagree
with your opinion regarding documentation. Advanced stuff doesn't need
to be explained in the tutorials, of course, but the reference manual
should contain more information -- after al, it's *the* reference
manual, isn't it?

(btw some people consider the source code to be the authoritative
source of information. I don't agree, I believe that source code,
independent of how well crafted and beautiful, is not a substitute for
good documentation. Even the best code sometimes fails to give
important metainformation, such as the context, history or intentions
of the programmer. Literate programming is a good way to get the two
concepts -- source code and docs -- together, though.)

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)