Re-executing the code object from a class 'declaration'

Discussion in 'Python' started by Carlos Ribeiro, Oct 6, 2004.

  1. As part of a pet project of mine, I needed to find a way to re-execute
    a class declaration[1]. The reason behind it was simple: the class
    declaration in question had some external dependencies that could
    potentially change during the lifetime of the program. I needed to be
    able to execute the class 'declaration' block again to produce new
    classes (in opposition to new instances, which would not solve my
    problem)[2].

    I solved the problem using inspect to find the current code object
    *inside* the class declaration, and stored it an attribute of the
    class itself. Then, a method of the class can be used to re-create it.
    A metaclass-enabled version is shown below:

    class Meta(type):
    def __new__(klass, name, bases, dct):
    dct['my_code'] = inspect.currentframe(1).f_code
    return type(name, bases, dct)

    class MyClass:
    __metaclass__ = Meta
    creation_time = time.ctime()
    print "I'm inside my class definition!"
    def re_execute(klass):
    exec klass.my_code
    return MyClass
    re_execute = classmethod(re_execute)

    Testing:

    >>> class MyClass:

    ...
    I'm inside my class definition!
    >>> id(MyClass)

    8737376
    >>> MyClass.creation_time

    'Tue Oct 05 21:50:25 2004'
    >>> NewClass = MyClass.re_execute()

    I'm inside my class definition!
    >>> id(NewClass)

    8736912
    >>> NewClass.creation_time

    'Tue Oct 05 21:51:20 2004'

    Implementation notes:

    1) The exec statement actually binds the name to the same original
    name, but in the current local scope. That's why it does a 'return
    MyClass' in the re_execute() method. The MyClass symbol refers to the
    local symbol with this name that was just re-created by the exec
    method.

    2) If you call the exec from the same scope where the class was
    originally created, then it is automatically bound to the same name:

    >>> id(MyClass)

    8737376
    >>> MyClass.creation_time

    'Tue Oct 05 21:50:25 2004'
    >>> exec MyClass.my_code

    I'm inside my class definition!
    >>> id(MyClass)

    8744512
    >>> MyClass.creation_time

    'Tue Oct 05 21:56:53 2004'

    3) I thought that the same technique could be made to work without a
    metaclass, but I couldn't make it work. I haven't invested much time
    to understand why -- it just works as implemented above. (It seems
    that it does not bind correctly to the original name as it does if you
    use the metaclass)

    ----
    [1] After a long and interesting discussion for which I have to thank
    Alex Martelli and several other members of c.l.py, I've come to
    realize that there are no real class declarations in Python. But for
    all effects, it's just easier to call them this way, even if the
    underlying mechanism is different from the one used on other
    traditional languages.

    [2] Think about it as a templating mechanism, similar to the C++ one,
    but for Python; this mechanism generates new classes that can be
    instantiated as many times as needed. It's not the same thing as to
    customize a instance during its initialization.

    --
    Carlos Ribeiro
    Consultoria em Projetos
    blog: http://rascunhosrotos.blogspot.com
    blog: http://pythonnotes.blogspot.com
    mail:
    mail:
     
    Carlos Ribeiro, Oct 6, 2004
    #1
    1. Advertising

  2. Michele Simionato, Oct 6, 2004
    #2
    1. Advertising

  3. Carlos Ribeiro <> wrote:

    > As part of a pet project of mine, I needed to find a way to re-execute
    > a class declaration[1]. The reason behind it was simple: the class


    If you start calling it 'class statement' I may read and perhaps help
    with the rest of your post, but since I consider calling 'declaration'
    something that is no more and no less than an executable statement an
    excellent way to try and confuse the issues (by leading many readers to
    expect something completely different from the reality), I'm going to
    personally boycott any post deliberately perpetrating such wilful use of
    language to confuse readers, and I urge the same course of action on
    anybody who cares about clarity, understanding and communication.


    Alex
     
    Alex Martelli, Oct 6, 2004
    #3
  4. Carlos Ribeiro

    Steve Holden Guest

    Alex Martelli wrote:

    > Carlos Ribeiro <> wrote:
    >
    >
    >>As part of a pet project of mine, I needed to find a way to re-execute
    >>a class declaration[1]. The reason behind it was simple: the class

    >
    >
    > If you start calling it 'class statement' I may read and perhaps help
    > with the rest of your post, but since I consider calling 'declaration'
    > something that is no more and no less than an executable statement an
    > excellent way to try and confuse the issues (by leading many readers to
    > expect something completely different from the reality), I'm going to
    > personally boycott any post deliberately perpetrating such wilful use of
    > language to confuse readers, and I urge the same course of action on
    > anybody who cares about clarity, understanding and communication.
    >
    >
    > Alex


    How would you feel about "class definition"?

    regards
    Steve
    --
    http://www.holdenweb.com
    http://pydish.holdenweb.com
    Holden Web LLC +1 800 494 3119
     
    Steve Holden, Oct 6, 2004
    #4
  5. Steve Holden <> wrote:
    ...
    > >>As part of a pet project of mine, I needed to find a way to re-execute
    > >>a class declaration[1]. The reason behind it was simple: the class

    ...
    > > If you start calling it 'class statement' I may read and perhaps help
    > > with the rest of your post, but since I consider calling 'declaration'

    ...
    > How would you feel about "class definition"?


    Since we have a 'def' statement it seems confusing to me to talk of
    'definition' about something else than a 'def' (even there I prefer "a
    def statement" as clearer and more direct, but I guess I'll have to
    accept "a function definition" given ``def''... I do wish that keyword
    was 'function' rather than 'def', but, oh well, too late for that;-).


    Alex
     
    Alex Martelli, Oct 6, 2004
    #5
  6. In respectful consideration to Alex's opposition to the misuse of the
    word 'declaration', here is a repost with the suggested changes:
    ---
    As part of a pet project of mine, I needed to find a way to re-execute
    a class statement. The reason behind it was simple: the class
    statement in question had some external dependencies that could
    potentially change during the lifetime of the program. I needed to be
    able to execute the class statement block again to produce new
    classes (in opposition to new instances, which would not solve my
    problem)[1].

    I solved the problem using inspect to find the current code object
    *inside* the class statement, and stored it in an attribute of the
    class itself. Then, a method of the class can be used to re-create it.
    A metaclass-enabled version is shown below:

    class Meta(type):
    def __new__(klass, name, bases, dct):
    dct['my_code'] = inspect.currentframe(1).f_code
    return type(name, bases, dct)

    class MyClass:
    __metaclass__ = Meta
    creation_time = time.ctime()
    print "I'm inside my class definition!"
    def re_execute(klass):
    exec klass.my_code
    return MyClass
    re_execute = classmethod(re_execute)

    Testing:

    >>> class MyClass:

    ...
    I'm inside my class definition!
    >>> id(MyClass)

    8737376
    >>> MyClass.creation_time

    'Tue Oct 05 21:50:25 2004'
    >>> NewClass = MyClass.re_execute()

    I'm inside my class definition!
    >>> id(NewClass)

    8736912
    >>> NewClass.creation_time

    'Tue Oct 05 21:51:20 2004'

    Implementation notes:

    1) The exec statement actually binds the name to the same original
    name, but in the current local scope. That's why it does a 'return
    MyClass' in the re_execute() method. The MyClass symbol refers to the
    local symbol with this name that was just re-created by the exec
    method.

    2) If you call the exec from the same scope where the class was
    originally created, then it is automatically bound to the same name:

    >>> id(MyClass)

    8737376
    >>> MyClass.creation_time

    'Tue Oct 05 21:50:25 2004'
    >>> exec MyClass.my_code

    I'm inside my class definition!
    >>> id(MyClass)

    8744512
    >>> MyClass.creation_time

    'Tue Oct 05 21:56:53 2004'

    3) I thought that the same technique could be made to work without a
    metaclass, but I couldn't make it work. I haven't invested much time
    to understand why -- it just works as implemented above. (It seems
    that it does not bind correctly to the original name as it does if you
    use the metaclass)

    ----
    [1] Think about it as a templating mechanism, similar to the C++ one,
    but for Python; this mechanism generates new classes that can be
    instantiated as many times as needed. It's not the same thing as to
    customize a instance during its initialization.

    --
    Carlos Ribeiro
    Consultoria em Projetos
    blog: http://rascunhosrotos.blogspot.com
    blog: http://pythonnotes.blogspot.com
    mail:
    mail:
     
    Carlos Ribeiro, Oct 6, 2004
    #6
  7. Carlos Ribeiro <> wrote:
    ...
    > As part of a pet project of mine, I needed to find a way to re-execute
    > a class statement. The reason behind it was simple: the class
    > statement in question had some external dependencies that could
    > potentially change during the lifetime of the program. I needed to be
    > able to execute the class statement block again to produce new
    > classes (in opposition to new instances, which would not solve my
    > problem)[1].


    Interesting problem.


    > I solved the problem using inspect to find the current code object
    > *inside* the class statement, and stored it in an attribute of the
    > class itself. Then, a method of the class can be used to re-create it.
    > A metaclass-enabled version is shown below:
    >
    > class Meta(type):
    > def __new__(klass, name, bases, dct):
    > dct['my_code'] = inspect.currentframe(1).f_code
    > return type(name, bases, dct)
    >
    > class MyClass:
    > __metaclass__ = Meta
    > creation_time = time.ctime()
    > print "I'm inside my class definition!"
    > def re_execute(klass):
    > exec klass.my_code
    > return MyClass
    > re_execute = classmethod(re_execute)


    Urk. Ahem, I mean, interesting solution. I _think_ it poses certain,
    ahem, problems, such as infinite recursion if a call to re_execute is in
    the same code object as that where the 'class MyClass' is, for example
    -- because basically you ARE re-executing the whole kaboodle. You're
    not just re-executing the 'class' statement, but everything that comes
    before or after it in the same containing codeobject. Your tests in an
    interactive interpreter are tricky that way, because in that special
    environment a code object is a tiny little thing. But stick this into a
    module and place a dis.dis(dct['my_code']) in the metaclass's __new__
    and you'll see that basically MyClass.re_execute isn't very different
    from a reload of the module. Isn't that a _bit_ too much in general?

    The code object for the body of the class can also be found (with some
    work): it will be among the co_consts of what you've stored as
    dct['my_code'], it will be an instance of types.CodeType, and its
    co_name will be the same as the 'name' parameter to Meta.__new__. If
    you have more than one code object with those characteristics, e.g. two
    classes with the same name at the same module's toplevel, you do have
    some trouble. But otherwise, you can build a function, with
    new.function(thatclassbodycodeobject, theglobalsfrominspect), which,
    when called (w/o arguments), reexecutes the classbody and returns the
    same kind of dictionary that Meta.__new__ is receiving as its 'dct'
    argument. Wouldn't this be preferable to re-executing the whole module
    body, perhaps?

    > Implementation notes:
    >
    > 1) The exec statement actually binds the name to the same original
    > name, but in the current local scope. That's why it does a 'return
    > MyClass' in the re_execute() method. The MyClass symbol refers to the
    > local symbol with this name that was just re-created by the exec
    > method.


    This also fails if you have multiple uses of name MyClass at toplevel of
    the same module -- it actually fails more often than my suggestions
    above, e.g if your module is:

    class MyClass:
    ... etc ...
    FakeName = MyClass
    MyClass = 23

    then your FakeName.re_execute() will return 23, while the approach I
    suggest will be able to recover in this case (the codeobject in the
    co_consts of the module's code object will still be the only one in that
    container that's named 'MyClass').


    > 2) If you call the exec from the same scope where the class was
    > originally created, then it is automatically bound to the same name:


    Same issue as above -- you're getting closer and closer to a module's
    reload (save in the anomalous case where you type the whole class
    statement at an interactive prompt;-). Note that it IS quite possible
    that what you want to do IS in fact to reload the module, period, but,
    if so, then the reload statement might be handier.

    > 3) I thought that the same technique could be made to work without a
    > metaclass, but I couldn't make it work. I haven't invested much time
    > to understand why -- it just works as implemented above. (It seems
    > that it does not bind correctly to the original name as it does if you
    > use the metaclass)


    It appears to me, but I haven't tested this, that putting the inspect
    introspection in the classbody itself should be a sounder technique, as
    it should be able to find itself and stash itself away more reliably
    than just based on name (which is the best the metaclass can do). The
    name binding issue is separate, however.

    > [1] Think about it as a templating mechanism, similar to the C++ one,
    > but for Python; this mechanism generates new classes that can be
    > instantiated as many times as needed. It's not the same thing as to
    > customize a instance during its initialization.


    I agree, there's one metalevel of difference between instantiating a
    class to make an instance, and instantiating a metaclass to make a
    class.


    Alex
     
    Alex Martelli, Oct 6, 2004
    #7
  8. On Wed, 6 Oct 2004 14:06:23 +0200, Alex Martelli <> wrote:
    > Urk. Ahem, I mean, interesting solution. I _think_ it poses certain,
    > ahem, problems, such as infinite recursion if a call to re_execute is in
    > the same code object as that where the 'class MyClass' is, for example
    > -- because basically you ARE re-executing the whole kaboodle. You're
    > not just re-executing the 'class' statement, but everything that comes
    > before or after it in the same containing codeobject. Your tests in an
    > interactive interpreter are tricky that way, because in that special
    > environment a code object is a tiny little thing. But stick this into a
    > module and place a dis.dis(dct['my_code']) in the metaclass's __new__
    > and you'll see that basically MyClass.re_execute isn't very different
    > from a reload of the module. Isn't that a _bit_ too much in general?


    (repeating to myself: dis is your friend. dis is your friend).

    Well, it seems that I'm running out of options. I've checked dis, and
    done a few more tests. I found a way to find the correct code object
    for the _body_ of the class statement (or at least I think so), but
    this does not solve the problem, as it will be shown below:

    -------------
    import inspect
    import dis
    import time
    import types

    class Meta(type):
    def __new__(klass, name, bases, dct):
    caller = inspect.stack(0)[1]
    caller_code = caller[0].f_code
    caller_filename = caller[1]
    caller_lineno = caller[2]
    print "Caller info: ", caller_filename, caller_lineno, caller_code
    print dis.dis(caller[0].f_code)
    for code in caller_code.co_consts:
    if isinstance(code, types.CodeType):
    if (code.co_filename == caller_filename) and \
    (code.co_firstlineno == caller_lineno):
    print "found!", code.co_name
    dct['my_code'] = code
    return type(name, bases, dct)

    class MyClass(object):
    __metaclass__ = Meta
    creation_time = time.ctime()
    print "I'm inside my class definition!"
    def re_execute(klass):
    exec klass.my_code
    return MyClass
    re_execute = classmethod(re_execute)
    -------------

    The code checks the filename and the line number to find the correct
    code object. It could also check the class name, just to make sure
    that everything matches. It isn't needed, as far as I know. But the
    problem is that the code block that is found refers to the class
    _body_ only. Checking dis.dis() helps to understand it:

    27 64 LOAD_CONST 4 ('MyClass')
    67 BUILD_TUPLE 0
    70 LOAD_CONST 5 (<code object MyClass at
    01209E60, file "c:\work\help-on-c-l-py\re_exec_class.py", line 27>)
    73 MAKE_FUNCTION 0
    76 CALL_FUNCTION 0
    79 BUILD_CLASS
    80 STORE_NAME 7 (MyClass)

    There is not a single code block that does all the tasks related to
    the class statement, and only it. In fact, there are two different
    code objects to choose from, and neither really solves the problem.
    One is the module object; as you've pointed out, it's too generic and
    executes too much stuff, not just the class statement. The other one
    is the <code object MyClass at 01209E60> referred to in the dis()
    output above, that is the _body_ of the class statement. Calling it
    doesn't call the metaclass, and neither does the name binding 'magic':

    >>> exec MyClass.my_code

    I'm inside my class definition!
    # it does not print any of the debug messages in the metaclass, and
    it's obvious why

    It also doesn't take into account that the class may have base
    classes. For example, changing the statement to 'class
    MyClass(object)', the compiled code looks like this:

    27 64 LOAD_CONST 4 ('MyClass')
    67 LOAD_NAME 7 (object)
    70 BUILD_TUPLE 1

    >From this point, few options are left. What I need to execute is in

    fact a part of the module code object -- just one line of it, that's
    the code that wraps the entire class statement. Besides giving up
    (which is a reasonable option, given the constraints), what is left
    are some horrible hacks. (I'm don't feel like discussing it here; it's
    like Mordor Black Speech, if shall not be uttered in a public forum
    :).

    BTW, the two code objects represent exactly the two alternative
    solutions that can be done using only pure Python code:

    1) reload the entire module (which is nearly equivalent to calling the
    module code object, with the difference that it doesn't reload and
    recompile the actual source code)

    2) enclose the class statement inside a def, and run the def to obtain
    the class template:

    def BuildTemplate(<template parameters>):
    class MyTemplate(...):
    ...
    return MyTemplate

    For my application, I'm tending towards (2), because it allows for a
    better modularization of the code. But I'm sorry, in a way, because it
    breaks the expect semantics in a way -- it's a function that returns a
    class. It's valid, but some may regard it as 'ugly' and 'unpythonic'.

    > > 3) I thought that the same technique could be made to work without a
    > > metaclass, but I couldn't make it work. I haven't invested much time
    > > to understand why -- it just works as implemented above. (It seems
    > > that it does not bind correctly to the original name as it does if you
    > > use the metaclass)

    >
    > It appears to me, but I haven't tested this, that putting the inspect
    > introspection in the classbody itself should be a sounder technique, as
    > it should be able to find itself and stash itself away more reliably
    > than just based on name (which is the best the metaclass can do). The
    > name binding issue is separate, however.


    I've tried it yesterday, and it won't work, for some still unknown
    reason. I could find the code object, but when I ran it, it would not
    bind to the name in the same way as the metaclass would do. But it
    doesn't matter at this point.

    > > [1] Think about it as a templating mechanism, similar to the C++ one,
    > > but for Python; this mechanism generates new classes that can be
    > > instantiated as many times as needed. It's not the same thing as to
    > > customize a instance during its initialization.

    >
    > I agree, there's one metalevel of difference between instantiating a
    > class to make an instance, and instantiating a metaclass to make a
    > class.



    --
    Carlos Ribeiro
    Consultoria em Projetos
    blog: http://rascunhosrotos.blogspot.com
    blog: http://pythonnotes.blogspot.com
    mail:
    mail:
     
    Carlos Ribeiro, Oct 6, 2004
    #8
  9. On 5 Oct 2004 23:33:01 -0700, Michele Simionato
    <> wrote:
    > which does something close to what you ask for. Of course the trick
    > there is NOT recommended for real code ;)


    That was the best advice I could have :) Really. I'm trying to outwit
    Python, and one has to have the good sense to stop doing it at some
    point. I'm giving up (unless I found a really great way to make it),
    and I'll use this idiom instead:

    def BuildTemplate(<template parameters>)
    class TemplateClass(...):
    ...
    return TemplateClass

    It's simple, and less hackish, but still unusual. I fear I may be
    missing something on my design, if I need to resort to such hacks.
    --
    Carlos Ribeiro
    Consultoria em Projetos
    blog: http://rascunhosrotos.blogspot.com
    blog: http://pythonnotes.blogspot.com
    mail:
    mail:
     
    Carlos Ribeiro, Oct 6, 2004
    #9
  10. Carlos Ribeiro <> wrote:
    ...
    > > module and place a dis.dis(dct['my_code']) in the metaclass's __new__
    > > and you'll see that basically MyClass.re_execute isn't very different
    > > from a reload of the module. Isn't that a _bit_ too much in general?

    >
    > (repeating to myself: dis is your friend. dis is your friend).


    If you're hacking at this level, it sure is;-).

    > done a few more tests. I found a way to find the correct code object
    > for the _body_ of the class statement (or at least I think so), but


    Good -- as I said I think it can be even more reliably found from inside
    the classbody itself, but, no big deal.

    > exec klass.my_code


    Ah, this is your problem -- that's not what you want to do now.

    > problem is that the code block that is found refers to the class
    > _body_ only. Checking dis.dis() helps to understand it:


    Exactly: it's what builds the dict you want to pass to the metaclass
    (once you wrap it into a new.function, as I mentioned).

    > There is not a single code block that does all the tasks related to
    > the class statement, and only it. In fact, there are two different
    > code objects to choose from, and neither really solves the problem.


    Not by itself, but add some code and you'll be fine.

    1. You need to build an f = new.function(klass.my_code, globals()) in
    your classmethod that now erroneously does the exec above quoted (you
    could in fact do this in the metaclass, save the new function rather
    than the code object -- get the right globals via inspect).

    2. you need to obtain the needed dict, that's just d = f() in your
    method

    3. you can now call your metaclass with the appropriate bases and name:
    klas = self.__class__ # or cls if this is a classmethod;-)
    return type(klas)(klas.__name__, klas.__bases__, d)

    voila, you're there. This is all coded to minimize the need to have
    this method located in a specific class -- as I said I think you
    probably want to have this stuff in the metaclass (if you have a
    metaclass at all), but, whatever.


    This is basically just mimicking what Python itself does (check the
    dis.dis for a class statement as a starter): get the class body's code
    object, build a function from it, call the function (it returns its
    locals() as the result -- that 'return' is of course in the class body's
    code object, and dis.dis shows that...), and that's how you get the
    classdict on the stack. Add name and bases, call the metaclass, and
    Bob's your uncle...


    Alex
     
    Alex Martelli, Oct 6, 2004
    #10
  11. Carlos Ribeiro <> wrote:

    > On 5 Oct 2004 23:33:01 -0700, Michele Simionato
    > <> wrote:
    > > which does something close to what you ask for. Of course the trick
    > > there is NOT recommended for real code ;)

    >
    > That was the best advice I could have :) Really. I'm trying to outwit
    > Python, and one has to have the good sense to stop doing it at some
    > point. I'm giving up (unless I found a really great way to make it),
    > and I'll use this idiom instead:
    >
    > def BuildTemplate(<template parameters>)
    > class TemplateClass(...):
    > ...
    > return TemplateClass
    >
    > It's simple, and less hackish, but still unusual. I fear I may be
    > missing something on my design, if I need to resort to such hacks.


    Personally, I _like_ using 'class' statements to make classes and 'def'
    statements to make functions; I don't see either as hacks. Messing with
    bytecodes, building a function with new.function around a certain code
    object, calling the metaclass explicitly... that feels more hackish to
    me, personally. I do it when I must, but 99.9% of the time I have no
    problem avoiding it in production code, it seems to me.


    Alex
     
    Alex Martelli, Oct 6, 2004
    #11
  12. On Wed, 6 Oct 2004 18:49:58 +0200, Alex Martelli <> wrote:
    > Carlos Ribeiro <> wrote:
    > ...
    > > > module and place a dis.dis(dct['my_code']) in the metaclass's __new__
    > > > and you'll see that basically MyClass.re_execute isn't very different
    > > > from a reload of the module. Isn't that a _bit_ too much in general?

    > >
    > > (repeating to myself: dis is your friend. dis is your friend).

    >
    > If you're hacking at this level, it sure is;-).
    >
    > > done a few more tests. I found a way to find the correct code object
    > > for the _body_ of the class statement (or at least I think so), but

    >
    > Good -- as I said I think it can be even more reliably found from inside
    > the classbody itself, but, no big deal.
    >
    > > exec klass.my_code

    >
    > Ah, this is your problem -- that's not what you want to do now.
    >
    > > problem is that the code block that is found refers to the class
    > > _body_ only. Checking dis.dis() helps to understand it:

    >
    > Exactly: it's what builds the dict you want to pass to the metaclass
    > (once you wrap it into a new.function, as I mentioned).
    >
    > > There is not a single code block that does all the tasks related to
    > > the class statement, and only it. In fact, there are two different
    > > code objects to choose from, and neither really solves the problem.

    >
    > Not by itself, but add some code and you'll be fine.
    >
    > 1. You need to build an f = new.function(klass.my_code, globals()) in
    > your classmethod that now erroneously does the exec above quoted (you
    > could in fact do this in the metaclass, save the new function rather
    > than the code object -- get the right globals via inspect).


    That's the part that I was missing. It looks less 'hackish' than I had
    expect. I was really afraid that I would need to manage bytecode-level
    hacks myself. I am not as concerned about getting the right globals,
    though -- it's good enough if I'm able to supply whatever globals I
    need upon calling, I think (unless I'm _really_ missing something).

    > 2. you need to obtain the needed dict, that's just d = f() in your
    > method


    There is still a question, but I think I already know the anwswer :)
    If the class *has* a base class, it has to be passed as a parameter. I
    assume that i can simply call it like this: d=f(bases); or instead,
    that I can fill the argdefs in the new.function call. I'll try it
    later.

    > 3. you can now call your metaclass with the appropriate bases and name:
    > klas = self.__class__ # or cls if this is a classmethod;-)
    > return type(klas)(klas.__name__, klas.__bases__, d)
    >
    > voila, you're there. This is all coded to minimize the need to have
    > this method located in a specific class -- as I said I think you
    > probably want to have this stuff in the metaclass (if you have a
    > metaclass at all), but, whatever.


    Tehre are two reasons to do it in a metaclass: first, I tested it and
    it works (of sorts), while for some reason it wont work for a
    non-metaclasses enabled class. Also, because in this scenario there
    are other things that I already need to check, and a metaclass does it
    nicely.

    > This is basically just mimicking what Python itself does (check the
    > dis.dis for a class statement as a starter): get the class body's code
    > object, build a function from it, call the function (it returns its
    > locals() as the result -- that 'return' is of course in the class body's
    > code object, and dis.dis shows that...), and that's how you get the
    > classdict on the stack. Add name and bases, call the metaclass, and
    > Bob's your uncle...


    For now, I think I'll just be using the class-inside-a-def idiom. But
    I'm still working around some design issues, and if the templating
    mechanism gets in the final production code (it's only one of a few
    competing designs), I'll surely try it again.

    --
    Carlos Ribeiro
    Consultoria em Projetos
    blog: http://rascunhosrotos.blogspot.com
    blog: http://pythonnotes.blogspot.com
    mail:
    mail:
     
    Carlos Ribeiro, Oct 6, 2004
    #12
  13. On Wed, 6 Oct 2004 18:58:40 +0200, Alex Martelli <> wrote:
    > Carlos Ribeiro <> wrote:
    > > def BuildTemplate(<template parameters>)
    > > class TemplateClass(...):
    > > ...
    > > return TemplateClass
    > >
    > > It's simple, and less hackish, but still unusual. I fear I may be
    > > missing something on my design, if I need to resort to such hacks.

    >
    > Personally, I _like_ using 'class' statements to make classes and 'def'
    > statements to make functions; I don't see either as hacks. Messing with
    > bytecodes, building a function with new.function around a certain code
    > object, calling the metaclass explicitly... that feels more hackish to
    > me, personally. I do it when I must, but 99.9% of the time I have no
    > problem avoiding it in production code, it seems to me.


    I'm not sure if I understood what you meant. My common sense tells me
    that the class-inside-a-def is more acceptable, given all the black
    magic that we had to go through to be able to re-execute the class
    statement. But the function above really returns a class... which is
    'wrong', but many counts (although nothing stops it from happening,
    after all, classes are objects pretty much like any other object,
    albeit with the special ability to build new objects instances).

    On the other hand, the re-execute-class-statement trick is a real hack
    -- specially when seen from the inside. However, if everything is
    clearly hidden in a metaclass (and I mean, a *real* metaclass, not a
    def __metaclass__()); better yet, if the metaclass itself is hidden in
    a base class; what do you think is cleaner?

    a) class-inside-a-def

    def BuildTemplate(<template parameters>)
    class TemplateClass(...):
    ...
    return TemplateClass
    T = BuildTemplate(<params>)

    b) re-exec-class

    # GenericTemplate is a metaclass-enabled ancestor
    from template import GenericTemplate
    class TemplateClass(GenericTemplate):
    ...

    T = TemplateClass.NewTemplate(<params>)

    c) neither of the above -- there must be a better way to do it.

    --
    Carlos Ribeiro
    Consultoria em Projetos
    blog: http://rascunhosrotos.blogspot.com
    blog: http://pythonnotes.blogspot.com
    mail:
    mail:
     
    Carlos Ribeiro, Oct 6, 2004
    #13
  14. Carlos Ribeiro <> wrote:
    ...
    > > Personally, I _like_ using 'class' statements to make classes and 'def'
    > > statements to make functions; I don't see either as hacks. Messing with

    ...
    > I'm not sure if I understood what you meant. My common sense tells me
    > that the class-inside-a-def is more acceptable, given all the black
    > magic that we had to go through to be able to re-execute the class


    Yep, sounds about right.

    > statement. But the function above really returns a class... which is
    > 'wrong', but many counts (although nothing stops it from happening,


    Ah, that's our disagreement, I guess. I consider Factory a perfectly
    sensible and respectable design pattern, and thus I see nothing wrong a
    function making and returning a new object (class or otherwise).

    > after all, classes are objects pretty much like any other object,
    > albeit with the special ability to build new objects instances).


    Any callable can build whatever kind of objects it wants -- class
    objects have no "special ability" in this regard. Where class objects
    are unique is in the automatic chain of delegation -- instance to class
    to bases -- when an attribute is being looked up.


    > On the other hand, the re-execute-class-statement trick is a real hack
    > -- specially when seen from the inside. However, if everything is
    > clearly hidden in a metaclass (and I mean, a *real* metaclass, not a
    > def __metaclass__()); better yet, if the metaclass itself is hidden in
    > a base class; what do you think is cleaner?


    I think that, when feasible, making classes by executing a class
    statement (and functions by executing a def statement) is cleanest.
    If you can get substantial added value using type(....) calls or
    new.function(...) calls or whatever, so be it, of course -- if you NEED
    to get to that level, so be it, there's nothing sinful or bad about it,
    it may just be a tad harder to understand and maintain... no big deal,
    not comparable to (e.g.) building up a string and then doing 'exec' on
    it (THAT _is_ sinful and bad and maybe will make Guido cry...;-).


    > a) class-inside-a-def
    >
    > def BuildTemplate(<template parameters>)
    > class TemplateClass(...):
    > ...
    > return TemplateClass
    > T = BuildTemplate(<params>)


    Quite clean, IMHO, if it does all you need.


    > b) re-exec-class
    >
    > # GenericTemplate is a metaclass-enabled ancestor
    > from template import GenericTemplate
    > class TemplateClass(GenericTemplate):
    > ...
    >
    > T = TemplateClass.NewTemplate(<params>)


    To be honest I never really understood why you liked that one so much,
    apart from the fascinating possibilities it offers for delving into
    normally ignored details of 'class' statements' execution;-)

    Seriously: classes have one incredible syntax advantage when you expose
    them to users... users who write class statements get to write any gd
    code they like in a context where every localname created ends up in a
    dict, then your metaclass gets to process the dict, etc, etc. In
    comparison, functions are fine, with keyword parameters, for the
    equivalent of assignment statements, i.e.,

    class foo(subletWayToGetMetaclassOmega):
    bar = 23

    and

    foo = factoryFunctionOmega(bar=23)

    are roughly equivalent; but as soon as the body of 'class foo' starts
    having def statements, class statements, loops, etc, the factory
    function has a hard time keeping up. This syntax advantage has little
    to do with the *intended* semantics of 'class' vs function calls, and is
    the root of most "metaclass abuse" I've seen (and perpetrated -- though
    not in production code...).

    But it doesn't seem to be what you're after, and a couple days ago I saw
    somebody else asking about a metaclass to be called directly (where a
    class statement _was_ clearly what they needed, IMHO), so there must be
    something more going on here... I think I'll nickname it "class
    struggle"... many people DO seem to be struggling with classes, these
    days...


    > c) neither of the above -- there must be a better way to do it.


    ....must there...? Maybe I'm being short-sighted (wouldn't be the first
    time), but I can't see. If you could have your heart's desire, within
    the general framework of Python, how WOULD you code what you're after
    (which I'm still not entirely clear about)? I mean, how would it look
    to the user -- how it looks like _inside_ is a relatively minor issue...


    Alex
     
    Alex Martelli, Oct 6, 2004
    #14
  15. Carlos Ribeiro <> wrote:
    ...
    > > 1. You need to build an f = new.function(klass.my_code, globals()) in
    > > your classmethod that now erroneously does the exec above quoted (you
    > > could in fact do this in the metaclass, save the new function rather
    > > than the code object -- get the right globals via inspect).

    >
    > That's the part that I was missing. It looks less 'hackish' than I had
    > expect. I was really afraid that I would need to manage bytecode-level
    > hacks myself. I am not as concerned about getting the right globals,
    > though -- it's good enough if I'm able to supply whatever globals I
    > need upon calling, I think (unless I'm _really_ missing something).


    I don't think you're missing anything -- just DO consider that if you
    pass your own globals they'll be used to solve every name that's not
    local to the class body, e.g. a '__metaclass__ = metaname' assignment in
    classbody (of course, you can finesse this by calling the metaclass
    explicitly anyway;-).


    > > 2. you need to obtain the needed dict, that's just d = f() in your
    > > method

    >
    > There is still a question, but I think I already know the anwswer :)
    > If the class *has* a base class, it has to be passed as a parameter. I
    > assume that i can simply call it like this: d=f(bases); or instead,
    > that I can fill the argdefs in the new.function call. I'll try it
    > later.


    Nope, f doesn't get the bases AT ALL; the bases (as a tuple) go in only
    when you call the metaclass.


    > > 3. you can now call your metaclass with the appropriate bases and name:
    > > klas = self.__class__ # or cls if this is a classmethod;-)
    > > return type(klas)(klas.__name__, klas.__bases__, d)
    > >
    > > voila, you're there. This is all coded to minimize the need to have
    > > this method located in a specific class -- as I said I think you
    > > probably want to have this stuff in the metaclass (if you have a
    > > metaclass at all), but, whatever.

    >
    > Tehre are two reasons to do it in a metaclass: first, I tested it and
    > it works (of sorts), while for some reason it wont work for a
    > non-metaclasses enabled class. Also, because in this scenario there
    > are other things that I already need to check, and a metaclass does it
    > nicely.


    OK, then, that's where you pass the bases in, not before.

    > For now, I think I'll just be using the class-inside-a-def idiom. But


    If it can give you all you need, that may be best.

    > I'm still working around some design issues, and if the templating
    > mechanism gets in the final production code (it's only one of a few
    > competing designs), I'll surely try it again.


    OK, keep us informed of developments, then!


    Alex
     
    Alex Martelli, Oct 6, 2004
    #15
  16. On Wed, 6 Oct 2004 21:28:40 +0200, Alex Martelli <> wrote:
    > Carlos Ribeiro <> wrote:
    > ...
    > > > 1. You need to build an f = new.function(klass.my_code, globals()) in
    > > > your classmethod that now erroneously does the exec above quoted (you
    > > > could in fact do this in the metaclass, save the new function rather
    > > > than the code object -- get the right globals via inspect).

    > >
    > > That's the part that I was missing. It looks less 'hackish' than I had
    > > expect. I was really afraid that I would need to manage bytecode-level
    > > hacks myself. I am not as concerned about getting the right globals,
    > > though -- it's good enough if I'm able to supply whatever globals I
    > > need upon calling, I think (unless I'm _really_ missing something).

    >
    > I don't think you're missing anything -- just DO consider that if you
    > pass your own globals they'll be used to solve every name that's not
    > local to the class body, e.g. a '__metaclass__ = metaname' assignment in
    > classbody (of course, you can finesse this by calling the metaclass
    > explicitly anyway;-).


    I was missing the new.function() call -- I hadn't checked how could I
    build a function out of a code block up to that point, and I was
    afraid that I would need to dive deeply into Python internals. It's
    not that bad.

    > > > 2. you need to obtain the needed dict, that's just d = f() in your
    > > > method

    > >
    > > There is still a question, but I think I already know the anwswer :)
    > > If the class *has* a base class, it has to be passed as a parameter. I
    > > assume that i can simply call it like this: d=f(bases); or instead,
    > > that I can fill the argdefs in the new.function call. I'll try it
    > > later.

    >
    > Nope, f doesn't get the bases AT ALL; the bases (as a tuple) go in only
    > when you call the metaclass.


    Well. I was mistaken (and that's not be the first time). It goes like
    this; I first checked the dis.dis() code:

    27 64 LOAD_CONST 4 ('MyClass')
    67 LOAD_NAME 7 (object)
    70 BUILD_TUPLE 1
    73 LOAD_CONST 5 (<code object MyClass at
    0120A0A0, file "c:\work\help-on-c-l-py\re_exec_class.py", line 27>)
    76 MAKE_FUNCTION 0
    79 CALL_FUNCTION 0
    82 BUILD_CLASS
    83 STORE_NAME 8 (MyClass)

    .... with that build_tuple there... besides that I knew I *can* access
    the bases from within my class statement body:

    class MyClass(MyBaseClass):
    bar = MyBaseClass.foo

    .... so I had suspected that the bases *were* passed to f(). But wait!
    Actually, Python is building the stack for the BUILD_CLASS opcode.
    Right?

    > > > 3. you can now call your metaclass with the appropriate bases and name:
    > > > klas = self.__class__ # or cls if this is a classmethod;-)
    > > > return type(klas)(klas.__name__, klas.__bases__, d)
    > > >
    > > > voila, you're there. This is all coded to minimize the need to have
    > > > this method located in a specific class -- as I said I think you
    > > > probably want to have this stuff in the metaclass (if you have a
    > > > metaclass at all), but, whatever.

    > >
    > > Tehre are two reasons to do it in a metaclass: first, I tested it and
    > > it works (of sorts), while for some reason it wont work for a
    > > non-metaclasses enabled class. Also, because in this scenario there
    > > are other things that I already need to check, and a metaclass does it
    > > nicely.

    >
    > OK, then, that's where you pass the bases in, not before.
    >
    > > For now, I think I'll just be using the class-inside-a-def idiom. But

    >
    > If it can give you all you need, that may be best.
    >
    > > I'm still working around some design issues, and if the templating
    > > mechanism gets in the final production code (it's only one of a few
    > > competing designs), I'll surely try it again.

    >
    > OK, keep us informed of developments, then!
    >
    >
    >
    >
    > Alex
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >



    --
    Carlos Ribeiro
    Consultoria em Projetos
    blog: http://rascunhosrotos.blogspot.com
    blog: http://pythonnotes.blogspot.com
    mail:
    mail:
     
    Carlos Ribeiro, Oct 6, 2004
    #16
  17. On Wed, 6 Oct 2004 21:18:40 +0200, Alex Martelli <> wrote:
    > ... no big deal,
    > not comparable to (e.g.) building up a string and then doing 'exec' on
    > it (THAT _is_ sinful and bad and maybe will make Guido cry...;-).


    I swear I'll not go down to this level ;-)

    > To be honest I never really understood why you liked that one so much,
    > apart from the fascinating possibilities it offers for delving into
    > normally ignored details of 'class' statements' execution;-)


    That was part of the fun ;-)

    > Seriously: classes have one incredible syntax advantage when you expose
    > them to users... users who write class statements get to write any gd
    > code they like in a context where every localname created ends up in a
    > dict, then your metaclass gets to process the dict, etc, etc. In
    > comparison, functions are fine, with keyword parameters, for the
    > equivalent of assignment statements, i.e.,
    >
    > class foo(subletWayToGetMetaclassOmega):
    > bar = 23
    >
    > and
    >
    > foo = factoryFunctionOmega(bar=23)
    >
    > are roughly equivalent; but as soon as the body of 'class foo' starts
    > having def statements, class statements, loops, etc, the factory
    > function has a hard time keeping up. This syntax advantage has little
    > to do with the *intended* semantics of 'class' vs function calls, and is
    > the root of most "metaclass abuse" I've seen (and perpetrated -- though
    > not in production code...).


    Well, I'm looking into it for a couple of reasons. One of them is
    exactly this -- code inside a class is much easier to write and to
    structure than similar code inside a def... specialy when it comes to
    data declarations[1]. A def is pretty much limited to one level; with
    classes, it's easy to 'nest' groups of data declarations, which make
    it easier to build complex hierarchical stuff. The other reason is to
    be able to build something related to C++ templates -- a factory that
    produces customized versions of a class based on parameters, that may
    include other classes. I *thought* that a purely class-based
    templating would make it easier to understand and manage, specially if
    the magic is done "behind the scenes". But I'm not so sure now.

    [1] I think that this is the correct term, in this context... but I
    may be wrong (again).

    > But it doesn't seem to be what you're after, and a couple days ago I saw
    > somebody else asking about a metaclass to be called directly (where a
    > class statement _was_ clearly what they needed, IMHO), so there must be
    > something more going on here... I think I'll nickname it "class
    > struggle"... many people DO seem to be struggling with classes, these
    > days...
    >
    >
    > > c) neither of the above -- there must be a better way to do it.

    >
    > ...must there...? Maybe I'm being short-sighted (wouldn't be the first
    > time), but I can't see. If you could have your heart's desire, within
    > the general framework of Python, how WOULD you code what you're after
    > (which I'm still not entirely clear about)? I mean, how would it look
    > to the user -- how it looks like _inside_ is a relatively minor issue...


    Well, I don't think that (c) applies. But I was afraid that other
    people, coming with a fresh mental picture of the problem, could point
    to other types of solution. That's what (c) really means.

    >
    > Alex
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >



    --
    Carlos Ribeiro
    Consultoria em Projetos
    blog: http://rascunhosrotos.blogspot.com
    blog: http://pythonnotes.blogspot.com
    mail:
    mail:
     
    Carlos Ribeiro, Oct 6, 2004
    #17
  18. Carlos Ribeiro <> wrote:
    ...
    > > Nope, f doesn't get the bases AT ALL; the bases (as a tuple) go in only
    > > when you call the metaclass.

    >
    > Well. I was mistaken (and that's not be the first time). It goes like
    > this; I first checked the dis.dis() code:
    >
    > 27 64 LOAD_CONST 4 ('MyClass')
    > 67 LOAD_NAME 7 (object)
    > 70 BUILD_TUPLE 1
    > 73 LOAD_CONST 5 (<code object MyClass at
    > 0120A0A0, file "c:\work\help-on-c-l-py\re_exec_class.py", line 27>)
    > 76 MAKE_FUNCTION 0
    > 79 CALL_FUNCTION 0
    > 82 BUILD_CLASS
    > 83 STORE_NAME 8 (MyClass)
    >
    > ... with that build_tuple there... besides that I knew I *can* access
    > the bases from within my class statement body:
    >
    > class MyClass(MyBaseClass):
    > bar = MyBaseClass.foo
    >
    > ... so I had suspected that the bases *were* passed to f(). But wait!
    > Actually, Python is building the stack for the BUILD_CLASS opcode.
    > Right?


    Right. When you mention MyBaseClass in the body, it gets looked up in
    the globals, just where it was looked up when you passed it as a base.
    The 0 in MAKE_FUNCTION _and_ CALL_FUNCTION opcodes refers to the number
    of arguments to the function, btw...


    Alex
     
    Alex Martelli, Oct 6, 2004
    #18
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.

Share This Page