stackoverflow question

Discussion in 'Python' started by Ethan Furman, Mar 9, 2012.

  1. Ethan Furman

    Ethan Furman Guest

    Hey all!

    I posted a question/answer on SO earlier, but there seems to be some
    confusion around either the question or the answer (judging from the
    comments).

    http://stackoverflow.com/q/9638921/208880

    If anyone here is willing to take a look at it and let me know if I did
    not write it well, I would appreciate the feedback.


    Here's the question text:
    ------------------------
    I'm writing a metaclass to do some cool stuff, and part of its
    processing is to check that certain attributes exist when the class is
    created. Some of these are mutable, and would normally be set in
    `__init__`, but since `__init__` isn't run until the instance is created
    the metaclass won't know that the attribute *will* be created, and
    raises an error. I could do something like:

    class Test(meta=Meta):
    mutable = None
    def __init__(self):
    self.mutable = list()

    But that isn't very elegant, and also violates DRY.

    What I need is some way to have:

    class Test(metaclass=Meta):
    mutable = list()

    t1 = Test()
    t2 = Test()
    t1.mutable.append('one')
    t2.mutable.append('two')
    t1.mutable # prints ['one']
    t2.mutable # prints ['two']

    Any ideas on how this can be accomplished?

    Also, the metaclass doing the checking doesn't care what type of object
    the attribute is, only that it is there.
    ---------------------------

    and the answer text:
    -------------------
    There are a couple ways to do this:

    1. Have the metaclass check all the attributes, and if they are one
    of the mutables (`list`, `dict`, `set`, etc.) replace the attribute with
    a descriptor that will activate on first access and update the instance
    with a fresh copy of the mutable.

    2. Provide the descriptor from (1) as a decorator to be used when
    writing the class.

    I prefer (2) is it gives complete control to the class author, and
    simplifies those cases where the class-level mutable attribute *should*
    be shared amongst all the instances.

    Here's the decorator-descriptor:

    class ReplaceMutable:
    def __init__(self, func):
    self.func = func
    def __call__(self):
    return self
    def __get__(self, instance, owner):
    if instance is None:
    return self
    result = self.func()
    setattr(instance, self.func.__name__, result)
    return result

    and the test class:

    class Test:
    @ReplaceMutable
    def mutable():
    return list()

    t1 = Test()
    t2 = Test()
    t1.mutable.append('one')
    t2.mutable.append('two')
    print(t1.mutable)
    print(t2.mutable)

    How it works:

    Just like `property`, `ReplaceMutable` is a descriptor object with the
    same name as the attribute it is replacing. Unlike `property`, it does
    not define `__set__` nor `__delete__`, so when code tries to rebind the
    name (`mutable` in the test above) in the instance Python will allow it
    to do so. This is the same idea behind caching descriptors.

    `ReplaceMutable` is decorating a function (with the name of the desired
    attribute) that simply returns whatever the instance level attribute
    should be initialized with (an empty `list` in the example above). So
    the first time the attribute is looked up on an instance it will not be
    found in the instance dictionary and Python will activate the
    descriptor; the descriptor then calls the function to retrieve the
    initial object/data/whatever, stores it in the instance, and then
    returns it. The next time that attribute is accessed *on that instance*
    it will be in the instance dictionary, and that is what will be used.
    -----------------------------------

    Thanks,
    ~Ethan~
     
    Ethan Furman, Mar 9, 2012
    #1
    1. Advertising

  2. On 2012-03-09 22:10:18 +0000, Ethan Furman said:

    > Hey all!
    >
    > I posted a question/answer on SO earlier, but there seems to be some
    > confusion around either the question or the answer (judging from the
    > comments).
    >
    > http://stackoverflow.com/q/9638921/208880
    >
    > If anyone here is willing to take a look at it and let me know if I did
    > not write it well, I would appreciate the feedback.
    >
    >
    > Here's the question text:
    > ------------------------
    > I'm writing a metaclass to do some cool stuff, and part of its
    > processing is to check that certain attributes exist when the class is
    > created. Some of these are mutable, and would normally be set in
    > `__init__`, but since `__init__` isn't run until the instance is
    > created the metaclass won't know that the attribute *will* be created,
    > and raises an error. I could do something like:
    >
    > class Test(meta=Meta):
    > mutable = None
    > def __init__(self):
    > self.mutable = list()
    >
    > But that isn't very elegant, and also violates DRY.
    >
    > What I need is some way to have:
    >
    > class Test(metaclass=Meta):
    > mutable = list()
    >
    > t1 = Test()
    > t2 = Test()
    > t1.mutable.append('one')
    > t2.mutable.append('two')
    > t1.mutable # prints ['one']
    > t2.mutable # prints ['two']
    >
    > Any ideas on how this can be accomplished?
    >
    > Also, the metaclass doing the checking doesn't care what type of object
    > the attribute is, only that it is there.
    > ---------------------------


    Why check what you can ensure? The __init__ function your metaclass
    passes to type() doesn't have to be the __init__ method your metaclass
    received. Consider the following:

    >>> import functools as f
    >>>
    >>> def make_init(real_init):
    >>> """Define an __init__ method that ensures ``self.mutable`` is set. If the
    >>> passed ``real_init`` function later replaces ``self.mutable``, that value
    >>> is preserved; otherwise, ``self.mutable`` is set to a new, empty list.
    >>>
    >>> Arguments to the generated ``__init__`` method are passed to the original
    >>> ``real_init`` unchanged.
    >>> """
    >>> def __init__(self, *args, **kwargs):
    >>> self.mutable = list()
    >>> if real_init is not None:
    >>> return real_init(self, *args, **kwargs)
    >>> if real_init is not None:
    >>> f.update_wrapper(__init__, real_init)
    >>> return __init__
    >>>
    >>> class Meta(type):
    >>> def __new__(meta, name, parents, attributes):
    >>> attributes['__init__'] = make_init(attributes.get('__init__', None))
    >>> return type.__new__(Meta, name, parents, attributes)
    >>>
    >>> class C(object):
    >>> __metaclass__ = Meta
    >>>
    >>> a, b = C(), C()
    >>>
    >>> a.mutable.append(3)
    >>> b.mutable.append(5)
    >>>
    >>> a.mutable

    [3]
    >>> b.mutable

    [5]

    All instances of classes whose metaclass is Meta will, guaranteed, have
    an instance field named 'mutable'. Its value is a list created at
    instance creation time, unless the instance's __init__ provides a
    different value.

    What've I missed?

    -o
     
    Owen Jacobson, Mar 10, 2012
    #2
    1. Advertising

  3. Ethan Furman

    Ethan Furman Guest

    Owen Jacobson wrote:
    > On 2012-03-09 22:10:18 +0000, Ethan Furman said:
    >
    >> Hey all!
    >>
    >> I posted a question/answer on SO earlier, but there seems to be some
    >> confusion around either the question or the answer (judging from the
    >> comments).
    >>
    >> http://stackoverflow.com/q/9638921/208880
    >>
    >> If anyone here is willing to take a look at it and let me know if I
    >> did not write it well, I would appreciate the feedback.
    >>
    >>
    >> Here's the question text:
    >> ------------------------
    >> I'm writing a metaclass to do some cool stuff, and part of its
    >> processing is to check that certain attributes exist when the class is
    >> created. Some of these are mutable, and would normally be set in
    >> `__init__`, but since `__init__` isn't run until the instance is
    >> created the metaclass won't know that the attribute *will* be created,
    >> and raises an error. I could do something like:
    >>
    >> class Test(meta=Meta):
    >> mutable = None
    >> def __init__(self):
    >> self.mutable = list()
    >>
    >> But that isn't very elegant, and also violates DRY.
    >>
    >> What I need is some way to have:
    >>
    >> class Test(metaclass=Meta):
    >> mutable = list()
    >>
    >> t1 = Test()
    >> t2 = Test()
    >> t1.mutable.append('one')
    >> t2.mutable.append('two')
    >> t1.mutable # prints ['one']
    >> t2.mutable # prints ['two']
    >>
    >> Any ideas on how this can be accomplished?
    >>
    >> Also, the metaclass doing the checking doesn't care what type of
    >> object the attribute is, only that it is there.
    >> ---------------------------

    >
    > Why check what you can ensure? The __init__ function your metaclass
    > passes to type() doesn't have to be the __init__ method your metaclass
    > received. Consider the following:
    >
    >>>> import functools as f
    >>>>
    >>>> def make_init(real_init):
    >>>> """Define an __init__ method that ensures ``self.mutable`` is
    >>>> set. If the
    >>>> passed ``real_init`` function later replaces ``self.mutable``,
    >>>> that value
    >>>> is preserved; otherwise, ``self.mutable`` is set to a new, empty
    >>>> list.
    >>>>
    >>>> Arguments to the generated ``__init__`` method are passed to the
    >>>> original
    >>>> ``real_init`` unchanged.
    >>>> """
    >>>> def __init__(self, *args, **kwargs):
    >>>> self.mutable = list()
    >>>> if real_init is not None:
    >>>> return real_init(self, *args, **kwargs)
    >>>> if real_init is not None:
    >>>> f.update_wrapper(__init__, real_init)
    >>>> return __init__
    >>>>
    >>>> class Meta(type):
    >>>> def __new__(meta, name, parents, attributes):
    >>>> attributes['__init__'] =
    >>>> make_init(attributes.get('__init__', None))
    >>>> return type.__new__(Meta, name, parents, attributes)
    >>>>
    >>>> class C(object):
    >>>> __metaclass__ = Meta
    >>>>
    >>>> a, b = C(), C()
    >>>>
    >>>> a.mutable.append(3)
    >>>> b.mutable.append(5)
    >>>>
    >>>> a.mutable

    > [3]
    >>>> b.mutable

    > [5]
    >
    > All instances of classes whose metaclass is Meta will, guaranteed, have
    > an instance field named 'mutable'. Its value is a list created at
    > instance creation time, unless the instance's __init__ provides a
    > different value.


    The idea is good. The devil is in the details, as usual. How is the
    metaclass going to know:

    1) which attributes to replace
    2) what to replace them with?

    ~Ethan~
     
    Ethan Furman, Mar 10, 2012
    #3
  4. On 2012-03-10 22:21:55 +0000, Ethan Furman said:

    > Owen Jacobson wrote:
    >> On 2012-03-09 22:10:18 +0000, Ethan Furman said:
    >>
    >>> Hey all!
    >>>
    >>> I posted a question/answer on SO earlier, but there seems to be some
    >>> confusion around either the question or the answer (judging from the
    >>> comments).
    >>>
    >>> http://stackoverflow.com/q/9638921/208880
    >>>
    >>> If anyone here is willing to take a look at it and let me know if I did
    >>> not write it well, I would appreciate the feedback.
    >>>
    >>>
    >>> Here's the question text:
    >>> ------------------------
    >>> I'm writing a metaclass to do some cool stuff, and part of its
    >>> processing is to check that certain attributes exist when the class is
    >>> created. Some of these are mutable, and would normally be set in
    >>> `__init__`, but since `__init__` isn't run until the instance is
    >>> created the metaclass won't know that the attribute *will* be created,
    >>> and raises an error. I could do something like:
    >>>
    >>> class Test(meta=Meta):
    >>> mutable = None
    >>> def __init__(self):
    >>> self.mutable = list()
    >>>
    >>> But that isn't very elegant, and also violates DRY.
    >>>
    >>> What I need is some way to have:
    >>>
    >>> class Test(metaclass=Meta):
    >>> mutable = list()
    >>>
    >>> t1 = Test()
    >>> t2 = Test()
    >>> t1.mutable.append('one')
    >>> t2.mutable.append('two')
    >>> t1.mutable # prints ['one']
    >>> t2.mutable # prints ['two']
    >>>
    >>> Any ideas on how this can be accomplished?
    >>>
    >>> Also, the metaclass doing the checking doesn't care what type of object
    >>> the attribute is, only that it is there.
    >>> ---------------------------

    >>
    >> Why check what you can ensure? The __init__ function your metaclass
    >> passes to type() doesn't have to be the __init__ method your metaclass
    >> received.


    [… __init__-generation technique elided …]

    >> All instances of classes whose metaclass is Meta will, guaranteed, have
    >> an instance field named 'mutable'. Its value is a list created at
    >> instance creation time, unless the instance's __init__ provides a
    >> different value.

    >
    > The idea is good. The devil is in the details, as usual. How is the
    > metaclass going to know:
    >
    > 1) which attributes to replace
    > 2) what to replace them with?


    I can think of at least three techniques; others are certainly possible:

    1. As with the example code, the list is hard-coded in the metaclass's
    source code.
    2. The list (or, rather, a dictionary) is drawn from a class attribute
    of the class being created:

    class Foo(object):
    mandatory_fields = {'mutable': list, 'more_stuff': str}
    __metaclass__ = IntrospectingMetaclass

    3. A metaclass-returning factory produces new metaclasses on demand,
    each of which has a dict of mandatory fields baked into it. (This is a
    hybrid of the two approaches, and can easily have some messy side
    effects on your app's type ecology if used carelessly.)

    Of course, you can also treat this the other way around: instead of
    enforcing that instances have specific fields, you could have users of
    those instances be aware that the field might not exist, and wrap
    access to the field in a try/except NameError block or use getattr to
    read the attribute.

    What's appropriate really depends on how you plan to use this
    metaclass, and on the nature of the higher-level problem to which "I
    know, I'll use metaclasses" is your answer. How about telling us a
    slightly broader story about your problem?

    -o
     
    Owen Jacobson, Mar 11, 2012
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Rob Meade

    Visual Studio: StackOverflow

    Rob Meade, Oct 26, 2005, in forum: ASP .Net
    Replies:
    23
    Views:
    8,297
    Rob Meade
    Nov 6, 2005
  2. Bryan Castillo

    StackOverflow Exception in JNI

    Bryan Castillo, Jun 10, 2004, in forum: Java
    Replies:
    6
    Views:
    5,130
    Gordon Beaton
    Jun 11, 2004
  3. sravan
    Replies:
    2
    Views:
    1,369
    Alan Moore
    Sep 3, 2004
  4. Efrat Ben-David
    Replies:
    2
    Views:
    1,316
    Andrew Thompson
    Sep 26, 2004
  5. Replies:
    3
    Views:
    985
    Arne Vajhøj
    Nov 2, 2009
Loading...

Share This Page