meta-class review

[Ethan Furman]

On one the many mini-reports we use, we have a bunch of counts that are
frequently zero; because the other counts can also be low, it becomes
easy to miss the non-zero counts. For example:

Code Description

Conv Errors : 6

31,N DPV Failure : 4
10: Invalid Address : 0
11: Invalid C/S/Z : 0
12: Invalid State : 0
13: Invalid City : 0
17: Insufficient Information : 0
33: Non-Deliverable : 0
98: Non-USPS zip : 0

21: Address Not Found : 0
22: Multiple Responses : 3
23: Error in Primary : 0
24: Error in Secondary : 0


So I thought I would print '-' instead...

Code Description

Conv Errors : 6

31,N DPV Failure : 4
10: Invalid Address : -
11: Invalid C/S/Z : -
12: Invalid State : -
13: Invalid City : -
17: Insufficient Information : -
33: Non-Deliverable : -
98: Non-USPS zip : -

21: Address Not Found : -
22: Multiple Responses : 3
23: Error in Primary : -
24: Error in Secondary : -


Much easier to pick out the numbers now. To support this, the code
changed slightly -- it went from

'%-25s: %7d' % ('DPV Failure', counts['D'])

to

'%-25s: %7s' % ('DPV Failure', counts['D'] if counts['D'] else '-'))

This became a pain after a dozen lines, prompting my previous question
about the difference between %s and %d when printing integers. With the
excellent replies I received I coded a short class:

class DashInt(int):
def __str__(x):
if x:
return str(x)
return '-'

and my line printing code shrunk back to it's previous size. Well, it
wasn't long before I realized that when a DashInt was added to an int,
an int came back... and so did the '0's. So I added some more lines to
the class.

def __add__(x, other):
result = super(DashInt, x).__add__(other)
return result

and then I tried to do a floating type operation, so added yet more lines...

def __add__(x, other):
result = super(DashInt, x).__add__(other)
if result == NotImplemented:
return NotImplemented
return result

and so on and so on for the basic math functions that I will be using...
what a pain! And then I had a thought... metaclasses! If DashInt used
a metaclass that would automatically check the result, and if it was
base class wrap it up in the new subclass, my DashInt class could go
back to being five simple lines, plus one more for the metaclass specifier.

So DashInt currently looks like this:

class TempInt(int):
__metaclass__ = Perpetuate
def __str__(x):
if x == 0:
return '-'
return int.__str__(x)

and Perpetuate looks like this:

class Perpetuate(type):
def __init__(yo, *args, **kwargs):
super(type, yo).__init__(*args)
def __new__(metacls, cls_name, cls_bases, cls_dict):
if len(cls_bases) > 1:
raise TypeError("multiple bases not allowed")
result_class = type.__new__( \
metacls, cls_name, cls_bases, cls_dict)
base_class = cls_bases[0]
known_methods = set()
for method in cls_dict.keys():
if callable(getattr(result_class, method)):
known_methods.add(method)

base_methods = set()
for method in base_class.__dict__.keys():
if callable(getattr(base_class, method, None)) and \
method not in ('__new__'):
base_methods.add(method)

for method in base_methods:
if method not in known_methods:
setattr(result_class, method, \
_wrap(base_class, getattr(base_class, method)))

return result_class


def _wrap(base, code):
def wrapper(self, *args, **kwargs):
result = code(self, *args, **kwargs)
if type(result) == base:
return self.__class__(result)
return result
wrapper.__name__ = code.__name__
wrapper.__doc__ = code.__doc__
return wrapper

It seems to work fine for normal operations. I had to exclude __new__
because it was a classmethod, and I suspect I would have similar issues
with staticmethods.

Any comments appreciated, especially ideas on how to better handle
class- and staticmethods

~Ethan~

 

[Carl Banks]

On one the many mini-reports we use, we have a bunch of counts that are
frequently zero; because the other counts can also be low, it becomes
easy to miss the non-zero counts.  For example:

Code  Description

       Conv Errors              :       6

31,N  DPV Failure              :       4
10:   Invalid Address          :       0
11:   Invalid C/S/Z            :       0
12:   Invalid State            :       0
13:   Invalid City             :       0
17:   Insufficient Information :       0
33:   Non-Deliverable          :       0
98:   Non-USPS zip             :       0

21:   Address Not Found        :       0
22:   Multiple Responses       :       3
23:   Error in Primary         :       0
24:   Error in Secondary       :       0

So I thought I would print '-' instead...

Code  Description

       Conv Errors              :       6

31,N  DPV Failure              :       4
10:   Invalid Address          :       -
11:   Invalid C/S/Z            :       -
12:   Invalid State            :       -
13:   Invalid City             :       -
17:   Insufficient Information :       -
33:   Non-Deliverable          :       -
98:   Non-USPS zip             :       -

21:   Address Not Found        :       -
22:   Multiple Responses       :       3
23:   Error in Primary         :       -
24:   Error in Secondary       :       -

Much easier to pick out the numbers now.  To support this, the code
changed slightly -- it went from

'%-25s: %7d' % ('DPV Failure', counts['D'])

to

'%-25s: %7s' % ('DPV Failure', counts['D'] if counts['D'] else '-'))

This became a pain after a dozen lines, prompting my previous question
about the difference between %s and %d when printing integers.  With the
excellent replies I received I coded a short class:

class DashInt(int):
     def __str__(x):
         if x:
             return str(x)
         return '-'

and my line printing code shrunk back to it's previous size.  Well, it
wasn't long before I realized that when a DashInt was added to an int,
an int came back... and so did the '0's.  So I added some more lines to
the class.

     def __add__(x, other):
         result = super(DashInt, x).__add__(other)
         return result

and then I tried to do a floating type operation, so added yet more lines....

     def __add__(x, other):
         result = super(DashInt, x).__add__(other)
         if result == NotImplemented:
             return NotImplemented
         return result

and so on and so on for the basic math functions that I will be using...
what a pain!  And then I had a thought... metaclasses!  If DashInt used
a metaclass that would automatically check the result, and if it was
base class wrap it up in the new subclass, my DashInt class could go
back to being five simple lines, plus one more for the metaclass specifier.

So DashInt currently looks like this:

class TempInt(int):
     __metaclass__ = Perpetuate
     def __str__(x):
         if x == 0:
             return '-'
         return int.__str__(x)

and Perpetuate looks like this:

class Perpetuate(type):
     def __init__(yo, *args, **kwargs):
         super(type, yo).__init__(*args)
     def __new__(metacls, cls_name, cls_bases, cls_dict):
         if len(cls_bases) > 1:
             raise TypeError("multiple bases not allowed")
         result_class = type.__new__( \
           metacls, cls_name, cls_bases, cls_dict)
         base_class = cls_bases[0]
         known_methods = set()
         for method in cls_dict.keys():
             if callable(getattr(result_class, method)):
                 known_methods.add(method)

         base_methods = set()
         for method in base_class.__dict__.keys():
             if callable(getattr(base_class, method, None)) and \
                     method not in ('__new__'):
                 base_methods.add(method)

         for method in base_methods:
             if method not in known_methods:
                 setattr(result_class, method, \
                         _wrap(base_class, getattr(base_class, method)))

         return result_class

def _wrap(base, code):
     def wrapper(self, *args, **kwargs):
         result = code(self, *args, **kwargs)
         if type(result) == base:
             return self.__class__(result)
         return result
     wrapper.__name__ = code.__name__
     wrapper.__doc__ = code.__doc__
     return wrapper

It seems to work fine for normal operations.  I had to exclude __new__
because it was a classmethod, and I suspect I would have similar issues
with staticmethods.

Any comments appreciated, especially ideas on how to better handle
class- and staticmethods

Well, it's definitely overkill for printing a dash instead of a zero,
but a lot of people have asked how to create a subtype of int (or
other builtin) that coerces the other operand, and your idea is
interesting in that you don't have to write boilerplate to override
all the operations.

Main drawback is that it's incomplete. For example, it doesn't coerce
properties. int.real returns the real part of the int (i.e., the int
itself). A subclass's real attribute should return an instance of the
subclass, but it won't. Another example is float.__divmod__, which
returns a tuple. Your coercive type would fail to convert the items
of that tuple. A metaclass like this I think would be possible, with
the understanding that it can never be foolproof, but it needs more
work.

Pointers:
Defining __init__ isn't necessary for this metaclass.

The "len(cls_bases) > 1" test can be thwarted if the base type
multiply inherits from other types itself. The best thing to do is
handle the case of arbitrary type hierarchies, but if you don't want
to do that then the right way to catch it is to create the subtype
then check that the __mro__ is (type, base_type, object).


Carl Banks

 

[Ethan Furman]

Well, it's definitely overkill for printing a dash instead of a zero,
but a lot of people have asked how to create a subtype of int (or
other builtin) that coerces the other operand, and your idea is
interesting in that you don't have to write boilerplate to override
all the operations.

Main drawback is that it's incomplete. For example, it doesn't coerce
properties. int.real returns the real part of the int (i.e., the int
itself). A subclass's real attribute should return an instance of the
subclass, but it won't.

Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'int' has no attribute 'real'

What am I missing here?

Another example is float.__divmod__, which
returns a tuple. Your coercive type would fail to convert the items
of that tuple.

Good point -- I'll get that included.

A metaclass like this I think would be possible, with
the understanding that it can never be foolproof, but it needs more
work.

Pointers:
Defining __init__ isn't necessary for this metaclass.

The "len(cls_bases) > 1" test can be thwarted if the base type
multiply inherits from other types itself. The best thing to do is
handle the case of arbitrary type hierarchies, but if you don't want
to do that then the right way to catch it is to create the subtype
then check that the __mro__ is (type, base_type, object).

Thanks for the tips, Carl.

What I had wanted was to be able to specify which type(s) to look for in
cases of multiple inheritance, but I'm not sure how to pass parameters
to the metaclass in python2...

Can anybody shed some light on that?

Thanks!

~Ethan~