Generic constructors and duplication of internal Python logic

J

John J. Lee

This is one of those things that I can't quite believe I've never
needed to do before.

I've got a set classes, each of which has a set of attributes that all
behave very similarly. So, I have a class attribute (Blah.attr_spec
below), which is used by a mixin class to implement various methods
that would otherwise be highly repetitious across these classes. I'd
like to do the same for the constructor, to avoid this kind of
nonsense:

class Blah(NamesMixin):
attr_spec = ["foo", "bar", "baz",
("optional1", None), ("optional2", None)]
def __init__(self, foo, bar, baz,
optional1=None, optional2=None):
self.foo, self.bar, self.baz = \
foo, bar, baz
self.optional1, self.optional2 = \
optional1, optional2

So, I wrote a mixin class whose __init__ looks at the attr_spec
attribute, and uses args and kwds (below) to assign attributes in the
same sort of way as the special-case code above:

class ArgsMixin:
def __init__(self, *args, **kwds):
# set attributes based on arguments passed in, as done
# manually in Blah.__init__, above
... lots of logic already present in Python goes here...

That immediately leads to duplication of Python's internal logic: I
have to check things like:

-are there too many positional arguments?
-any unexpected keyword arguments?
-multiple keyword arguments?
-any duplication between positional and keyword arguments?

etc.

Surely there's some easy way of making use of Python's internal logic
here? For some reason, I can't see how. Can anybody see a way?


John
 
P

Peter Otten

John said:
This is one of those things that I can't quite believe I've never
needed to do before.

I've got a set classes, each of which has a set of attributes that all
behave very similarly. So, I have a class attribute (Blah.attr_spec
below), which is used by a mixin class to implement various methods
that would otherwise be highly repetitious across these classes. I'd
like to do the same for the constructor, to avoid this kind of
nonsense:

class Blah(NamesMixin):
attr_spec = ["foo", "bar", "baz",
("optional1", None), ("optional2", None)]
def __init__(self, foo, bar, baz,
optional1=None, optional2=None):
self.foo, self.bar, self.baz = \
foo, bar, baz
self.optional1, self.optional2 = \
optional1, optional2

So, I wrote a mixin class whose __init__ looks at the attr_spec
attribute, and uses args and kwds (below) to assign attributes in the
same sort of way as the special-case code above:

class ArgsMixin:
def __init__(self, *args, **kwds):
# set attributes based on arguments passed in, as done
# manually in Blah.__init__, above
... lots of logic already present in Python goes here...

That immediately leads to duplication of Python's internal logic: I
have to check things like:

-are there too many positional arguments?
-any unexpected keyword arguments?
-multiple keyword arguments?
-any duplication between positional and keyword arguments?

etc.

Surely there's some easy way of making use of Python's internal logic
here? For some reason, I can't see how. Can anybody see a way?

You could use a noop method check_attrs() to define the argspec.
check_attrs() is then called from the mixin's __init__() just to check that
the parameters comply.

import inspect

def make_attrspec(f):
a = inspect.getargspec(f)
names = a[0]
if names[0] in ["self", "cls"]:
# XXX for now relies on naming convention
del names[0]
defaults = a[3]
for i in range(-1, -len(defaults)-1, -1):
names = names, defaults
return names

class ArgsMixin:
def __init__(self, *args, **kwds):
self.check_attrs(*args, **kwds)

class Blah(ArgsMixin):
def check_attrs(self, foo, bar, baz, optional1="first",
optional2="second"):
pass
attr_spec = make_attrspec(check_attrs)


print Blah.attr_spec
Blah(1, 2, 3)
Blah(1, 2, optional1="o1", baz=99)
Blah(1, 2)

I'm not sure whether check_attrs() should be a class or static method, so I
made it a standard method for now.
Todo: automagically "normalize" the argument list, e. g. convert
Blah(1, 2, optional1="o1", baz=99) to Blah(1, 2, 99, optional1="o1").
A workaround would be to make them all keyword arguments

kwds.update(dict(zip(Blah.attr_spec, args)))

after the self.check_attrs() call.

Peter
 
M

Michele Simionato

I have to check things like:

-are there too many positional arguments?
-any unexpected keyword arguments?
-multiple keyword arguments?
-any duplication between positional and keyword arguments?

etc.

Surely there's some easy way of making use of Python's internal logic
here? For some reason, I can't see how. Can anybody see a way?


John

Have you thought of performing the checks in the __call__ method
of a custom metaclass?


Michele Simionato
 
J

John J. Lee

Peter Otten said:
John J. Lee wrote: [...]
You could use a noop method check_attrs() to define the argspec.
check_attrs() is then called from the mixin's __init__() just to check that
the parameters comply.

import inspect

def make_attrspec(f):
a = inspect.getargspec(f)
names = a[0]
if names[0] in ["self", "cls"]:
# XXX for now relies on naming convention
del names[0]
defaults = a[3]
for i in range(-1, -len(defaults)-1, -1):
names = names, defaults
return names

class ArgsMixin:
def __init__(self, *args, **kwds):
self.check_attrs(*args, **kwds)

class Blah(ArgsMixin):
def check_attrs(self, foo, bar, baz, optional1="first",
optional2="second"):
pass
attr_spec = make_attrspec(check_attrs)


Clever. But how to get __init__ to assign the arguments to the
instance?

b = Blah(foo=1, bar=2, optional1=4)
assert b.foo, b.bar, b.optional1 = 1, 2, 4


That half of the problem is missing in your solution.

[...]
Todo: automagically "normalize" the argument list, e. g. convert
Blah(1, 2, optional1="o1", baz=99) to Blah(1, 2, 99, optional1="o1").

As long as baz ends up getting assigned the value 99 and optional1
gets the value "o1", I don't care how that's achieved (except that I'd
like it done without having to write out all the logic as I have ATM
-- I want to reuse Python's own internal knowledge of argument lists
to get my attributes assigned).

A workaround would be to make them all keyword arguments

kwds.update(dict(zip(Blah.attr_spec, args)))

after the self.check_attrs() call.

Not acceptable in my case.


John
 
M

Michele Simionato

No. How would that help?

I had in mind something like that:

class _WithConstructorChecked(type): # helper metaclass
def __call__(cls, *args, **kw):
assert len(args)<=2, "%s called with more than 2 args" % cls
assert kw.has_key("kw"), "%s needs a 'kw=' argument" % cls
return super(_WithConstructorChecked,cls).__call__(*args,**kw)

class WithConstructorChecked(object): # mixin class
__metaclass__ = _WithConstructorChecked

class C(WithConstructorChecked):
def __init__(self, *args, **kw):
pass

c=C(1,2,kw=3) # ok; try different signatures to get assertion errors

In this example the mixin class WithConstructorChecked ensures that C is
called with at least two positional arguments and a keyword argument named
'kw'. The code of class C is not touched at all, you just add
WithConstructorChecked to the list of its bases.

Is that what you are looking for?


Michele Simionato
 
P

Peter Otten

John said:
Peter Otten said:
John J. Lee wrote: [...]
You could use a noop method check_attrs() to define the argspec.
check_attrs() is then called from the mixin's __init__() just to check
that the parameters comply.

import inspect

def make_attrspec(f):
a = inspect.getargspec(f)
names = a[0]
if names[0] in ["self", "cls"]:
# XXX for now relies on naming convention
del names[0]
defaults = a[3]
for i in range(-1, -len(defaults)-1, -1):
names = names, defaults
return names

class ArgsMixin:
def __init__(self, *args, **kwds):
self.check_attrs(*args, **kwds)

class Blah(ArgsMixin):
def check_attrs(self, foo, bar, baz, optional1="first",
optional2="second"):
pass
attr_spec = make_attrspec(check_attrs)


Clever. But how to get __init__ to assign the arguments to the
instance?

b = Blah(foo=1, bar=2, optional1=4)
assert b.foo, b.bar, b.optional1 = 1, 2, 4


That half of the problem is missing in your solution.


I'll give it a try (untested):

class ArgsMixin:
def __init__(self, *args, **kwds):
self.check_attrs(*args, **kwds)
# positionals provided
for n, v in zip(self.attr_spec, args):
setattr(self, n, v)
# positionals using defaults
for nv in self.attr_spec[len(args):]:
if not isinstance(nv, basestring):
n, v = nv
if not n in kwds:
setattr(self, n, v)
# keyword args
for n, v in kwds.iteritems():
setattr(self, n, v)

The clumsy basestring check could be avoided if you either split attr_spec
in the tuple/non-tuple parts or precalculate the first non-tuple position.

Peter
 
J

John J. Lee

Peter Otten said:
John said:
Peter Otten said:
John J. Lee wrote: [...]
You could use a noop method check_attrs() to define the argspec.
check_attrs() is then called from the mixin's __init__() just to check
that the parameters comply.

import inspect

def make_attrspec(f):
a = inspect.getargspec(f)
names = a[0]
if names[0] in ["self", "cls"]:
# XXX for now relies on naming convention
del names[0]
defaults = a[3]
for i in range(-1, -len(defaults)-1, -1):
names = names, defaults
return names

class ArgsMixin:
def __init__(self, *args, **kwds):
self.check_attrs(*args, **kwds)

class Blah(ArgsMixin):
def check_attrs(self, foo, bar, baz, optional1="first",
optional2="second"):
pass
attr_spec = make_attrspec(check_attrs)


Clever. But how to get __init__ to assign the arguments to the
instance?

b = Blah(foo=1, bar=2, optional1=4)
assert b.foo, b.bar, b.optional1 = 1, 2, 4


That half of the problem is missing in your solution.


I'll give it a try (untested):

class ArgsMixin:
def __init__(self, *args, **kwds):
self.check_attrs(*args, **kwds)
# positionals provided
for n, v in zip(self.attr_spec, args):
setattr(self, n, v)


But the actual args may (quite legally) be longer than the args part
of attr_spec, and then:
attr_spec = ["foo", "bar", ("a", "b")]
args = [1, 2]
zip(attr_spec, args) [('foo', 1), ('bar', 2)]
args = [1, 2, 3]
zip(attr_spec, args) [('foo', 1), ('bar', 2), (('a', 'b'), 3)] # wrong, can't do setattr!

This can be fixed, but then we move towards my clumsy-but-correct (I
hope) implementation. I'm disappointed there's no simple solution
that makes better use of Python's own knowledge of argument
specifications.

# positionals using defaults
for nv in self.attr_spec[len(args):]:
if not isinstance(nv, basestring):
n, v = nv
if not n in kwds:
setattr(self, n, v)
# keyword args
for n, v in kwds.iteritems():
setattr(self, n, v)
[...]


John
 
J

John J. Lee

I had in mind something like that:

class _WithConstructorChecked(type): # helper metaclass
def __call__(cls, *args, **kw):
assert len(args)<=2, "%s called with more than 2 args" % cls
assert kw.has_key("kw"), "%s needs a 'kw=' argument" % cls
return super(_WithConstructorChecked,cls).__call__(*args,**kw)

class WithConstructorChecked(object): # mixin class
__metaclass__ = _WithConstructorChecked

class C(WithConstructorChecked):
def __init__(self, *args, **kw):
pass

c=C(1,2,kw=3) # ok; try different signatures to get assertion errors

In this example the mixin class WithConstructorChecked ensures that C is
called with at least two positional arguments and a keyword argument named
'kw'. The code of class C is not touched at all, you just add
WithConstructorChecked to the list of its bases.

Is that what you are looking for?

No. :)

See my replies to Peter: I'm trying to re-use Python's own knowledge
of to get attributes assigned, without repetition (see my original
post for the kind of repetition that motivated me), and without simply
liberally assigning as attributes any old arguments that get passed
in. Seems like it's not possible, though. I don't see how a
metaclass helps here.


John
 
P

Peter Otten

John J. Lee wrote:

[my buggy code]
This can be fixed, but then we move towards my clumsy-but-correct (I
hope) implementation. I'm disappointed there's no simple solution
that makes better use of Python's own knowledge of argument
specifications.

I warned you that my code was not thoroughly tested, and I aggree with you
that this is getting clumsy.
But will I stop trying to come up with something better? no way. Here's my
next try, and I'm likely to present something completely different again if
you don't like it - not sure if that's a promise or a threat :)

import inspect, sys

def generateSource(m):
a = inspect.getargspec(m)
names = a[0]
if a[1] or a[2]:
raise Execption("*args/**kw not supported")
nself = names[0]
assignments = [" %s.%s = %s" % (nself, n, n) for n in names[1:]]
return "def %s%s:\n%s" % (m.__name__,
inspect.formatargspec(*inspect.getargspec(m)),
"\n".join(assignments))

def generateMethod(m):
"""Generate a method with the same signiture
as m. For every argument arg a corresponding line

self.arg = arg

is added.
"""
# source code generation factored out for easier debugging
ns = {}
exec generateSource(m) in ns
return ns[m.__name__]

class Demo(object):
def __init__(self, foo, bar, baz=2, bang=3):
pass
__init__ = generateMethod(__init__)
def __str__(self):
return ", ".join(["%s=%r" % (n, getattr(self, n))
for n in "foo bar baz bang".split()])

print "--- generatatSource(Demo.__init__) ---"
print generateSource(Demo.__init__)
print "--------------------------------------"

print Demo(1, 2, 3)
print Demo(1, 2)
print Demo(1, 2, bang=99)
print Demo(1, bang=99, bar=11)
print Demo(1, bar=11)

Peter
 
P

Peter Otten

Peter said:
But will I stop trying to come up with something better? no way. Here's my
next try, and I'm likely to present something completely different again
if you don't like it - not sure if that's a promise or a threat :)

A threat.

def setLocals(d, selfName="self"):
self = d.pop(selfName)
for n, v in d.iteritems():
setattr(self, n, v)

class Demo(object):
def __init__(self, foo, bar, baz=2, bang=3):
setLocals(locals())


Peter
 
J

John J. Lee

Peter Otten said:
A threat.

def setLocals(d, selfName="self"):
self = d.pop(selfName)
for n, v in d.iteritems():
setattr(self, n, v)

class Demo(object):
def __init__(self, foo, bar, baz=2, bang=3):
setLocals(locals())

Perfect! About a hundred times better than my solution. Maybe this
should be a Python Cookbook entry?


John
 
S

Sebastien de Menten

Here is a metaclass for uber-lazy user :)

Concretely, at the creation of the class it takes the source of the
__init__ function and add, at the first line of __init__, the line
that sets the attributes :
self.foo, self.bar, self.baz, self.optional1, self.optional2 =
foo, bar, baz, optional1, optional2


Nothing is done for *args or **kwargs but they could also be assigned
to some attributes of the class (like self.args and self.kwargs ?).
Moreover, for really lazy users, the line for calling the __init__ of
the father of the class (if necessary)
super(Father, self).__init__()
could also be dynamically added



#----------------------------------------------------
import inspect,re

class autoArgInit(type):
"""
Replace any occurence of xxx in the class by a col version and a
row version and yyy by its complementary row/col
"""
def __new__(cls,classname,bases,classdict):
# get the __init__ function
function_object = classdict["__init__"]

# get the source of the function
function_source = inspect.getsource(function_object)

# detect indentation of the function definition and remove it
such that def __init__ is at beginning of line
indentation_level = re.match("( *)def
(.*)\(",function_source).groups()[0]
function_source =
re.sub("^"+indentation_level,"",function_source)
function_source =
re.sub("\n"+indentation_level,"\n",function_source)

# split the lines to add new lines easely
function_lines = function_source.split("\n")

# detect indentation inside the function
indentation =
re.match("(\s*)\S*",function_lines[1]).groups()[0]

# take argument list without self
args = inspect.getargspec(function_object)[0][1:]

# create the line for assignment
assign_code = indentation + ", ".join(map(lambda s:
"self.%s"%(s),args)) + " = " + ", ".join(args)

# insert it in the code
function_lines.insert(1,assign_code)

# join the code again
new_function_source = "\n".join(function_lines)

# evaluate it and replace the __init__ definition in classdict
exec new_function_source in function_object.func_globals,
classdict

return type.__new__(cls,classname,bases,classdict)

class test(object):
__metaclass__ = autoArgInit
def __init__(self, baz, top, foo=3, r = 5, *args, **kwargs):
assert self.baz == baz
assert self.top == top
assert self.foo == foo
assert self.r == r

test(3,4,6)
#----------------------------------------------------


Seb






This is one of those things that I can't quite believe I've never
needed to do before.

I've got a set classes, each of which has a set of attributes that all
behave very similarly. So, I have a class attribute (Blah.attr_spec
below), which is used by a mixin class to implement various methods
that would otherwise be highly repetitious across these classes. I'd
like to do the same for the constructor, to avoid this kind of
nonsense:

class Blah(NamesMixin):
attr_spec = ["foo", "bar", "baz",
("optional1", None), ("optional2", None)]
def __init__(self, foo, bar, baz,
optional1=None, optional2=None):
self.foo, self.bar, self.baz = \
foo, bar, baz
self.optional1, self.optional2 = \
optional1, optional2

So, I wrote a mixin class whose __init__ looks at the attr_spec
attribute, and uses args and kwds (below) to assign attributes in the
same sort of way as the special-case code above:

class ArgsMixin:
def __init__(self, *args, **kwds):
# set attributes based on arguments passed in, as done
# manually in Blah.__init__, above
... lots of logic already present in Python goes here...

That immediately leads to duplication of Python's internal logic: I
have to check things like:

-are there too many positional arguments?
-any unexpected keyword arguments?
-multiple keyword arguments?
-any duplication between positional and keyword arguments?

etc.

Surely there's some easy way of making use of Python's internal logic
here? For some reason, I can't see how. Can anybody see a way?


John
 
M

Michele Simionato

Here is a metaclass for uber-lazy user :)

Concretely, at the creation of the class it takes the source of the
__init__ function and add, at the first line of __init__, the line
that sets the attributes :

Unfortunately this approach does not work if the source is not available
(this happens when you are in the interpreter, or when you only have a .pyc
file). I was playing this kind of tricks some time ago, then I decided that
it was best to switch to Lisp/Scheme for this kind of stuff ;)


Michele Simionato
 
S

Sebastien de Menten

Here is a metaclass for uber-lazy user :)
Unfortunately this approach does not work if the source is not available
(this happens when you are in the interpreter, or when you only have a .pyc
file). I was playing this kind of tricks some time ago, then I decided that
it was best to switch to Lisp/Scheme for this kind of stuff ;)


Michele Simionato

I agree that it is quite low-level. However, 1) I think people are
"very" rarely creationg new classes directly in the interpreter (it's
a pain!) 2) I don't get the point about the .pyc (the metaclass is in
the pyc ? the class is in the pyc ?)
About the List/scheme option, it would be definitely way easier to do
it with macros in those languages but, hey, if i'm working with
python, i need a solutioon in python, don't I ?

Seb
 
M

Michele Simionato

2) I don't get the point about the .pyc

inspect.getsource look at the .py file; in some situations (i.e. trying to
obfuscate code) you may want to ship the .pyc file only; so inspect.getsource
cannot work (unless you decompile the .pyc file and restore .py).

For instance try

$ cat example.py
import inspect,sys

print inspect.getsource(sys.modules["__main__" ])

run it and then remove example.py.

Also, the approach breaks down if the code is executed dynamically;
for instance C-c C-c in emacs would not work.


Michele Simionato
 
P

Peter Otten

Michele said:
(e-mail address removed) (Sebastien de Menten) wrote in message


Unfortunately this approach does not work if the source is not available
(this happens when you are in the interpreter, or when you only have a
.pyc file). I was playing this kind of tricks some time ago, then I
decided that it was best to switch to Lisp/Scheme for this kind of stuff
;)

Here's a variant that operates on the byte code:

import opcode

class List(list):
def ensure(self, value):
try:
return self.index(value)
except ValueError:
self.append(value)
return len(self)-1

class Recorder(object):
def __init__(self, code):
self.func_code = code
self._code = map(ord, code.co_code)[:-4]
self._names = List(code.co_names)

def __getattr__(self, name):
opc = opcode.opmap[name.upper()]
def record(self, arg=None):
# XXX limit name resolution/addition to the proper opcodes
if isinstance(arg, str):
arg = self._names.ensure(arg)
self._code.append(opc)
if arg is not None:
self._code.append(arg & 0xff)
self._code.append(arg >> 8)
setattr(self.__class__, name, record)
return getattr(self, name)

def code(self):
return ''.join(map(chr, self._code))
def names(self):
return tuple(self._names)

def autoinit(f):
co = f.func_code

r = Recorder(co)
for i in range(1, co.co_argcount):
r.load_fast(i)
r.load_fast(0) # self
r.store_attr(co.co_varnames)
r.load_const(0) # None
r.return_value()

new_names = r.names()
new_code = r.code()

codeobj = type(co)(co.co_argcount, co.co_nlocals, co.co_stacksize,
co.co_flags, new_code, co.co_consts, new_names,
co.co_varnames, co.co_filename, co.co_name,
co.co_firstlineno, co.co_lnotab, co.co_freevars,
co.co_cellvars)
return type(f)(codeobj, f.func_globals, f.func_name, f.func_defaults,
f.func_closure)

class AutoInit(type):
def __new__(cls, classname, bases, classdict):
classdict["__init__"] = autoinit(classdict["__init__"])
return type.__new__(cls, classname, bases, classdict)

class Demo(object):
__metaclass__ = AutoInit
def __init__(self, baz, top, foo=3, r=None):
if r is None:
r = ["default"]
foo *= 2
baz *= 3
helper = 42 #ignored

def __str__(self):
return ("Demo(baz=%(baz)r, top=%(top)r, foo=%(foo)r, r=%(r)r)"
% self.__dict__)


if __name__ == "__main__":
print Demo(1, 2)
print Demo(10, 20, 30, r=["other"])
print Demo(100, foo="other", top=200)

I guess that was just a complicated way to fail the sanity check :)

Peter
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top