general class functions

[syd]

In my project, I've got dozens of similar classes with hundreds of
description variables in each. In my illustrative example below, I
have a Library class that contains a list of Nation classes. In one
case, I might want a Library class with only Nations of with the
continent variable "Europe", and so I'll do something like
library.getContinent('Europe') which will return a Library() instance
with only European nations. Similarly, I could also want a Library()
instance with only 'small' size nations.

My question follows: is there a way I can make a general function for
getting a subclass with my specific criteria? For instance, can I
pass in an equivalence test or something like that? I have hundreds
of variables like "nation", "continent", and so forth, and do want to
write a separate "get______" function for each.

This stumped everyone I work with. Help would be greatly
appreciated!!

#!/usr/bin/python
class Library:
def __init__(self,list=None):
self.list=list
def add(self,nation):
try: self.list.append(nation)
except: self.list=[nation]
def defineNation(self,name,continent,size):
nation=Library.Nation()
nation.name=name
nation.continent=continent
nation.size=size
self.add(nation)
def getContinent(self,continent):
library=Library()
for nation in self.list:
if nation.isContinent(continent)==True:
library.add(nation)
return library
def getSize(self,size):
library=Library()
for nation in self.list:
if nation.isSize(size)==True:
library.add(nation)
return library
def getNameList(self):
outList=[]
for nation in self.list:
outList.append(nation.name)
return outList
class Nation:
def __init__(self,name=None,continent=None,size=None):
self.name=name # eg, Spain
self.continent=continent # eg, Europe
self.size=size # eg, medium
def isContinent(self,continent):
if self.continent==continent: return True
else: return False def isSize(self,size):
if self.size==size: return True
else: return False

library=Library()
library.defineNation('Spain','Europe','medium')
library.defineNation('USA','NorthAmerica','large')
library.defineNation('Luxembourg','Europe','small')
library.defineNation('Portugal','Europe','medium')

print library.getNameList()
print library.getContinent('Europe').getNameList()
print library.getContinent('Europe').getSize('small').getNameList()

 

[Kent Johnson]

In my project, I've got dozens of similar classes with hundreds of
description variables in each. In my illustrative example below, I
have a Library class that contains a list of Nation classes. In one
case, I might want a Library class with only Nations of with the
continent variable "Europe", and so I'll do something like
library.getContinent('Europe') which will return a Library() instance
with only European nations. Similarly, I could also want a Library()
instance with only 'small' size nations.

My question follows: is there a way I can make a general function for
getting a subclass with my specific criteria? For instance, can I
pass in an equivalence test or something like that? I have hundreds
of variables like "nation", "continent", and so forth, and do want to
write a separate "get______" function for each.

You can use getattr to read the Nation attributes by name. Add this
method to Library:
def getMatches(self,attribute, value):
library=Library()
for nation in self.list:
if getattr(nation, attribute) == value:
library.add(nation)
return library

Now you can call
print library.getMatches('continent', 'Europe').getNameList()
print library.getMatches('continent', 'Europe').getMatches('size',
'small').getNameList()

Is that what you want?
Kent

This stumped everyone I work with. Help would be greatly
appreciated!!

#!/usr/bin/python
class Library:
def __init__(self,list=None):
self.list=list
def add(self,nation):
try: self.list.append(nation)
except: self.list=[nation]
def defineNation(self,name,continent,size):
nation=Library.Nation()
nation.name=name
nation.continent=continent
nation.size=size
self.add(nation)
def getContinent(self,continent):
library=Library()
for nation in self.list:
if nation.isContinent(continent)==True:
library.add(nation)
return library
def getSize(self,size):
library=Library()
for nation in self.list:
if nation.isSize(size)==True:
library.add(nation)
return library
def getNameList(self):
outList=[]
for nation in self.list:
outList.append(nation.name)
return outList
class Nation:
def __init__(self,name=None,continent=None,size=None):
self.name=name # eg, Spain
self.continent=continent # eg, Europe
self.size=size # eg, medium
def isContinent(self,continent):
if self.continent==continent: return True
else: return False def isSize(self,size):
if self.size==size: return True
else: return False

library=Library()
library.defineNation('Spain','Europe','medium')
library.defineNation('USA','NorthAmerica','large')
library.defineNation('Luxembourg','Europe','small')
library.defineNation('Portugal','Europe','medium')

print library.getNameList()
print library.getContinent('Europe').getNameList()
print library.getContinent('Europe').getSize('small').getNameList()

 

[Andrea Griffini]

My question follows: is there a way I can make a general function for
getting a subclass with my specific criteria? For instance, can I
pass in an equivalence test or something like that? I have hundreds
of variables like "nation", "continent", and so forth, and do want to
write a separate "get______" function for each.

Adding the following method to Library

def __getattr__(self,name):
if name.startswith("get_"):
attrname = name[4:]
def filter(value):
library=Library()
for nation in self.list:
if getattr(nation,attrname) == value:
library.add(nation)
return library
return filter
else:
return self.__dict__[name]

you will be able to write

L.get_continent('Europe').get_size('small')

because for anything starting with "get_..." will be
generated on the fly a filtering function that looks
for exact match of an attribute. No need for
getContinent, getSize, isContinent and isSize.

HTH
Andrea

 

[syd]

Many many thanks, Andrea and Kent!

To keep changes to calls in my current code to a minimum, I'm hoping I
can encode Andrea's "on-the-fly" get function. I added Andrea's
__getattr__ to my Library() class, populated the class with a few
nations, and tried:
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "foo.py", line 16, in __getattr__
return self.__dict__[name]
KeyError: '__repr__'

I then commented out lines 15 and 16...
#else:
#return self.__dict__[name]

.... and re-ran.
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: 'NoneType' object is not callable

I tweaked a bit, but I could not get to the bottom of this problem. I
see the logic here (and it's brilliant I should add, Andrea), but I'm
not sure what is wrong.

Ideas?

 

[Kent Johnson]

Many many thanks, Andrea and Kent!

To keep changes to calls in my current code to a minimum, I'm hoping I
can encode Andrea's "on-the-fly" get function. I added Andrea's
__getattr__ to my Library() class, populated the class with a few
nations, and tried:


Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "foo.py", line 16, in __getattr__
return self.__dict__[name]
KeyError: '__repr__'

Instead of
return self.__dict__[name]
you need
raise AttributeError, name

self.__dict__ has already been tried before __getattr__ is called, you
don't need to try it again.

Kent

 

[Dan Perl]

Many many thanks, Andrea and Kent!

To keep changes to calls in my current code to a minimum, I'm hoping I
can encode Andrea's "on-the-fly" get function. I added Andrea's
__getattr__ to my Library() class, populated the class with a few
nations, and tried:

This line creates and returns an instance of Library. I assume that you
imported foo in a python session and then you executed that line. That
results in a call to Library.__repr__( ), which is not defined. First of
all, I recommend you change class Library to subclass from object. That's
just good practice in general and in this case it will give you a default
__repr__( ) method. But that's probably not what you want. What you
probably want is to still use a line like:
print library.getContinent('Europe').getNameList()
Otherwise, library.get_continent('Europe') is going to print something like:
<__main__.Library object at 0x00A9F430>

Dan

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "foo.py", line 16, in __getattr__
return self.__dict__[name]
KeyError: '__repr__'

I then commented out lines 15 and 16...
#else:
#return self.__dict__[name]

... and re-ran.
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: 'NoneType' object is not callable

I tweaked a bit, but I could not get to the bottom of this problem. I
see the logic here (and it's brilliant I should add, Andrea), but I'm
not sure what is wrong.

Ideas?

 

[bruno modulix]

Kent Johnson a écrit :

You can use getattr to read the Nation attributes by name. Add this
method to Library:
def getMatches(self,attribute, value):
library=Library()
for nation in self.list:
if getattr(nation, attribute) == value:
library.add(nation)
return library

Now you can call
print library.getMatches('continent', 'Europe').getNameList()
print library.getMatches('continent', 'Europe').getMatches('size',
'small').getNameList()

<just for the fun...>

You could make it even more simple - and avoid creating useless
instances of Library and Nation - with keyword arguments:

class Library:
[...]
def getAll(self, **kwargs):
library=Library()
for nation in self.list:
keep = True
for attribute, value in kwargs.items():
if not getattr(nation, attribute) == value:
keep = false
break
if keep :
library.add(nation)
return library

print library.getAll(continent='Europe').getNameList()
print library.getAll(continent='Europe', size='small').getNameList()

This is and 'and' search, you could also have an 'or' search method:

def getAny(self, **kwargs):
library=Library()
for nation in self.list:
for attribute, value in kwargs.items():
if getattr(nation, attribute) == value:
library.add(nation)
break
return library

print library.getAny(continent='Europe', size='small').getNameList()

and then a more generic search:
def getMatches(self, op, **kwargs):
fun = {'or': self.getAny, 'and' : self.getAll}.get(op)
return fun(**kwargs):

print library.getMatches('and',
continent='Europe',
size='small').getNameList()

print library.getMatches('or',
continent='Europe',
size='small').getNameList()

</just for the fun...>

Bruno

 

[Steven Bethard]

class Library:
[...]
def getAll(self, **kwargs):
library=Library()
for nation in self.list:
keep = True
for attribute, value in kwargs.items():
if not getattr(nation, attribute) == value:
keep = false
break
if keep :
library.add(nation)
return library

I haven't been watching this thread, but I saw a pattern here that Python allows
a much easier solution to. The code:

keep = True
for item in lst:
if test(item):
keep = False
break
if keep:
...

can generally be rewritten as:

for item in lst:
if test(item):
break
else:

For example:
.... keep = True
.... for item in lst:
.... if test(item):
.... keep = False
.... break
.... if keep:
.... print "reached if"
.... .... for item in lst:
.... if test(item):
.... break
.... else:
.... print "reached if"
.... reached if

Always good to take advantage of a language construct when you can. =)

Steve

 

[Andrea Griffini]

Instead of
return self.__dict__[name]
you need
raise AttributeError, name

self.__dict__ has already been tried before __getattr__ is called, you
don't need to try it again.

I hope one day I'll understand python lookup

....

Would it be worse or better to make Library a new-style
class, using __getattribute__ instead of __getattr__ and
defaulting to object.__getattribute__(self,name) in the
"else" part ?

Andrea

 

[syd]

Instead of

return self.__dict__[name]
you need
raise AttributeError, name

Booyah, Kent! This works. For posterity, I'll dump a working code:

#!/usr/bin/python
class Library(object):
def __init__(self,list=None):
self.list=list
def __getattr__(self,name):
if name.startswith("get_"):
attrname = name[4:]
def filter(value):
library=Library()
for nation in self.list:
if getattr(nation,attrname) == value:
library.add(nation)
return library
return filter
else:
raise AttributeError,name
def add(self,nation):
try: self.list.append(nation)
except: self.list=[nation]
def defineNation(self,name,continent,size):
nation=Library.Nation()
nation.name=name
nation.continent=continent
nation.size=size
self.add(nation)
def getNameList(self):
outList=[]
for nation in self.list:
outList.append(nation.name)
return outList
class Nation(object):
def __init__(self,name=None,continent=None,size=None):
self.name=name # eg, Spain
self.continent=continent # eg, Europe
self.size=size # eg, medium

library=Library()
library.defineNation('Spain','Europe','medium')
library.defineNation('USA','NorthAmerica','large')
library.defineNation('Luxembourg','Europe','small')
library.defineNation('Portugal','Europe','medium')

print library.getNameList()
print library.get_continent('Europe').getNameList()

 

[syd]

This line creates and returns an instance of Library. I assume that you
imported foo in a python session and then you executed that line. That
results in a call to Library.__repr__( ), which is not defined. First of
all, I recommend you change class Library to subclass from object. That's
just good practice in general and in this case it will give you a default
__repr__( ) method. But that's probably not what you want. What you
probably want is to still use a line like:
print library.getContinent('Europe').getNameList()
Otherwise, library.get_continent('Europe') is going to print something like:
<__main__.Library object at 0x00A9F430>

You're 100% right, Dan -- I meant to have the "getNameList()" tacked
on.

But more importantly, why is subclassing object a good practice in
general? I see under dir(object) that you get some good default
methods. I looked under the documentation, but could not find a good
specific examples as to why this was a good practice. Can you
elaborate, please?

 

[syd]

Would it be worse or better to make Library a new-style
class, using __getattribute__ instead of __getattr__ and
defaulting to object.__getattribute__(self,name) in the
"else" part ?

Can you elaborate here, please? I'm not sure I understand what you mean.

 

[Dan Perl]

I probably cannot do justice to the topic, although I have personally
encountered cases where by following the advice of people more experienced
in python than me I was able to improve my code. But I can point you to a
couple of links:
http://www.python.org/2.2.1/descrintro.html
http://www.python.org/doc/2.2.1/whatsnew/

In a nutshell, subclassing from object is part of the new-style classes
introduced in python 2.2. Quoting from "Learning Python" 2nd ed., "perhaps
the most visible change in new style classes is their slightly different
treatment of inheritance for the so-called diamond pattern of multiple
inheritance trees-where more than one superclass leads to the same higher
superclass further above". Look also at cooperative methods in the links I
mentioned above (if I remember correctly, cooperative methods have been the
main reason for me to change classes to the new style).

I'm sure other people in this newsgroup can make a better case for new-style
classes than I can.

HTH,

Dan

 

[Dan Perl]

Can you elaborate here, please? I'm not sure I understand what you mean.

__getattr__ existed since "classic" classes. It is only called if the
regular way of searching for the attribute doesn't find it (it's not in
self.__dict__). __getattribute__ was introduced with new style classes and
it is called for *all* attribute references. In your case at least, you can
either implement the __getattr__ hook or override the __getattribute__
method with pretty much the same results. I see a use in overriding
__getattribute__ if you want to change the behavior for all the attributes,
but you are interested only in "get_*" methods.

Again, see http://www.python.org/2.2.1/descrintro.html for more (I was
actually loosely quoting from there in my explanation).

I can't say though whether it would be worse or better. I'm also curious to
hear opinions on that.

Dan

 

[Andrea Griffini]

Can you elaborate here, please? I'm not sure I understand what you mean.

IIUC In python there are two ways of "intercepting"
when someone tries to access a member of an instance
of a class. The first way is defining "__getattr__"
(a method that is called when the normal lookup in
the instance __dict__ failed) and another is defining
"__getattribute__" that is called BEFORE trying any
other lookup (the latter works only for new-style
classes; those that inherit from 'object').

In the first case you need to raise AttributeError
for members you don't want to "emulate" to have the
interpreter continue the lookup; in the second case
you've instead to chain the request up in the
class hierarchy (e.g. to 'object').

Probably as often happens there are pros and cons
with each solution; but I'm quite new to python and
basically virgin of any python internal trick... so
I was asking in the NG for an explanation.

Looks like you're a newbie too; I wouldn't worry ...
probably there are a *lot* of python programmers out
there that never investigated on those implementation
details and still can live happy

Andrea

 

[Alex Martelli]

But more importantly, why is subclassing object a good practice in

"Subclassing object" gives you newstyle classes, instead of the default
classic classes we're stuck with, for backwards compatibility, until a
Python 3.0 comes out that can break such compatibility. Other ways to
make a class newstyle include inheriting from any other newstyle type
(including builtin types) and setting __metaclass__=type .

The newstyle object model fixes many tiny but irritating warts that the
classic object model is stuck with, such as instance dicts always taking
precedence over classes (and thus the inability to use descriptors, e.g.
properties, to full advantage), the classic model's oversimplified
concept of method resolution order in multiple-inheritance situations,
and the intrinsic asymmetry of class->metaclass vs instance->class
semantics in the classic object model.


Alex

 

[syd]

Recap: For a general "container" type class, I wanted to grab another
instance of this class based upon attributes of the items contained
within. Eg, for an illustrative library of nations, I want to be able
to do operations like
library.get_continent('Europe').get_size('medium').

A good sol'n: Use __getattr__ to intercept the "get_continent"
on-the-fly and split it into "get_" (the function I want) and
"continent" (the attribute I want to select for). By using the
built-in getattr(), I can grab the value of this attribute in each
component object in my library. Thus, each "get_foo" does not need to
be hard coded.

Some things I learned: subclass object -- ie, "class Library(object):"
-- for post python-2.2 for a variety of reasons including better
agreement with upcoming versions. Also, __getattr__ intercepts *after*
looking in __dict__, and __getattribute__ (when you've subclassed
object) intercepts *before*.

First off, thanks to everyone for helping me to this point! Looking
forward, I've got two other related issues:
would obviously seem be to subclass each to a general "container". Ie,

class Library_A(Container): ...
class Library_B(Container): ...

The problem: in the current setup, a library_a.get_continent('Europe')
would pass back an instance of Container and *not* Library_A. The
obvious implication is that any attributes/method specific to Library_A
are gone. (In the code at the bottom, we'd lose the "defineNation"
method). My (bad?) remedy: when you create library_a, give it a
"blankSelf" attribute that it can deepcopy before you pass it back.
Within the __getattr__:

def get(value):
container=copy.deepcopy(self.blankSelf)
if self.isNotEmpty()==True:
for item in self.list:
if getattr(item,attribute)==value:
container.add(item)
return container

This hard-code / deepcopy does not seem ideal!
attribute of the component class but rather a method. In our
illustrative example, we could imagine a
"get_nameLongerThanFourLetters" where we add the component nation class
when "nation.is_nameLongerThanFourLetters()==True".

Above, we use getattr() to grab the attribute on-the-fly. Can we grab
a method on-the-fly, too? Ie, when we see "get_foo", use the "is_foo"
criteria from the class below.

I hope this makes sense. In response to Andrea's comments, I'll state
the obvious: I'm fairly new to the intricacies of Python class
structures . I'm a student (non CS), and my year or so of Python
experience before this project has been "functional" (alternatively:
quick hacks). But I love the language! And I want to learn! Thanks
again, everyone.

Appendix: Here's the code for the illustrative example...

#!/usr/bin/python
import copy
class Container(object):
def __init__(self,list=None,blankSelf=None):
self.list=list
self.blankSelf=blankSelf
def __getattr__(self,name):
def get(value):
container=copy.deepcopy(self.blankSelf)
if self.isNotEmpty()==True:
for item in self.list:
if getattr(item,attribute)==value:
container.add(item)
return container
def getMethodAndAttribute():
if name.startswith('get_'):
return 'get',name[4:]
else:
return '',''
method,attribute=getMethodAndAttribute()
if method=='get': return get
else: raise AttributeError,name
def add(self,item):
try: self.list.append(item)
except: self.list=[item]
def count(self):
if type(self.list)==type([]): return len(self.list)
else: return 0
def getNameList(self):
outList=[]
for nation in self.list:
outList.append(nation.name)
return outList
def isNotEmpty(self):
if self.count()>0: return True
else: return False
class Library(Container):
def __init__(self):
pass
def defineNation(self,name,continent,size):
nation=Library.Nation()
nation.name=name
nation.continent=continent
nation.size=size
self.add(nation)
class Nation:
def __init__(self,name=None,continent=None,size=None):
self.name=name # eg, Spain
self.continent=continent # eg, Europe
self.size=size # eg, medium

library=Library()
library.blankSelf=Library()
library.defineNation('Spain','Europe','medium')
library.defineNation('USA','NorthAmerica','large')
library.defineNation('Luxembourg','Europe','small')
library.defineNation('Portugal','Europe','medium')

print library.getNameList()
print library.get_continent('Europe').getNameList()

 

[Jeff Shannon]

would obviously seem be to subclass each to a general "container". Ie,

class Library_A(Container): ...
class Library_B(Container): ...

The problem: in the current setup, a library_a.get_continent('Europe')
would pass back an instance of Container and *not* Library_A. The
obvious implication is that any attributes/method specific to Library_A
are gone.

As I understand it, your object's (most derived) type will be stored as
self.__class__. So, in order to have Library_A create more Library_A
instances, and Library_B create more Library_B instances, you can
instead do something like:

new_copy = self.__class__()

2) I've got a bunch of "get_foo" type things where "foo" is not an


attribute of the component class but rather a method. [...]

Above, we use getattr() to grab the attribute on-the-fly. Can we grab
a method on-the-fly, too?

Sure -- methods are just attributes that happen to be callable. You get
a method reference in the same way that you get any attribute, and you
call it the same way that you call any function reference:

method = Collection.get_foo('bar')
method()

will call whatever method get_foo('bar') returns.

Jeff Shannon
Technician/Programmer
Credit International

 

[syd]

new_copy = self.__class__()

Awesome! I didn't know I could call it like that... thanks, Jeff.

method = Collection.get_foo('bar')
method()

I suppose I knew that, but I wasn't thinking. I can do something like
(for nation as a component of library):

(criteria='foo' or 'bar' or whatever... grabbed from __getattr__)
isMethodString='is'+criteria
isMethod=getattr(library,isMethodString)

then the criteria for inclusion becomes when isMethod()==True

----------------------------------------------

One final question that I can't seem to get, but I know I should be
able to do:

Given a Library class with possible component class types Nation_A,
Nation_B, and so forth where type(Nation_A)<>type(Nation_B). If I had
the string 'Nation_B', how could I get an empty class Nation_B?

ie,
dir()=['Library','Nation_A','Nation_B',...]
desiredEmptyClassString=dir[2]
desiredEmptyClass=getEmptyClassFromString(desiredEmptyClassString)
Does a "getEmptyClassFromString" method exist?

 

[Jeff Shannon]

One final question that I can't seem to get, but I know I should be
able to do:

Given a Library class with possible component class types Nation_A,
Nation_B, and so forth where type(Nation_A)<>type(Nation_B). If I had
the string 'Nation_B', how could I get an empty class Nation_B?

ie,
dir()=['Library','Nation_A','Nation_B',...]
desiredEmptyClassString=dir[2]
desiredEmptyClass=getEmptyClassFromString(desiredEmptyClassString)
Does a "getEmptyClassFromString" method exist?

Not directly, but it can be faked.

It's easiest if you're getting the class from an imported module.
Remember that a module is just another object, and a class object is
just another callable.

import MyLibrary

desired_class = getattr(MyLibrary, desired_empty_class_string)
myobject = desired_class()

It's a bit trickier if the class is defined in the current module, but
the principle is similar. You just need a way to look up an attribute
in the *current* module's namespace, rather than in another module's
namespace. As it turns out, the globals() built-in function will return
the current global namespace, which is actually a dictionary.

desired_class = globals()[desired_empty_class_string]
myobject = desired_class()

Personally, I feel nervous any time I have code that uses something like
globals() -- I feel like I'm starting to poke at internals. It should
be completely safe, especially using it in a read-only way like this,
but just out of inherent cautiousness I prefer to avoid it when I can.
So, I'd be likely to put all of the interesting class objects into my
own dict, and call them from there.

LibraryFactory = { 'Library': Library, 'Nation_A':Nation_A, ... }

myobject = LibraryFactory[desired_empty_class_string]()

Note that, if you really are selecting the string via an integer index
into a list, you could instead key this dictionary off of the integers:

LibraryFactory = {0:Library, 1:Nation_A, 2:Nation_B, ...}
myobject = LibraryFactory[index]()

Using a dict like this makes me feel a bit more comfortable than using
globals() (though I realize that this is not necessarily a matter of
rational reasoning), and it also seems to me to be a bit more
encapsulated (relevant items are specifically gathered in one particular
place, rather than just being scattered about the global namespace and
picked up as needed). But using globals() is a perfectly viable option
as well.

Jeff Shannon
Technician/Programmer
Credit International