Late initialization using __getattribute__

[Bruno Desthuilliers]

bukzor a écrit :

I want to make a MixIn class that waits to initialize its super-
classes until an attribute of the object is accessed. Not generally
useful, but desirable in my case. I've written this, and it works, but
would like to take any suggestions you guys have.

You shouldn't mess with __getattribute__ unless you really know what
you're doing and are ok to suffer the constant performance hit you'll
get. Remember that __getattribute__ actually *is* the implementation of
attribute lookup rules, and is called on each and every attribute
lookup. Your below snippet would be much better using __getattr__ (which
is only called as a last resort).

I've commented out
the "delattr" call because it throws an AttributeError (although I
don't know why).

__getattribute__ belongs to the class, not to the instance. delattr()
only removes instance attributes. You'd have to remove __getattribute__
from the LateInitMixIn class object itself, but then it would break the
whole thing.

class LateInitMixIn(object):
def __init__(self):
print "LateInit initialization"
self.inited = False
def __getattribute__(self, attr):
print "Doing __getattribute__"
getattr = lambda attrbject.__getattribute__(self, attr)
if not getattr("inited"):
super(LateInitMixIn, self).__init__()
setattr(self, "inited", True)
#delattr(self, "__getattribute__")
return getattr(attr)

Here's another possible implementation (which doesn't solve all
problems, cf below) using __getattr__:

class LateInitMixin(object):
def __init__(self):
print "not yet"
self.__initialized = False


def __getattr__(self, name):
if self.__initialized:
raise AttributeError(
"object %s has no attribute '%s'" % (type(self), name)
)
super(LateInitMixin, self).__init__()
self.__initialized = True
return getattr(self, name)

class Base(object):
def __init__(self):
print "yet"
self.base = True

class LateInit(LateInitMixin, Base):
pass

def main():
print "shouldn't init"
S = LateInit()
print "should init"
print S.base

if __name__=="__main__":
main()



Ok, now, the other problem : what if Base.__init__ expects args ?

 

[Bruno Desthuilliers]

bukzor a écrit :
(snip)

Thanks for the reply. Just to see it not work, I tried to remove
__getattribute__ from LateInitMixIn, but couldn't get it to work.

??? Sorry, I don't get what you mean...

My Base class is a C class (_mysql.connection from MySQLdb) that
sometimes segfaults if you try to use it before it's fully
initialized,

Then don't use it before it's fully initialized.

patient: "doctor, when I do this, it hurts"
doctor : "then don't do it"

!-)

More seriously, I have used MySQLdb for years on more than a dozen linux
distribs, and never had such a problem. Is this a known bug ? Or is
there something wrong with your setup ?

so unfortunately I think I need to use __getattribute__
to do this. I'm doing all this just to make the connection not
actually connect until used.

I may be dumb, but I don't get how this is supposed to solve your
problem. But anyway : there's a known design pattern for what you're
trying to do, that doesn't require mixins nor messing with
__getattribute__ (which, I repeat, is more often than not something you
*don't* want to do). The name of the design pattern is "proxy". I
strongly suggest that you 1/ try to cure the real problem instead of
hacking around and 2/ read about the proxy design pattern.


My 2 cents...

 

[bukzor]

I want to make a MixIn class that waits to initialize its super-
classes until an attribute of the object is accessed. Not generally
useful, but desirable in my case. I've written this, and it works, but
would like to take any suggestions you guys have. I've commented out
the "delattr" call because it throws an AttributeError (although I
don't know why).



class LateInitMixIn(object):
def __init__(self):
print "LateInit initialization"
self.inited = False
def __getattribute__(self, attr):
print "Doing __getattribute__"
getattr = lambda attrbject.__getattribute__(self, attr)
if not getattr("inited"):
super(LateInitMixIn, self).__init__()
setattr(self, "inited", True)
#delattr(self, "__getattribute__")
return getattr(attr)



class Base(object):
def __init__(self):
print "Base initialization"
self.base = True
class LateInit(LateInitMixIn, Base): pass

def main():
S = LateInit()
print S
print
print "Should do Base init after now"
print S.base
print S.base

if __name__=="__main__": main()




This gives the following output:
LateInit initialization
<__main__.LateInit object at 0x2a960c1c50>

Should do Base init after now
Doing __getattribute__
Base initialization
True
Doing __getattribute__
True

 

[bukzor]

bukzor a écrit :


You shouldn't mess with __getattribute__ unless you really know what
you're doing and are ok to suffer the constant performance hit you'll
get. Remember that __getattribute__ actually *is* the implementation of
attribute lookup rules, and is called on each and every attribute
lookup. Your below snippet would be much better using __getattr__ (which
is only called as a last resort).


__getattribute__ belongs to the class, not to the instance. delattr()
only removes instance attributes. You'd have to remove __getattribute__
from the LateInitMixIn class object itself, but then it would break the
whole thing.




Here's another possible implementation (which doesn't solve all
problems, cf below) using __getattr__:

class LateInitMixin(object):
     def __init__(self):
         print "not yet"
         self.__initialized = False

     def __getattr__(self, name):
         if self.__initialized:
             raise AttributeError(
                 "object %s has no attribute '%s'" % (type(self), name)
                 )
         super(LateInitMixin, self).__init__()
         self.__initialized = True
         return getattr(self, name)

class Base(object):
     def __init__(self):
         print "yet"
         self.base = True

class LateInit(LateInitMixin, Base):
     pass

def main():
     print "shouldn't init"
     S = LateInit()
     print "should init"
     print S.base

if __name__=="__main__":
     main()

Ok, now, the other problem : what if Base.__init__ expects args ?

Thanks for the reply. Just to see it not work, I tried to remove
__getattribute__ from LateInitMixIn, but couldn't get it to work.

My Base class is a C class (_mysql.connection from MySQLdb) that
sometimes segfaults if you try to use it before it's fully
initialized, so unfortunately I think I need to use __getattribute__
to do this. I'm doing all this just to make the connection not
actually connect until used.

 

[bukzor]

bukzor a écrit :
(snip)


??? Sorry, I don't get what you mean...

Since you said __getattribute__ is an attribute of the class, I tried
to run (something to the effect of) delattr(self.__class__,
"__getattribute__"), but it threw an AttributeError.

... I have used MySQLdb for years on more than a dozen linux
distribs, and never had such a problem. Is this a known bug ? Or is
there something wrong with your setup ?

I'm trying to optimize my number of connections by not fully
initializing (read: not connecting) my connection until it's used in
some way. Of course the maintainer didn't envision this (mis)use, so
the object sometimes throws bad errors until it's fully initialized.

Of course the *correct* way to do this is to be more careful about
when I create connections, but I think I should be able to get this to
work, and it (would be) much easier to do it The Right Way once it
works.

I may be dumb, but I don't get how this is supposed to solve your
problem. But anyway : there's a known design pattern for what you're
trying to do, that doesn't require mixins nor messing with
__getattribute__ (which, I repeat, is more often than not something you
*don't* want to do). The name of the design pattern is "proxy". I
strongly suggest that you 1/ try to cure the real problem instead of
hacking around and 2/ read about the proxy design pattern.

My 2 cents...

I like the idea of mix-ins, but can't figure out how to make a proxy
work that way. For a long time I had a proxy class that added five or
six features on top of the MySQLdb package, but it wasn't configurable
enough, and I'm working on splitting each feature into its own MixIn
class.




As an aside, this is working for me pretty well. The "reconnect"
method (inheritied from a "Reconnectable" mixin) uses several of the
object's attributes, so I need to set _inited beforehand so that I
don't get into an infinite __getattribute__ loop. What I'd *really*
like to do is remove __getattribute__ from the object at that point.
def __getattribute__(self, attr):
"connect if it would otherwise cause an error."
getattr = lambda attrbject.__getattribute__(self, attr)

if not getattr("_inited"):
print "connecting."
setattr(self, "_inited", True)
getattr("reconnect")()
return getattr(attr)


Thanks for your help,
--Buck

 

[Michele Simionato]

I'm trying to optimize my number of connections by not fully
initializing (read: not connecting) my connection until it's used in
some way.

I had the same use case and I solved with a simple property. Here is
the code
I have for pymssql:

@property
def conn(self):
if self._conn is None:
self._conn = _mssql.connect(self.host, self.user,self.passwd)

self._conn.select_db(self.dbname)
return self._conn

The connection is really instantiate only when you call self.conn to
perform the query, not when you instantiate the class.

 

[Bruno Desthuilliers]

bukzor a écrit :

Since you said __getattribute__ is an attribute of the class, I tried
to run (something to the effect of) delattr(self.__class__,
"__getattribute__"),

Ah, ok. Definitively *not* a thing to do...

but it threw an AttributeError.

You'd have to call it on the mixin class itself, not on a subclass.

I'm trying to optimize my number of connections

Connection pooling anyone ? IIRC, this is something that already exists
in quite a couple libs / frameworks. Is there a reason you try to roll
your own ?

by not fully
initializing (read: not connecting) my connection until it's used in
some way. Of course the maintainer didn't envision this (mis)use, so
the object sometimes throws bad errors until it's fully initialized.
Ok.

Of course the *correct* way to do this is to be more careful about
when I create connections, but I think I should be able to get this to
work, and it (would be) much easier to do it The Right Way once it
works.

Please pardon me for repeating the same thing over and over, but messing
with __getattribute__ is certainly not the way to go.

I like the idea of mix-ins, but can't figure out how to make a proxy
work that way.

You mean, "how to use a proxy for lazy initialization" ? Heck, that's
the exact use case in the GoF.

For a long time I had a proxy class that added five or
six features on top of the MySQLdb package, but it wasn't configurable
enough, and I'm working on splitting each feature into its own MixIn
class.


As an aside, this is working for me pretty well. The "reconnect"
method (inheritied from a "Reconnectable" mixin) uses several of the
object's attributes, so I need to set _inited beforehand so that I
don't get into an infinite __getattribute__ loop. What I'd *really*
like to do is remove __getattribute__ from the object at that point.

You can't. Or, more exactly, all you can do is remove __getattribute__
from the mixin class - but then the mixin class won't work anymore. I
don't mean to be condescendant, but it looks like you don't have a clear
understanding of Python's object model here - else you wouldn't even
consider doing such a thing. FWIW, I posted a solution based on the
__getattr__ hook, which did work - at least for the "specs" implied by
your code snippet.

 

[bukzor]

You mean, "how to use a proxy for lazy initialization" ? Heck, that's
the exact use case in the GoF.

I mean, "how to make a MixIn class that uses the proxy pattern". I'd
like to be able to do something like this:

class SuperCursor(FeatureOneMixIn, FeatureTwoMixin, ...,
VanillaCursor): pass

This works with my current implementation. After thinking about it
more, I think I've got such a thing written. I had to use
inspect.getmro and new.classobj to do it, but it works and it honors
the usual mro (as far as I can tell). I've put the code at the bottom
to (try to) maintain readability.

You can't. Or, more exactly, all you can do is remove __getattribute__
from the mixin class - but then the mixin class won't work anymore. I
don't mean to be condescendant, but it looks like you don't have a clear
understanding of Python's object model here - else you wouldn't even
consider doing such a thing. FWIW, I posted a solution based on the
__getattr__ hook, which did work - at least for the "specs" implied by
your code snippet.

My toy example turned out to be not the best representation of the
problem.
The base class has attributes that "exist" but either throw errors or
segfault
if used before reconnect() is called. This means that I need to
capture more than
just the attributes that would throw AttributeError.




#CODE########################################
class Base(object):
def __init__(self, *args, **kwargs):
self.args = args
self.kwargs = kwargs
def __str__(self): return "<Base object created with %s %s>" %
(self.args, self.kwargs)

class MixIn2(object):
def __str__(self):
return "<MixIn2 with %s>" % super(MixIn2, self).__str__()

class MixIn1(object):
def __str__(self):
return "<MixIn1 with %s>" % super(MixIn1, self).__str__()

class ProxyMixIn(object):
def __init__(self, *args, **kwargs):
self.__proxied = None
self.__args = args
self.__kwargs = kwargs
def __getattr__(self, attr):
print "Getting", attr
try: return getattr(self.__proxied, attr)
except AttributeError:
if self.__proxied: raise
else:
from inspect import getmro
mro = getmro(self.__class__)
mro = mro[list(mro).index(ProxyMixIn) + 1:]
print "Proxied mro", mro
from new import classobj
self.__proxied = classobj("Proxied", mro, globals())
(*self.__args, **self.__kwargs)
return getattr(self.__proxied, attr)
def __str__(self):
return "<Proxy of %s>" % super(ProxyMixIn, self).__str__()

class Proxy(MixIn1, ProxyMixIn, MixIn2, Base): pass

def main():
p = Proxy(1,2,3, one=1, two=2)
print p
main()

#OUTPUT######################################
Getting args
Proxied mro (<class '__main__.MixIn2'>, <class '__main__.Base'>, <type
'object'>)
Getting kwargs
<MixIn1 with <Proxy of <MixIn2 with <Base object created with (1, 2,
3) {'two': 2, 'one': 1}>>>>

 

[Bruno Desthuilliers]

bukzor a écrit :

I mean, "how to make a MixIn class that uses the proxy pattern".

That's not how proxies work. They use composition/delegation, not MI.

I'd
like to be able to do something like this:

class SuperCursor(FeatureOneMixIn, FeatureTwoMixin, ...,
VanillaCursor): pass

Yuck.

Sorry, but it reminds me of the worst MI abuses in Zope2.

(snip)

My toy example turned out to be not the best representation of the
problem.

This is often the case with toy examples.

The base class has attributes that "exist" but either throw errors or
segfault
if used before reconnect() is called.

Then these attributes should probably call reconnect() by themselves.
Decorators and/or computed attributes may help here.

 

[Carl Banks]

I mean, "how to make a MixIn class that uses the proxy pattern".

You don't. That's not how proxies work.

I'd like to be able to do something like this:

class SuperCursor(FeatureOneMixIn, FeatureTwoMixin, ...,
VanillaCursor): pass

Why does it have to look like that? A good programmer lets the code
look however it has to look to most effectively do it's job.

With a proxy, the "base class" isn't a base class but a member. Here
is a very simple example:

class SuperCursor(object):
def __init__(self):
self._cursor = VanillaCursor()
self._connected = False
def __getattr__(self,attr):
if not self._connected:
self._cursor.connect()
self._connected = True
return getattr(self._cursor,attr)

cursor = SuperCursor()

That doens't use a mixin, but why should it?



Carl Banks

 

[bukzor]

I'd like to be able to do something like this:

Why does it have to look like that?  A good programmer lets the code
look however it has to look to most effectively do it's job.

With a proxy, the "base class" isn't a base class but a member.  Here
is a very simple example:

class SuperCursor(object):
    def __init__(self):
        self._cursor = VanillaCursor()
        self._connected = False
    def __getattr__(self,attr):
        if not self._connected:
            self._cursor.connect()
            self._connected = True
        return getattr(self._cursor,attr)

cursor = SuperCursor()

That doesn't use a mixin, but why should it?

The point of using a mixin is to not limit myself to inheriting from
VanillaCursor. I want to put this on top of various subclasses of the
vanilla cursor, like TimeLimitedCursor or RetryingCursor. I have four
other mixins that operate this way, so it's desirable to keep this one
in line with that.

 

[Carl Banks]

The point of using a mixin is to not limit myself to inheriting from
VanillaCursor. I want to put this on top of various subclasses of the
vanilla cursor, like TimeLimitedCursor or RetryingCursor. I have four
other mixins that operate this way, so it's desirable to keep this one
in line with that.

http://www.bartleby.com/59/3/foolishconsi.html

I think that desire is hurting you more than it's helping. It's fine
to be consistent for consistency's sake, but you are taking
consistency to an unhealthy extreme. A mixin is simply the wrong tool
to do this with.

My advice: either use a proxy, or manage your connections better.


Carl Banks