Problem--Extending the behavior of an upstream package

[Carl Banks]

Ok, I think I know I want to do this, but I thought I'd run it by yins
guys

I have a fairly large and complex application with two top-level
packages, which we'll call upstream and mine. The "upstream" package
isn't really upstream. I own both packages, but I think of the
upstream package as a third-party library relative to the mine
package. The mine package contains code specific to the application.

Anyway, the upstream modules provide lots of classes and other
functionality, but I want to extend their behavior in various ways.
Say for instance I have a class called Box defined in the module
upstream.packaging. In this particular application, it makes sense
for all boxes to have a certain behavior, such as a certain way of
being drawn. So I define a module called mine.custom_packaging which
defines a subclass of upstream.packaging.Box. That's straightforward
enough.

The problem comes when a different part of the upstream package also
subclasses or creates a Box. When an upstream function creates a box,
it creates an upstream.packaging.Box instead of a
mine.custom_packaging.Box, but I'd want it to do the latter.

If this were something that happened only once or twice, it'd be no
big deal, I'd just work around it. However, it's very common. The
upstream package is almost a complete application unto itself; it only
needs data and a small amount of top-level code to run. The sort of
tight internal coupling that exists within the upstream package makes
it a logistical problem to extend it. And the thing is, I can't think
of a convenient way to do it that's tolerably magical, and I am quite
tolerant of homebrew magic in my code. A particularly troublesome
situation is when a base class and subclass are defined in the same
module, and I want to customize both. Even things like disambiguating
nomenclature trip me up.

It isn't just classes, BTW. Sometimes I want to override functions,
or modify permanent data.

So, how would you approach something like this?

(After a little discussion--or not--I'll post some things I've
considered, including a few approaches I've actually implemented. But
I don't want to taint everyone's ideas just yet.)


Carl Banks

 

[Aahz]

The problem comes when a different part of the upstream package also
subclasses or creates a Box. When an upstream function creates a box,
it creates an upstream.packaging.Box instead of a
mine.custom_packaging.Box, but I'd want it to do the latter.

The only clean way I'm aware of is to pass in an instance and use
instance.__class__ to create new instances. Did you come up with
another idea?

 

[Carl Banks]

The only clean way I'm aware of is to pass in an instance and use
instance.__class__ to create new instances.  Did you come up with
another idea?

Ok, so after lots of false starts and muttering, and much mindbending
to get my head around the import dynamics, I came up with this simple
(not straightforward) recipe, which does not require an import hook,
but has one major drawback (that I can live with). I defined this
function:


def adapt(base_full_mod_name):
mp = base_full_mod_name.split('.')
if len(mp) < 2:
raise ValueError('can only adapt packaged modules')
adapt_full_mod_name = '.'.join(["custom"] + mp[1:])
keepref_full_mod_name = '.'.join(["_upstream"] + mp[1:])
adapt_mod_name = mp[-1]
parent_full_mod_name = '.'.join(mp[:-1])
try:
mod = __import__(adapt_full_mod_name,globals(),locals(),
('*',))
except ImportError:
pass
else:
sys.modules[keepref_full_mod_name] = sys.modules
[base_full_mod_name]
sys.modules[base_full_mod_name] = mod
setattr(sys.modules[parent_full_mod_name],adapt_mod_name,mod)


Most of the modules in package "upstream" have a little bit of
boilerplate at the very end that looks something like this:

from . import adaptation
adaptation.adapt(__name__)


Believe it or not, this causes whoever imports the module to actually
receive a different module. It exploits the fact that the semantics
of __import__ are, for the lack of a better word, retarded. If you
write "from upstream import staging", staging would end up bound to
the module custom.stating (if custom.staging exists).

This allows you to subclass upsteam.staging.Box naturally, and anyone
who tries to use upstream.staging.Box will actually be using
custom.staging.Box. (Actually not quite true, a user in the same
module would still be using Box, but that's easily work-around-able.)

The major drawback is that "from upstream.staging import Box" will
fail. It will import upstream.staging.Box, not custom.stating.Box" if
it's the first time upstream.stating is imported. I can live with it
since I expect the modules to be carefully imported in order before
anyone uses their contents.

A minor drawback is it only works for modules in a package, no
biggie. It probably crashes and burns in circular import situations,
and there's almost certainly some other pitfalls.


It's probably too magical for most people, but as I said I can
tolerate a lot of magic. I think it's ok because all adaptible
modules have boilerplate at the end cluing a user that some other
behavior might be tacked onto this module. That was the key sticking
point for me, I could have used an import hook, but I REALLY didn't
want a user to see "import upstream.staging" and not have any clue why
they were importing custom.staging.


Carl Banks