Overriding variables used by base classes

[Corey Lubin]

[original.py]
someGlobal=1

class Original:
def foo(self):
# Make use of someGlobal
[/original.py]

[tainted.py]
from original import *
someGlobal=2

class Tainted(Original):
pass
[/tainted.py]

How might one make it such that Tainted.foo makes use of
tainted.someGlobal, rather than original.someGlobal? Must I use
eval/exec? Note that the point is that I don't want to have to
duplicate the definition of foo.

Corey

 

[Ben Finney]

from original import *
someGlobal=2

Your use of "global" and "from foo import *" belie an approach counter
to encapsulation and modularity. What is it you're trying to achieve?

 

[Duncan Booth]

(e-mail address removed) (Corey Lubin) wrote in

[original.py]
someGlobal=1

class Original:
def foo(self):
# Make use of someGlobal
[/original.py]

[tainted.py]
from original import *
someGlobal=2

class Tainted(Original):
pass
[/tainted.py]

How might one make it such that Tainted.foo makes use of
tainted.someGlobal, rather than original.someGlobal? Must I use
eval/exec? Note that the point is that I don't want to have to
duplicate the definition of foo.

Make foo a classmethod, lose the globals altogether and make them class
variables:

class Original:
someVar = 42
def foo(cls):
print cls.someVar

foo=classmethod(foo)

class Tainted(Original):
someVar = 24

Original().foo() # prints 42
Tainted().foo() # prints 24

Oh, and lose the 'from original import *', just do 'import original' and
refer to original.Original in your class definition.

 

[Corey Lubin]

It seems that in trying to present simple concept code, for the
purpose of example, I forgot to mention some of the restrictions on
what can be done. original.py is not of my own creation; it's a
seperately maintained module, external to the project, and thus cannot
be modified.

For those that care to hear the full background/details of this
problem: I make use of HTMLParser ("Original", from the example), but
it's not resilient enough for real world pseudo-HTML, so I decided to
modify it through extension. I've subclassed it (currently with no
overriden methods, in an attempt to allow the subclass to benefit from
changes to the base class) and attempted to override a global variable
that the base class uses, however my approach doesn't work. In this
case, "doesn't work" means that, in terms of my example, Tainted.foo
uses original.someGlobal rather than tainted.someGlobal. This happens
presumably because Tainted.foo is actually Original.foo (the foo
method is inherited) and inherited methods apparently don't pay any
mind to the namespace of their inheritors (take note that I'm not
saying that they /should/; I haven't given any thought to that; I'm
just saying that I expected this approach to work). This apparent
failure in my approach leads me to think that eval/exec is my
solution; I'm just hoping to hear that there is something more simple
than that.

My "import everything into this namespace and just modify what I need
to" approach was my attempt at duplicating the original module (for
miniature modification) without actually duplicating it's code (so
that my modified version won't grow stale as the original module's
code is updated by it's maintainer).

Example translation guide (for those that feel the details make a
diff.):
original.py = HTMLParser.py
original.someGlobal = HTMLParser.locatestarttagend (a regex object)
original.Original.foo = HTMLParse.HTMLParser.check_for_whole_start_tag

I hope this clears some things up.

Corey

 

[Bengt Richter]

It seems that in trying to present simple concept code, for the
purpose of example, I forgot to mention some of the restrictions on
what can be done. original.py is not of my own creation; it's a
seperately maintained module, external to the project, and thus cannot
be modified.

For those that care to hear the full background/details of this
problem: I make use of HTMLParser ("Original", from the example), but
it's not resilient enough for real world pseudo-HTML, so I decided to
modify it through extension. I've subclassed it (currently with no
overriden methods, in an attempt to allow the subclass to benefit from
changes to the base class) and attempted to override a global variable
that the base class uses, however my approach doesn't work. In this
case, "doesn't work" means that, in terms of my example, Tainted.foo
uses original.someGlobal rather than tainted.someGlobal. This happens
presumably because Tainted.foo is actually Original.foo (the foo
method is inherited) and inherited methods apparently don't pay any
mind to the namespace of their inheritors (take note that I'm not
saying that they /should/; I haven't given any thought to that; I'm
just saying that I expected this approach to work). This apparent
failure in my approach leads me to think that eval/exec is my
solution; I'm just hoping to hear that there is something more simple
than that.

My "import everything into this namespace and just modify what I need
to" approach was my attempt at duplicating the original module (for
miniature modification) without actually duplicating it's code (so
that my modified version won't grow stale as the original module's
code is updated by it's maintainer).

Example translation guide (for those that feel the details make a
diff.):
original.py = HTMLParser.py
original.someGlobal = HTMLParser.locatestarttagend (a regex object)
original.Original.foo = HTMLParse.HTMLParser.check_for_whole_start_tag

I hope this clears some things up.

Still not 100% sure what you're doing, but suppose that you wanted to get the
effect of modifying HTMLParser.py without modifying the original file, and be
able to proceed as if your modified file were "myHTMLParser.py".

I think you could make a small actual myHTMLParser.py that starts by _executing_
(not importing) the code of the original HTMLParser, and then just modfies the
result of that as desired. E.g., (untested beyond what you see below ;-)

==< myHTMLParser.py >======
import sys, os
origname = 'HTMLParser'
for lookdir in sys.path:
path = os.path.join(lookdir, origname+'.py')
print path
if os.path.exists(path):
print 'found', path
break
else:
raise ImportError,'Could not find "%s.py"' %origname
execfile(path, globals())

def demo(*args): print 'demo',args
locatestarttagend = demo
HTMLParser.check_for_whole_start_tag = demo
newglobal = 'new global'
===========================

Now, using it interactively:

(of course, you can take out the print statements that generated this printout...
(BTW, since I blithely ignored the .zip in the path, this won't work for something in the zip file.)
HTMLParser.py
c:\pywk\HTMLParser.py
C:\WINNT\System32\python23.zip\HTMLParser.py
D:\python23\DLLs\HTMLParser.py
D:\python23\lib\HTMLParser.py
found D:\python23\lib\HTMLParser.py
)

Now we'll check our mods:
'new global'
demo ()
demo (<myHTMLParser.HTMLParser instance at 0x00900828>,)

Maybe this gives some ideas? It's not a subclass/base class thing, but maybe it will
do what you need. Of course, you could subclass or rebind or whatever in the mod code
(which only executes when you import myHTMLParser, BTW).

Regards,
Bengt Richter

 

[Hung Jung Lu]

[original.py]
someGlobal=1

class Original:
def foo(self):
# Make use of someGlobal
[/original.py]

[tainted.py]
from original import *
someGlobal=2

class Tainted(Original):
pass
[/tainted.py]

How might one make it such that Tainted.foo makes use of
tainted.someGlobal, rather than original.someGlobal?

Use

[tainted.py]
import original
original.someGlobal=2

class Tainted(original.Original):
pass
[/tainted.py]

-------------------------

(a) Python's namespace mechanism is hard to understand for newcomers.
It is a fact, no matter how hard people try to deny it: your question
has been asked a few hundred times before. But once you understand
that assignment in Python is name binding and is very different from
assignment in other languages like C/C++, you'll get used to it. Same
with import and from ... import ... statements.

Namespace mechanism is the heart and soul of Python. When you have
more time, try to read more and understand it. Otherwise you'll run
into surprises in many places. When writing each line of Python code,
you need to keep in mind what namespaces you are dealing with: an
object's namespace? local dictionary? global dictionary? built-in?

(b) Stay with your present approach. There is still no need for exec.
Python is powerful enough to allow you to tweak many aspects of
existing modules/classes/functions/methods without needing to evaluate
code strings. What you are doing is called "patching". You are making
a patch for existing module/class. If the changes are like adding
attributes, surround a method with some around codes, those can be
done easily with inheritance. If you need to override a foreign class
method because you want to change the code in some spots inside the
method, and the foreign class is used by other foreign
classes/modules, you can write your code in your own module/class and
replace the foreign implementation. Look into the documentation for
the "new" module. In short, there are many ways to dynamically modify
Python modules/classes/functions/methods.

(c) More generally, what you want to do falls into the arena of
Aspect-Oriented Programming. No need to understand it, now. But keep
it in mind.

regards,

Hung Jung

 

[Corey Lubin]

Still not 100% sure what you're doing, but suppose that you wanted to
get the effect of modifying HTMLParser.py without modifying the
original file, and be able to proceed as if your modified file were
"myHTMLParser.py".

Basically, yes. I want to effectively make a copy of a module and then
make adjustments to [the behaviour of] the copy, not to the original.

I think you could make a small actual myHTMLParser.py that starts by
_executing_ (not importing) the code of the original HTMLParser, and
then just modfies the result of that as desired. E.g., (untested
beyond what you see below ;-)

This could achieve what I want. However, I mentioned exec/eval in my
question and even then that was only in an attempt to state what I hope
I need not do. This approach strays even further down that road of by
exec'ing the entire file, getting even more complex and scary. In short,
this is not where I want to go.

Maybe this gives some ideas? It's not a subclass/base class thing, but
maybe it will do what you need. Of course, you could subclass or
rebind or whatever in the mod code (which only executes when you
import myHTMLParser, BTW).

No, it wasn't a sub/base class thing, but that wasn't a significant
requirement and you solved the real issue; I was just hoping for a
solution that is supported more directly by the language.

Regards,
Bengt Richter

Thanks for the time you put into the suggestion. The concept of this
approach might be useful in the future.

- Corey Lubin

 

[Corey Lubin]

[grr, had trouble posting via my own newsserver: "Done. Waiting for
confirmation"]

(e-mail address removed) (Hung Jung Lu) wrote in

Use

[tainted.py]
import original
original.someGlobal=2

class Tainted(original.Original):
pass
[/tainted.py]

-------------------------

I discarded this idea very early on for a couple of reasons, one of
which may not be very important and another which might not have been
valid after all.

First of all, I wanted my tainted module to completely mimic the
interface of the original module. Importing in the style that you have
would not do so, as all names other than "Tainted" would have to be
prefixed by "original". I don't blame you for that, as my simplified
example only mentioned the class "Original" and didn't make it clear
that there may be other names in the original interface. Furthermore,
in this particular case, there probably /aren't/ any other names that
I would want to have carried over into my tainted module.

My second reason for not following that path was that I had the
impression that a particular module can only be imported once; any
further imports of the same module would just be implemented by
binding new names to the first copy of that module. When you import
"original" in the code above, does "tainted" get it's own copy of the
"original" module, or would other imports of the module be referring
to the same thing? If the answer is the former, then my concerns over
this approach were not valid. If the answer is the latter, then this
approach isn't acceptable.

To further expand upon my concerns raised in the last paragraph, I
want to have the choice to use either or both of tainted.Tainted and
original.Original; and when I use them I want them to have distinct
behaviour. The way I understand things, the "original.someGlobal=2"
line in your suggestion would forever scar the "original" module in
the memory of the python interpreter instance running that code. Ok,
the "forever" wasn't necessary, but I believe my point was made: we
would no longer just be making changes to "tainted", but also to
"original", which other user code may depend upon to act in it's
original way. In actuality this may not be an issue and I may only
need to use the modified version, but I tend to think in terms of
principles rather than whether something happens to work out in a
particular instance.

Just incase there is any ambiguity over my intentions, I've provided
code below:

[user.py]
import tainted
t = tainted.Tainted()
t.foo() # should behave in the altered/tainted way

import original
o = original.Original
o.foo() # should behave in the original way

tainted.someGlobal # should exist (not as important or hard to fix)
tainted.original.someGlobal # as opposed to this; AND...
tainted.someGlobal == original.someGlobal # should be false
[/user.py]

(a) Python's namespace mechanism is hard to understand for newcomers.
It is a fact, no matter how hard people try to deny it: your question
has been asked a few hundred times before. But once you understand
that assignment in Python is name binding and is very different from
assignment in other languages like C/C++, you'll get used to it. Same
with import and from ... import ... statements.

Namespace mechanism is the heart and soul of Python. When you have
more time, try to read more and understand it. Otherwise you'll run
into surprises in many places. When writing each line of Python code,
you need to keep in mind what namespaces you are dealing with: an
object's namespace? local dictionary? global dictionary? built-in?

Thanks for understanding my perspective, rather than working around it
or trying to change it. I'm taking your advice and have already
[re]read the related sections in the official Python tutorial and the
language reference.

(b) Stay with your present approach. There is still no need for exec.
Python is powerful enough to allow you to tweak many aspects of
existing modules/classes/functions/methods without needing to evaluate
code strings. What you are doing is called "patching". You are making
a patch for existing module/class. If the changes are like adding
attributes, surround a method with some around codes, those can be
done easily with inheritance. If you need to override a foreign class
method because you want to change the code in some spots inside the
method, and the foreign class is used by other foreign
classes/modules, you can write your code in your own module/class and
replace the foreign implementation. Look into the documentation for
the "new" module. In short, there are many ways to dynamically modify
Python modules/classes/functions/methods.

You may or may not completely get my drift -- in that I'd like to
preserve the behaviour of the original code -- but this is all,
nonetheless, relevant and helpful.

(c) More generally, what you want to do falls into the arena of
Aspect-Oriented Programming. No need to understand it, now. But keep
it in mind.

Will do.

Thanks,
Corey Lubin

 

[Hung Jung Lu]

First of all, I wanted my tainted module to completely mimic the
interface of the original module. Importing in the style that you have
would not do so, as all names other than "Tainted" would have to be
prefixed by "original".

OK, I understand your next point. But let me comment on this first
point, unrelated to the next point. In general, it's a better practice
to avoid the 'from ... import *' syntax, since it makes your code
harder to read: when you see a name inside your module, you are not
sure whether it comes from somewhere else, especially if you have
several 'from ... import *' statements. This recommendation is common.
To give an example: in the past wxPython's examples where filled with
'from' statements, but nowadays more and more people are using
straight 'import' statements, and explicitly type 'wx.' prefix to
module attributes. It is more work, for sure, but if really necessary,
you can (a) use the 'import ... as ...' syntax, (b) if something is
repeatedly used, you can use an alias name, just use the statement
'alias = mymodule.myattribute'.

My second reason for not following that path was that I had the
impression that a particular module can only be imported once; any
further imports of the same module would just be implemented by
binding new names to the first copy of that module.

That is correct.

To further expand upon my concerns raised in the last paragraph, I
want to have the choice to use either or both of tainted.Tainted and
original.Original; and when I use them I want them to have distinct
behaviour.

Hmm... for your purpose, it may be easier to use the exec statement,
after all.

f = open('foreign_module.py')
module_code = f.read().strip()
f.close()
exec module_code

Whatever asset in the foreign module now becomes part of your module,
as a fresh copy. I would like to point out that this is not a common
approach. Python allows you to do many things. But some approaches
have more side effects than others. For instance, in the above code,
your program will now depend on the module file. This is not a problem
when you are just using Python scripts. But if one day you decide to
package your program into an executable and ship it to other people,
e.g. by using py2exe, this approach would be kind of bad, since you
will also have to ship the foreign module script, or ship its compiled
code, or use some other ways of hacking.

regards,

Hung Jung