Overriding "__setattr__" of a module - possible?

J

John Nagle

Is it possible to override "__setattr__" of a module? I
want to capture changes to global variables for debug purposes.

None of the following seem to have any effect.

modu.__setattr__ = myfn

setattr(modu, "__setattr__", myfn)

delattr(modu, "__setattr__")

John Nagle
 
M

Michele Simionato

   Is it possible to override "__setattr__" of a module?  I
want to capture changes to global variables for debug purposes.

   None of the following seem to have any effect.

        modu.__setattr__ = myfn

        setattr(modu, "__setattr__", myfn)

        delattr(modu, "__setattr__")

                                John Nagle

There is a dirty trick which involves fiddling with sys.modules.
For instance:

$ cat x.py
import sys

class FakeModule(object):
def __init__(self, dic):
vars(self).update(dic)
def __setattr__(self, name, value):
print "setting %s=%s" % (name, value)
object.__setattr__(self, name, value)

a = 1
def f():
print 'called f'

sys.modules[__name__] = FakeModule(globals())

Here is an ipython session:

In [1]: import x

In [2]: x.a
Out[2]: 1

In [3]: x.f
Out[3]: <function f at 0x93f5614>

In [4]: x.f()
called f

In [5]: x.a=2
setting a=2

In [6]: x.a
Out[6]: 2
 
J

John Nagle

There is a dirty trick which involves fiddling with sys.modules.
For instance:

$ cat x.py
import sys

class FakeModule(object):

Cute, but it doesn't work in general. Faking a module as a
class fails when you simply call

x()

within the module.

I also tried

modu.__dict__['__setattr__'] = myfun
...
modu.foo = 1 # "myfun" does not get called.

Any more ideas?


John Nagle
 
S

Stephen Hansen

Cute, but it doesn't work in general. Faking a module as a
class fails when you simply call

x()

within the module.

Huh? Explain how it doesn't work? I've done it at least twice
(shamefully do I admit this) and it works fine. Real life code.
Any more ideas?

There aren't any; modules do not follow the class object protocol. They
are simple types with a __dict__ (which you can't change, either, so no
replacing it with a dict that implements __setattr__).

Replacing the module with a class in sys.modules is really the only way
to fake the behavior.

--

Stephen Hansen
... Also: Ixokai
... Mail: me+list/python (AT) ixokai (DOT) io
... Blog: http://meh.ixokai.io/


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (Darwin)

iQEcBAEBAgAGBQJMGFQTAAoJEKcbwptVWx/lYLYH/10ATTtT7yMHGV7un35QpvLf
JYLY+73gOIDHE+WP+PxmdUETZHPcckHEB4Cgat3GmT4Gvh7nYyyM/oV4BfiqRnE6
f3v/NIQYMxGaNVpGY2IkYPkfJqOtjza9eaVZtr6nTWcu6WaYCNYUULoYEnlFk/qW
FFlWqlx1aZLAz4Pwc48pH0FZsm1cqv8FB9aE5IIqQqMXf8DeU9krsk8Y3QuTa5fC
6jWBaHLEXM5Vw1oroQrMDs/rZX3iVTZH9wjgjjqioIC8rnarAMlg458P6TaNQKLe
p0TP3F07OK2sLm0H9DRSoFJbIrG8Em09ZwkItWTF5h7SSMNwHLvDIANe6ZM05uE=
=kP/1
-----END PGP SIGNATURE-----
 
J

John Nagle

Huh? Explain how it doesn't work? I've done it at least twice
(shamefully do I admit this) and it works fine. Real life code.


There aren't any; modules do not follow the class object protocol. They
are simple types with a __dict__ (which you can't change, either, so no
replacing it with a dict that implements __setattr__).

Replacing the module with a class in sys.modules is really the only way
to fake the behavior.

OK, working on this. I can make a module make itself into a
fake class, but can't yet do it to other modules from outside.

John Nagle
 
S

Stephen Hansen

OK, working on this. I can make a module make itself into a
fake class, but can't yet do it to other modules from outside.

Can you show a bit of code that you're doing? I've never had much
trouble with it: but I'm not sure what you mean by that final statement.

You're attempting to turn "other" modules into faux-modules-as-classes?
I don't think you can do that successfully. The whole faux-module
strategy only works when the module is designed around that principle, I
think.

--

Stephen Hansen
... Also: Ixokai
... Mail: me+list/python (AT) ixokai (DOT) io
... Blog: http://meh.ixokai.io/


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (Darwin)

iQEcBAEBAgAGBQJMGGKSAAoJEKcbwptVWx/lf0EH/3TLEsuwvJmBN0739MgfvANN
v1UANHnMYHrmVsx2/baQ6yT1yAM7LhThC++vDcgjZrNAALafM6fWS1nbVbydv33S
fMuuV7EH5KvxBWsmG7uGYXjbnUIADhb62dLgEKf8wOmrDj2ZbVawMNE+VImNCR0h
su0PJiCVcQ6Y/m1swCl1eMoErCnVPVu/yZdI857SHQDm0VFE5PdV/Tm+MKsBpJ+N
6uaQSSLSbj1Y+qPXPTf/vynh6raHopTcpkAxK0U0SWPWXv1uTriRrX4LYFMHNLCW
9DIWz5f6CmbUFpRgIwAHZPZBGaeOfK/ob8EdrTy3Ifgdkh6NQb1tnkxoOG+5IKU=
=KwHI
-----END PGP SIGNATURE-----
 
M

Michele Simionato

    OK, working on this.  I can make a module make itself into a
fake class, but can't yet do it to other modules from outside.

                                        John Nagle


I think you can with something like

import module
sys.modules[module.__name__] = FakeModule(vars(module))
 
L

Lie Ryan

Is it possible to override "__setattr__" of a module? I
want to capture changes to global variables for debug purposes.

None of the following seem to have any effect.

modu.__setattr__ = myfn

setattr(modu, "__setattr__", myfn)

delattr(modu, "__setattr__")

John Nagle

I doubt this is what you wanted, but...

import itertools
module = type(itertools)

class FakeModule(module):
def __init__(self, module):
super(FakeModule, self).__init__(module.__name__, module.__doc__)
self.__module = module
def __getattr__(self, attr):
return getattr(self.__module, attr)
def __setattr__(self, attr, val):
print self, attr, val
super(FakeModule, self).__setattr__(attr, val)
 
S

Steven D'Aprano

   Is it possible to override "__setattr__" of a module?  I
want to capture changes to global variables for debug purposes.
[...]
There is a dirty trick which involves fiddling with sys.modules. For
instance:
[snip example of installing a class in sys.modules]

I'm not sure that this is a dirty trick. Isn't it officially supported?
If I recall correctly, Python used to insist that modules in sys.modules
were actual module objects, and that restriction was lifted deliberately
to allow clever tricks like this. Or am I thinking of something else?
 
S

Stephen Hansen

You are wrong, my friend. :)

Modules follow the new style class and instance protocol. Modules aren't
classes but instances of the internal module type. Since you can't
overwrite magic methods on instances of new style classes you can't
overwrite __setattr__ on module level, too.

Well, fine, be that way, all right and correct and shiznit. :)

--

Stephen Hansen
... Also: Ixokai
... Mail: me+list/python (AT) ixokai (DOT) io
... Blog: http://meh.ixokai.io/


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (Darwin)

iQEcBAEBAgAGBQJMGPxPAAoJEKcbwptVWx/lx3kH/Ap7m65MXyjL2DE/7hBcOybo
JWa3YvnGug1rCLtuPjt2g4OkLrMZuaoLTGbwkprwljKPIy8seUMjJXU96rfvNIkb
SyyWnwEm+CjkaWTdHB4x4T/lc5L+O1B5GS0pfcUsa9k3XAPzPKVhqgUulpiTKFzS
IaOIcf1asvisS6bLhnTODE+ZAQJrssk8j5NVBemPKC86CQtiC6MLQq9bC3znwnjf
39NpVw10gcXpCO8Lf7mIRJM3kMsrrnz9TjXUGyR334GnAsGOt11cHTFL5GDdA+9E
lLS522lIcq1sep+fcOIGxBMPygeUQO2XWxPQcn3D1FNmSiFyZvrh/V+JvP8YnIk=
=Exik
-----END PGP SIGNATURE-----
 
J

John Nagle

There is a dirty trick which involves fiddling with sys.modules...

That doesn't do quite what you'd expect. Here's a version with
more debug output:

import sys

class FakeModule(object):
def __init__(self, dic):
vars(self).update(dic)
def __setattr__(self, name, value):
print "setting %s=%s" % (name, value)
object.__setattr__(self, name, value)

a = 1
def f():
print ('called f, a = %s' % (a,))

print("Patching " + __name__)
sys.modules[__name__] = FakeModule(globals())

---

C:\projects\newthreading>\python26\python
ActivePython 2.6.5.12 (ActiveState Software Inc.) based on
Python 2.6.5 (r265:79063, Mar 20 2010, 14:22:52) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.called f, a = None

---

Note that there are now two copies of "a", one bound to the module and
referenced in "f", and a second bound to the class, and referenced by
"x.a". Uh oh.

The problem here is that when "def f..." was defined, its reference
to "a" was bound to the object containing "a" at the time, which is
the module-level copy of the variable.

I don't think turning a module into a class is going to work.
Binding the functions into the class won't work, because Python
doesn't have class-level (as opposed to instance-level) functions.
If we move x.f into class FakeModule, then "x.f()" is called as
"f(x)", with an unexpected argument.

Any better ideas? All I really want to do is to override "__setattr__"
for a module.

John Nagle
 
I

Ian Kelly

Note that there are now two copies of "a", one bound to the module and
referenced in "f", and a second bound to the class, and referenced by
"x.a".  Uh oh.

The problem here is that when "def f..." was defined, its reference
to "a" was bound to the object containing "a" at the time, which is
the module-level copy of the variable.

I don't think turning a module into a class is going to work.
Binding the functions into the class won't work, because Python
doesn't have class-level (as opposed to instance-level) functions.
If we move x.f into class FakeModule, then "x.f()" is called as
"f(x)", with an unexpected argument.

Any better ideas?  All I really want to do is to override "__setattr__"
for a module.

It seems to me that a proxy for the actual module would work better.

import sys

class ModuleProxy(object):

def __init__(self, module):
object.__setattr__(self, 'module', module)

def __getattribute__(self, name):
module = object.__getattribute__(self, 'module')
return getattr(module, name)

def __setattr__(self, name, value):
module = object.__getattribute__(self, 'module')
print "setting %s=%s" % (name, value)
setattr(module, name, value)

a = 1
def f():
print a

sys.modules[__name__] = ModuleProxy(__import__(__name__))


Cheers,
Ian
 
J

John Nagle

Note that there are now two copies of "a", one bound to the module and
referenced in "f", and a second bound to the class, and referenced by
"x.a". Uh oh.

The problem here is that when "def f..." was defined, its reference
to "a" was bound to the object containing "a" at the time, which is
the module-level copy of the variable.

I don't think turning a module into a class is going to work.
Binding the functions into the class won't work, because Python
doesn't have class-level (as opposed to instance-level) functions.
If we move x.f into class FakeModule, then "x.f()" is called as
"f(x)", with an unexpected argument.

Any better ideas? All I really want to do is to override "__setattr__"
for a module.

It seems to me that a proxy for the actual module would work better.

import sys

class ModuleProxy(object):

def __init__(self, module):
object.__setattr__(self, 'module', module)

def __getattribute__(self, name):
module = object.__getattribute__(self, 'module')
return getattr(module, name)

def __setattr__(self, name, value):
module = object.__getattribute__(self, 'module')
print "setting %s=%s" % (name, value)
setattr(module, name, value)

a = 1
def f():
print a

sys.modules[__name__] = ModuleProxy(__import__(__name__))


Cheers,
Ian

That just leaves things in a state where even "sys" and "import"
are undefined.

John Nagle
 
G

Gabriel Genellina

Say what? It works fine for me.

<module 'sys' (built-in)>

It *mostly* works, but not always. Try this function:

def g(x):
global a
print 'global a -> ', x
a = x

py> import fake # ModuleProxy stuff
py> fake.f()
1
py> fake.a = 3
setting a=3
py> fake.f()
3
py> fake.g(8)
global a -> 8
py> fake.f()
8
py> fake.a
8

Note the fake.g(8) call: __setattr__ wasn't called.
If the OP wants to trace assignments to global variables, this becomes a
problem.

A function defined in a module holds a reference to the module's __dict__
in its func_globals attribute. Getting and setting global variables goes
directly to this dictionary, and does not use the module object at all.

Even worse, the LOAD_GLOBAL/STORE_GLOBAL opcodes (which implement getting
and setting global variables) assume func_globals is a true dictionary and
bypass any overriden __getitem__/__setitem__ methods (an optimization,
surely). I'm afraid it will be hard to intercept global variable usage in
these circumstances.
 
A

Alf P. Steinbach

* Gabriel Genellina, on 17.06.2010 09:25:
It *mostly* works, but not always. Try this function:

def g(x):
global a
print 'global a -> ', x
a = x

py> import fake # ModuleProxy stuff
py> fake.f()
1
py> fake.a = 3
setting a=3
py> fake.f()
3
py> fake.g(8)
global a -> 8
py> fake.f()
8
py> fake.a
8

Note the fake.g(8) call: __setattr__ wasn't called.
If the OP wants to trace assignments to global variables, this becomes a
problem.

A function defined in a module holds a reference to the module's
__dict__ in its func_globals attribute. Getting and setting global
variables goes directly to this dictionary, and does not use the module
object at all.

Even worse, the LOAD_GLOBAL/STORE_GLOBAL opcodes (which implement
getting and setting global variables) assume func_globals is a true
dictionary and bypass any overriden __getitem__/__setitem__ methods (an
optimization, surely). I'm afraid it will be hard to intercept global
variable usage in these circumstances.

Great exposition.

But who would have thunk that Python *isn't dynamic enough*? :)


Cheers,

- Alf
 
G

Gabriel Genellina

* Gabriel Genellina, on 17.06.2010 09:25:

Great exposition.

But who would have thunk that Python *isn't dynamic enough*? :)

Yep... There are other examples too (e.g. the print statement in 2.x
bypasses sys.stdout.write; see also a recent thread "Which objects are
expanded by double-star ** operator?")

Most of them seem to be speed optimizations, some might be considered
subtle bugs. But in this case (global variable references) speed is so
critical than even the dict lookup is inlined; the code in ceval.c says:

/* Inline the PyDict_GetItem() calls.
WARNING: this is an extreme speed hack.
Do not try this at home. */

Python is dynamic but not so much as to make it crawl like a snail...
 
F

Fuzzyman

Yep... There are other examples too (e.g. the print statement in 2.x  
bypasses sys.stdout.write;


What do you mean by this? The print statement in 2.x does *not* bypass
sys.stdout. It may use other methods besides write (writeln perhaps)
but you can *definitely* override sys.stdout to capture the output
from print statements.

Michael Foord
 
J

John Nagle

Note the fake.g(8) call: __setattr__ wasn't called.
If the OP wants to trace assignments to global variables, this becomes a
problem.

A function defined in a module holds a reference to the module's
__dict__ in its func_globals attribute. Getting and setting global
variables goes directly to this dictionary, and does not use the module
object at all.

Even worse, the LOAD_GLOBAL/STORE_GLOBAL opcodes (which implement
getting and setting global variables) assume func_globals is a true
dictionary and bypass any overriden __getitem__/__setitem__ methods (an
optimization, surely). I'm afraid it will be hard to intercept global
variable usage in these circumstances.

OK, thanks. You can't actually replace "__setattr__" for
a module, and the "module as class" hack is iffy. I didn't
really think this would work, but it was worth a try.

I'm trying out a proof of concept implementation for a new
approach to safe threading. It's somewhat similar in concept
to Alan Olsen's scheme. The basic difference is that once
the program goes multi-thread, code objects and some other
bindings are locked down and become unchangeable. Olsen
was climbing the walls trying to get the locking right for
the awful cases like redefining a function while another thread
is inside it. I'm trying to lock out some of those cases.
If you do that, removing the GIL requires less pain than
Olsen experienced.

The key idea is that the use cases for most of Python's
code dynamism are during setup and initialization. You usually
don't change code once the program has gone into its heavy
parallel processing phase. This suggests a practical compromise.

More on this once I have something people can download and try.
I'm doing a test implementation in Python so people can try the
concept and see if it works in practice. It won't go fast;
it's just to give a feel for what it would be like.

John Nagle
 
G

Gabriel Genellina

What do you mean by this? The print statement in 2.x does *not* bypass
sys.stdout. It may use other methods besides write (writeln perhaps)
but you can *definitely* override sys.stdout to capture the output
from print statements.

Suppose you want to implement a "tee" variant in Python: print output
should go to stdout and also to some file (with timestamp added, just to
be fancy). First attempt:

py> import sys
py> import time
py>
py> class tee(file):
.... def write(self, data):
.... file.write(self, '%s: %r\n' % (time.ctime(), data))
.... sys.__stdout__.write(data)
....
py> sys.stdout = tee('test.txt', 'w')
py> print "Hello world"
py> print "Bye"
py> ^Z

D:\TEMP>type test.txt
Hello world
Bye


Note:
- no output to stdout inside the interpreter
- no timestamp in the file

This modified version works fine:

py> class tee():
.... def __init__(self, filename, mode):
.... self.file = open(filename, mode)
.... def write(self, data):
.... self.file.write('%s: %r\n' % (time.ctime(), data))
.... sys.__stdout__.write(data)

What happened? When sys.stdout is an instance of some class inheriting
from file (that is, isinstance(sys.stdout, file) is true) then the print
statement ignores sys.stdout.write() completely -- instead it calls
directly some C stdio functions (fwrite).
The only way to influence 'print' is *not* to inherit from file in the
first place.

It's an optimization, sure. I guess it is there before inheriting from
builtin types was allowed (in such scenario, it's a perfectly valid
optimization). Now, perhaps the test for 'file' should be more strict,
only taking the C shortcut when using an actual file instance, not a
subclass of it. This would allow the example above to work correctly.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,901
Latest member
Noble71S45

Latest Threads

Top