doctests and decorators

[Eric Snow]

Apparently there is a known issue with doctests, in which tests in
functions using externally defined decorators are ignored. The
recommended fix is to update the order of checks in the _from_module
method of DocTestFinder in the doctest module. The bug and fix are
discussed at the following URLs (and several places in this group):

http://bugs.python.org/issue1108
http://mail.python.org/pipermail/python-list/2007-September/627866.html

The fix implies that the inpect.getmodule function will find the
module of the function object and not of the decorator. However, in
2.4 the inspect.getmodule function returns the module of the
decorator. I have subsequently tested this in 2.5 and 2.6, and it
also returns the module of the decorator. As such, the fix for
doctests does not work in my tests. Below is the test code that I
used:

<EXAMPLE>

test1.py
++++++++++++++++++++++++++++
def decorator(function):
def new_function(*args, **kwargs):
return function(*args, **kwargs)
return new_function

test2.py
++++++++++++++++++++++++++++
import test1
import inspect

class Test(object):
@test1.decorator
def test2(self): pass

def run_tests():
test = Test()
test.test2()

print("Test is class, test is instance, test2 is method of Test
(has decorator)")
print("test's module: %s" % inspect.getmodule(test))
print("Test's module: %s" % inspect.getmodule(Test))
print("test.test2's module: %s" % inspect.getmodule
(test.test2))
print("Test.test2's module: %s" % inspect.getmodule
(Test.test2))
print("test.test2's func_name: %s" % test.test2.func_name)
print("Test.test2's func_name: %s" % Test.test2.func_name)

if __name__ == "__main__":
run_tests()

</EXAMPLE>

Here is the output that I got in 2.4, 2.5, and 2.6:

Test is class, test is instance, test2 is method of Test (has
decorator)
test's module: <module '__main__' from 'test2.py'>
Test's module: <module '__main__' from 'test2.py'>
test.test2's module: <module 'test1' from '/tmp/test1.py'>
Test.test2's module: <module 'test1' from '/tmp/test1.py'>
test.test2's func_name: new_function
Test.test2's func_name: new_function

If things were working right, then the module for test.test2 would be
the same as the module for test. I must be missing something, as the
referenced discussion suggests a simple conclusion. Any ideas?

-eric

 

[Eric Snow]

Apparently there is a known issue with doctests, in which tests in
functions using externally defined decorators are ignored.  The
recommended fix is to update the order of checks in the _from_module
method of DocTestFinder in the doctest module.  The bug and fix are
discussed at the following URLs (and several places in this group):

http://bugs.python.org/issue1108htt...ermail/python-list/2007-September/627866.html

The fix implies that the inpect.getmodule function will find the
module of the function object and not of the decorator.  However, in
2.4 the inspect.getmodule function returns the module of the
decorator.  I have subsequently tested this in 2.5 and 2.6, and it
also returns the module of the decorator.  As such, the fix for
doctests does not work in my tests.  Below is the test code that I
used:

<EXAMPLE>

test1.py
++++++++++++++++++++++++++++
def decorator(function):
    def new_function(*args, **kwargs):
        return function(*args, **kwargs)
    return new_function

test2.py
++++++++++++++++++++++++++++
import test1
import inspect

class Test(object):
    @test1.decorator
    def test2(self): pass

def run_tests():
    test = Test()
    test.test2()

    print("Test is class, test is instance, test2 is method of Test
(has decorator)")
    print("test's module:          %s" % inspect.getmodule(test))
    print("Test's module:          %s" % inspect.getmodule(Test))
    print("test.test2's module:    %s" % inspect.getmodule
(test.test2))
    print("Test.test2's module:    %s" % inspect.getmodule
(Test.test2))
    print("test.test2's func_name: %s" % test.test2.func_name)
    print("Test.test2's func_name: %s" % Test.test2.func_name)

if __name__ == "__main__":
    run_tests()

</EXAMPLE>

Here is the output that I got in 2.4, 2.5, and 2.6:

Test is class, test is instance, test2 is method of Test (has
decorator)
test's module:          <module '__main__' from 'test2.py'>
Test's module:          <module '__main__' from 'test2.py'>
test.test2's module:    <module 'test1' from '/tmp/test1.py'>
Test.test2's module:    <module 'test1' from '/tmp/test1.py'>
test.test2's func_name: new_function
Test.test2's func_name: new_function

If things were working right, then the module for test.test2 would be
the same as the module for test.  I must be missing something, as the
referenced discussion suggests a simple conclusion.  Any ideas?

-eric

One work-around I found is the following change in example:

<EXAMPLE>

test1.py
++++++++++++++++++++++++++++
def decorator(function):
def new_function(*args, **kwargs):
return function(*args, **kwargs)
new_function.__module__ = function.__module__
new_function.__doc__ = function.__doc__
new_function.__name__ = function.__name__
return new_function

</EXAMPLE>

However, this seems pretty lame. The doctest module should be able to
figure out that the docstring belongs is there in the module.

-eric

 

[Eric Snow]

Eric Snow schrieb:




It's not an issue with the doctest module but with your code. You want
to use functools.wraps().

http://docs.python.org/library/functools.html#functools.wraps

Christian

Unfortunately, I am stuck on 2.4 for now, which does not have the
functools.

-eric

 

[Michele Simionato]

Unfortunately, I am stuck on 2.4 for now, which does not have the
functools.

-eric

But you can always use the decorator module: http://pypi.python.org/pypi/decorator

 

[Eric Snow]

But you can always use the decorator module:http://pypi.python.org/pypi/decorator

Thanks to both of you. Very helpful.

So in general should decorators always hide themselves? I am guessing
not, otherwise this would already be part of their behavior. Still,
is it the common case to camouflage the decorator like this? If so, I
would expect it to be the default behavior of decorators.

-eric

 

[Steven D'Aprano]

The Python goal is "no magic". So, if you want the stuff wrapped, you
do it (as the default traceback shows where the code actually goes). It
would be far more complicated to display the truth if decorators
defaulted to modifying the builtins, and you had to do magic to remove
that part of the decoration.

I'm afraid I can't understand what you're saying. What do you consider
"magic"? What's a "default traceback"? What do you mean, "display the
truth"?

A decorator has _very_ simple semantics,
while anything that automatically copied attributes would have funny
semantics indeed for use by funny decorators like:

[...]


functools.wraps() automatically copies attributes:
.... @functools.wraps(func)
.... def inner(*args):
.... return func(args) + 1
.... return inner
........ return 1
....'Attribute'