doctests and decorators

Discussion in 'Python' started by Eric Snow, Jun 16, 2009.

  1. Eric Snow

    Eric Snow Guest

    Apparently there is a known issue with doctests, in which tests in
    functions using externally defined decorators are ignored. The
    recommended fix is to update the order of checks in the _from_module
    method of DocTestFinder in the doctest module. The bug and fix are
    discussed at the following URLs (and several places in this group):

    http://bugs.python.org/issue1108
    http://mail.python.org/pipermail/python-list/2007-September/627866.html

    The fix implies that the inpect.getmodule function will find the
    module of the function object and not of the decorator. However, in
    2.4 the inspect.getmodule function returns the module of the
    decorator. I have subsequently tested this in 2.5 and 2.6, and it
    also returns the module of the decorator. As such, the fix for
    doctests does not work in my tests. Below is the test code that I
    used:

    <EXAMPLE>

    test1.py
    ++++++++++++++++++++++++++++
    def decorator(function):
    def new_function(*args, **kwargs):
    return function(*args, **kwargs)
    return new_function

    test2.py
    ++++++++++++++++++++++++++++
    import test1
    import inspect

    class Test(object):
    @test1.decorator
    def test2(self): pass

    def run_tests():
    test = Test()
    test.test2()

    print("Test is class, test is instance, test2 is method of Test
    (has decorator)")
    print("test's module: %s" % inspect.getmodule(test))
    print("Test's module: %s" % inspect.getmodule(Test))
    print("test.test2's module: %s" % inspect.getmodule
    (test.test2))
    print("Test.test2's module: %s" % inspect.getmodule
    (Test.test2))
    print("test.test2's func_name: %s" % test.test2.func_name)
    print("Test.test2's func_name: %s" % Test.test2.func_name)

    if __name__ == "__main__":
    run_tests()

    </EXAMPLE>

    Here is the output that I got in 2.4, 2.5, and 2.6:

    Test is class, test is instance, test2 is method of Test (has
    decorator)
    test's module: <module '__main__' from 'test2.py'>
    Test's module: <module '__main__' from 'test2.py'>
    test.test2's module: <module 'test1' from '/tmp/test1.py'>
    Test.test2's module: <module 'test1' from '/tmp/test1.py'>
    test.test2's func_name: new_function
    Test.test2's func_name: new_function

    If things were working right, then the module for test.test2 would be
    the same as the module for test. I must be missing something, as the
    referenced discussion suggests a simple conclusion. Any ideas?

    -eric
     
    Eric Snow, Jun 16, 2009
    #1
    1. Advertising

  2. Eric Snow

    Eric Snow Guest

    On Jun 16, 9:59 am, Eric Snow <> wrote:
    > Apparently there is a known issue with doctests, in which tests in
    > functions using externally defined decorators are ignored.  The
    > recommended fix is to update the order of checks in the _from_module
    > method of DocTestFinder in the doctest module.  The bug and fix are
    > discussed at the following URLs (and several places in this group):
    >
    > http://bugs.python.org/issue1108htt...ermail/python-list/2007-September/627866.html
    >
    > The fix implies that the inpect.getmodule function will find the
    > module of the function object and not of the decorator.  However, in
    > 2.4 the inspect.getmodule function returns the module of the
    > decorator.  I have subsequently tested this in 2.5 and 2.6, and it
    > also returns the module of the decorator.  As such, the fix for
    > doctests does not work in my tests.  Below is the test code that I
    > used:
    >
    > <EXAMPLE>
    >
    > test1.py
    > ++++++++++++++++++++++++++++
    > def decorator(function):
    >     def new_function(*args, **kwargs):
    >         return function(*args, **kwargs)
    >     return new_function
    >
    > test2.py
    > ++++++++++++++++++++++++++++
    > import test1
    > import inspect
    >
    > class Test(object):
    >     @test1.decorator
    >     def test2(self): pass
    >
    > def run_tests():
    >     test = Test()
    >     test.test2()
    >
    >     print("Test is class, test is instance, test2 is method of Test
    > (has decorator)")
    >     print("test's module:          %s" % inspect.getmodule(test))
    >     print("Test's module:          %s" % inspect.getmodule(Test))
    >     print("test.test2's module:    %s" % inspect.getmodule
    > (test.test2))
    >     print("Test.test2's module:    %s" % inspect.getmodule
    > (Test.test2))
    >     print("test.test2's func_name: %s" % test.test2.func_name)
    >     print("Test.test2's func_name: %s" % Test.test2.func_name)
    >
    > if __name__ == "__main__":
    >     run_tests()
    >
    > </EXAMPLE>
    >
    > Here is the output that I got in 2.4, 2.5, and 2.6:
    >
    > Test is class, test is instance, test2 is method of Test (has
    > decorator)
    > test's module:          <module '__main__' from 'test2.py'>
    > Test's module:          <module '__main__' from 'test2.py'>
    > test.test2's module:    <module 'test1' from '/tmp/test1.py'>
    > Test.test2's module:    <module 'test1' from '/tmp/test1.py'>
    > test.test2's func_name: new_function
    > Test.test2's func_name: new_function
    >
    > If things were working right, then the module for test.test2 would be
    > the same as the module for test.  I must be missing something, as the
    > referenced discussion suggests a simple conclusion.  Any ideas?
    >
    > -eric


    One work-around I found is the following change in example:

    <EXAMPLE>

    test1.py
    ++++++++++++++++++++++++++++
    def decorator(function):
    def new_function(*args, **kwargs):
    return function(*args, **kwargs)
    new_function.__module__ = function.__module__
    new_function.__doc__ = function.__doc__
    new_function.__name__ = function.__name__
    return new_function

    </EXAMPLE>

    However, this seems pretty lame. The doctest module should be able to
    figure out that the docstring belongs is there in the module.

    -eric
     
    Eric Snow, Jun 16, 2009
    #2
    1. Advertising

  3. Eric Snow

    Eric Snow Guest

    On Jun 16, 10:31 am, Christian Heimes <> wrote:
    > Eric Snow schrieb:
    >
    >
    >
    > > Apparently there is a known issue with doctests, in which tests in
    > > functions using externally defined decorators are ignored.  The
    > > recommended fix is to update the order of checks in the _from_module
    > > method of DocTestFinder in the doctest module.  The bug and fix are
    > > discussed at the following URLs (and several places in this group):

    >
    > >http://bugs.python.org/issue1108
    > >http://mail.python.org/pipermail/python-list/2007-September/627866.html

    >
    > > The fix implies that the inpect.getmodule function will find the
    > > module of the function object and not of the decorator.  However, in
    > > 2.4 the inspect.getmodule function returns the module of the
    > > decorator.  I have subsequently tested this in 2.5 and 2.6, and it
    > > also returns the module of the decorator.  As such, the fix for
    > > doctests does not work in my tests.  Below is the test code that I
    > > used:

    >
    > It's not an issue with the doctest module but with your code. You want
    > to use functools.wraps().
    >
    > http://docs.python.org/library/functools.html#functools.wraps
    >
    > Christian


    Unfortunately, I am stuck on 2.4 for now, which does not have the
    functools.

    -eric
     
    Eric Snow, Jun 16, 2009
    #3
  4. On Jun 16, 6:39 pm, Eric Snow <> wrote:
    > On Jun 16, 10:31 am, Christian Heimes <> wrote:
    >
    >
    >
    > > Eric Snow schrieb:

    >
    > > > Apparently there is a known issue with doctests, in which tests in
    > > > functions using externally defined decorators are ignored.  The
    > > > recommended fix is to update the order of checks in the _from_module
    > > > method of DocTestFinder in the doctest module.  The bug and fix are
    > > > discussed at the following URLs (and several places in this group):

    >
    > > >http://bugs.python.org/issue1108
    > > >http://mail.python.org/pipermail/python-list/2007-September/627866.html

    >
    > > > The fix implies that the inpect.getmodule function will find the
    > > > module of the function object and not of the decorator.  However, in
    > > > 2.4 the inspect.getmodule function returns the module of the
    > > > decorator.  I have subsequently tested this in 2.5 and 2.6, and it
    > > > also returns the module of the decorator.  As such, the fix for
    > > > doctests does not work in my tests.  Below is the test code that I
    > > > used:

    >
    > > It's not an issue with the doctest module but with your code. You want
    > > to use functools.wraps().

    >
    > >http://docs.python.org/library/functools.html#functools.wraps

    >
    > > Christian

    >
    > Unfortunately, I am stuck on 2.4 for now, which does not have the
    > functools.
    >
    > -eric


    But you can always use the decorator module: http://pypi.python.org/pypi/decorator
     
    Michele Simionato, Jun 16, 2009
    #4
  5. Eric Snow

    Eric Snow Guest

    On Jun 16, 11:24 am, Michele Simionato <>
    wrote:
    > On Jun 16, 6:39 pm, Eric Snow <> wrote:
    >
    >
    >
    > > On Jun 16, 10:31 am, Christian Heimes <> wrote:

    >
    > > > Eric Snow schrieb:

    >
    > > > > Apparently there is a known issue with doctests, in which tests in
    > > > > functions using externally defined decorators are ignored.  The
    > > > > recommended fix is to update the order of checks in the _from_module
    > > > > method of DocTestFinder in the doctest module.  The bug and fix are
    > > > > discussed at the following URLs (and several places in this group):

    >
    > > > >http://bugs.python.org/issue1108
    > > > >http://mail.python.org/pipermail/python-list/2007-September/627866.html

    >
    > > > > The fix implies that the inpect.getmodule function will find the
    > > > > module of the function object and not of the decorator.  However, in
    > > > > 2.4 the inspect.getmodule function returns the module of the
    > > > > decorator.  I have subsequently tested this in 2.5 and 2.6, and it
    > > > > also returns the module of the decorator.  As such, the fix for
    > > > > doctests does not work in my tests.  Below is the test code that I
    > > > > used:

    >
    > > > It's not an issue with the doctest module but with your code. You want
    > > > to use functools.wraps().

    >
    > > >http://docs.python.org/library/functools.html#functools.wraps

    >
    > > > Christian

    >
    > > Unfortunately, I am stuck on 2.4 for now, which does not have the
    > > functools.

    >
    > > -eric

    >
    > But you can always use the decorator module:http://pypi.python.org/pypi/decorator


    Thanks to both of you. Very helpful.

    So in general should decorators always hide themselves? I am guessing
    not, otherwise this would already be part of their behavior. Still,
    is it the common case to camouflage the decorator like this? If so, I
    would expect it to be the default behavior of decorators.

    -eric
     
    Eric Snow, Jun 16, 2009
    #5
  6. On Tue, 16 Jun 2009 12:04:32 -0700, Scott David Daniels wrote:

    > Eric Snow wrote:
    >> In general should decorators always hide themselves? I am guessing
    >> not, otherwise this would already be part of their behavior. Still, is
    >> it the common case to camouflage the decorator like this? If so, I
    >> would expect it to be the default behavior of decorators.

    >
    > The Python goal is "no magic". So, if you want the stuff wrapped, you
    > do it (as the default traceback shows where the code actually goes). It
    > would be far more complicated to display the truth if decorators
    > defaulted to modifying the builtins, and you had to do magic to remove
    > that part of the decoration.



    I'm afraid I can't understand what you're saying. What do you consider
    "magic"? What's a "default traceback"? What do you mean, "display the
    truth"?


    > A decorator has _very_ simple semantics,
    > while anything that automatically copied attributes would have funny
    > semantics indeed for use by funny decorators like:

    [...]


    functools.wraps() automatically copies attributes:

    >>> import functools
    >>> def dec(func):

    .... @functools.wraps(func)
    .... def inner(*args):
    .... return func(args) + 1
    .... return inner
    ....
    >>> def f(x):

    .... return 1
    ....
    >>> f.attr = "Attribute"
    >>> f = dec(f)
    >>> f(3)

    2
    >>> f.attr

    'Attribute'



    --
    Steven
     
    Steven D'Aprano, Jun 17, 2009
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Arien Malec

    PEP 318 decorators are not Decorators

    Arien Malec, Aug 13, 2004, in forum: Python
    Replies:
    11
    Views:
    571
    Arien Malec
    Aug 16, 2004
  2. Bjorn Tillenius

    unicode in doctests

    Bjorn Tillenius, Nov 29, 2004, in forum: Python
    Replies:
    0
    Views:
    256
    Bjorn Tillenius
    Nov 29, 2004
  3. Thomas Heller

    Running doctests with unittest

    Thomas Heller, Mar 9, 2005, in forum: Python
    Replies:
    1
    Views:
    459
    Jim Sizelove
    Mar 9, 2005
  4. Steven D'Aprano

    Doctests and decorated methods don't get along

    Steven D'Aprano, Feb 6, 2010, in forum: Python
    Replies:
    7
    Views:
    364
    Diez B. Roggisch
    Feb 9, 2010
  5. Michele Simionato
    Replies:
    0
    Views:
    220
    Michele Simionato
    Apr 11, 2010
Loading...

Share This Page