unittest: collecting tests from many modules?

[Jorgen Grahn]

I have a set of tests in different modules: test_foo.py, test_bar.py and so
on. All of these use the simplest possible internal layout: a number of
classes containing test*() methods, and the good old lines at the end:

if __name__ == "__main__":
unittest.main()

This is great, because each of the modules are runnable, and I can select
classes or tests to run on the commandline if I want to. However, running
all the tests from e.g. a Makefile is not that fun; I don't get a single
pass/fail summary across the modules.

What's the best way of creating a test.py which
- aggregates the tests from all the test_*.py modules?
- doesn't require me to enumerate all the test classes in test.py
(forcing each module to define test_foo.theSuite or someting would
be OK though)
- retains the ability to select tests and verbosity (-q, -v) from the
command line?

Something like:

import unittest
import test_foo
import test_bar

if __name__ == "__main__":
unittest.main(modules = ['test_foo',
'test_bar'])

Seems to me this should be possible, since all the logic for doing it /is/
there for the local module; I'd assume there would be a way to make unittest
search additional modules for test classes. But my head starts spinning
when I read the source code ...

/Jorgen

 

[John Roth]

I have a set of tests in different modules: test_foo.py, test_bar.py and so
on. All of these use the simplest possible internal layout: a number of
classes containing test*() methods, and the good old lines at the end:

if __name__ == "__main__":
unittest.main()

This is great, because each of the modules are runnable, and I can select
classes or tests to run on the commandline if I want to. However, running
all the tests from e.g. a Makefile is not that fun; I don't get a single
pass/fail summary across the modules.

What's the best way of creating a test.py which
- aggregates the tests from all the test_*.py modules?
- doesn't require me to enumerate all the test classes in test.py
(forcing each module to define test_foo.theSuite or someting would
be OK though)
- retains the ability to select tests and verbosity (-q, -v) from the
command line?

I use your second point: I build a TestAll module. However,
I'm going to look at building the test suite using the TestLoader
class in the next version of PyFit. It sholdn't be all that difficult
to find all the Test*.py modules in a directory, import them and
use the TestLoader class to add them to the test suite.

Or, for that matter, to use reflection to find all the classes that
derive from TestCase and add them to the suite manually. That
has the advantage that one could then select classes according
to some parameter.

John Roth

Something like:

import unittest
import test_foo
import test_bar

if __name__ == "__main__":
unittest.main(modules = ['test_foo',
'test_bar'])

Seems to me this should be possible, since all the logic for doing it /is/
there for the local module; I'd assume there would be a way to make
unittest
search additional modules for test classes. But my head starts spinning
when I read the source code ...

/Jorgen

 

[George Sakkis]

I have a set of tests in different modules: test_foo.py, test_bar.py and so
on. All of these use the simplest possible internal layout: a number of
classes containing test*() methods, and the good old lines at the end:

if __name__ == "__main__":
unittest.main()

This is great, because each of the modules are runnable, and I can select
classes or tests to run on the commandline if I want to. However, running
all the tests from e.g. a Makefile is not that fun; I don't get a single
pass/fail summary across the modules.

What's the best way of creating a test.py which
- aggregates the tests from all the test_*.py modules?
- doesn't require me to enumerate all the test classes in test.py
(forcing each module to define test_foo.theSuite or someting would
be OK though)
- retains the ability to select tests and verbosity (-q, -v) from the
command line?

Something like:

import unittest
import test_foo
import test_bar

if __name__ == "__main__":
unittest.main(modules = ['test_foo',
'test_bar'])

Seems to me this should be possible, since all the logic for doing it /is/
there for the local module; I'd assume there would be a way to make unittest
search additional modules for test classes. But my head starts spinning
when I read the source code ...

I had written a script to do something close to this; currently it
doesn't do any kind of aggregation, but it should be easy to extend it
as you like. What I don't like is the way it currently works: it
replaces sys.modules['__main__'] for each unit test and then it
execfile()s it, which seems like a hack. I didn't look into unittest's
internals in case there is a more elegant way around this; if there
isn't, a future version of unittest should address the automatic
aggregation of tests, as py.test does already.

The link to the script is http://rafb.net/paste/results/V0y16g97.html.

Hope this helps,
George

 

[Scott David Daniels]

I have a set of tests in different modules: test_foo.py, test_bar.py and so
on. All of these use the simplest possible internal layout: a number of
classes containing test*() methods, and the good old lines at the end:
if __name__ == "__main__":
unittest.main()
....

What's the best way of creating a test.py which
- aggregates the tests from all the test_*.py modules?
- doesn't require me to enumerate all the test classes in test.py
(forcing each module to define test_foo.theSuite or someting would
be OK though)
- retains the ability to select tests and verbosity (-q, -v) from the
command line?

Something like:

import unittest
import test_foo
import test_bar

if __name__ == "__main__":
unittest.main(modules = ['test_foo',
'test_bar'])

Seems to me this should be possible, since all the logic for doing it /is/
there for the local module; I'd assume there would be a way to make unittest
search additional modules for test classes. But my head starts spinning
when I read the source code ...

/Jorgen

How about some variant of:

import unittest
import test_foo, test_bar, ...

def set_globals(modules):
glbl = globals()
for module in modules:
modprefix = module.__name__[5:] + '__'
for element in dir(module):
data = getattr(module, element)
if isinstance(data, type) and issubclass(data,
unittest.TestCase):
glbl[modprefix + element] = data

if __name__ == "__main__":
module = type(unittest)
set_globals([mod for name, mod in globals().items()
if name.lower().beginswith('test')
and isinstance(mod, module)])
unittest.main()


--Scott David Daniels
(e-mail address removed)

 

[Bengt Richter]

I have a set of tests in different modules: test_foo.py, test_bar.py and so
on. All of these use the simplest possible internal layout: a number of
classes containing test*() methods, and the good old lines at the end:

if __name__ == "__main__":
unittest.main()

This is great, because each of the modules are runnable, and I can select
classes or tests to run on the commandline if I want to. However, running
all the tests from e.g. a Makefile is not that fun; I don't get a single
pass/fail summary across the modules.

What's the best way of creating a test.py which
- aggregates the tests from all the test_*.py modules?
- doesn't require me to enumerate all the test classes in test.py
(forcing each module to define test_foo.theSuite or someting would
be OK though)
- retains the ability to select tests and verbosity (-q, -v) from the
command line?

Something like:

import unittest
import test_foo
import test_bar

if __name__ == "__main__":
unittest.main(modules = ['test_foo',
'test_bar'])

Seems to me this should be possible, since all the logic for doing it /is/
there for the local module; I'd assume there would be a way to make unittest
search additional modules for test classes. But my head starts spinning
when I read the source code ...

I had written a script to do something close to this; currently it
doesn't do any kind of aggregation, but it should be easy to extend it
as you like. What I don't like is the way it currently works: it
replaces sys.modules['__main__'] for each unit test and then it
execfile()s it, which seems like a hack. I didn't look into unittest's
internals in case there is a more elegant way around this; if there
isn't, a future version of unittest should address the automatic
aggregation of tests, as py.test does already.

The link to the script is http://rafb.net/paste/results/V0y16g97.html.

I think if you execfile a script and supply the global dict initialized
to {'__name__':'__main__'} then I think that will satisfy the if __name__ ... condition
and the whole thing will run as if executed interactively from the command line.
So IWT you could just loop through a list of test modules doing that.
Haven't tried it though. E.g., maybe there are nasty reload problems for
shared modules among different tests?

Regards,
Bengt Richter

 

[Benji York]

What's the best way of creating a test.py which
- aggregates the tests from all the test_*.py modules?

You might want to check out the test runner in Zope 3
(svn://svn.zope.org/repos/main/Zope3/trunk/src/zope/app/testing)

It aggregates test reporting, does code coverage, lets you select
subsets of the tests to run, as well as control verbosity.

And, if you feel experimental you might want to preview the new Zope
test runner currently under development
(svn://svn.zope.org/repos/main/zope.testing).

 

[George Sakkis]

I had written a script to do something close to this; currently it
doesn't do any kind of aggregation, but it should be easy to extend it
as you like. What I don't like is the way it currently works: it
replaces sys.modules['__main__'] for each unit test and then it
execfile()s it, which seems like a hack. I didn't look into unittest's
internals in case there is a more elegant way around this; if there
isn't, a future version of unittest should address the automatic
aggregation of tests, as py.test does already.

I think if you execfile a script and supply the global dict initialized
to {'__name__':'__main__'} then I think that will satisfy the if __name__ ... condition
and the whole thing will run as if executed interactively from the command line.

Yes, execfile is called with {'__name__':'__main__'} for globals but
that's not enough because __main__ is already bound to the test.py
script, not the tested module. Even worse, after each test,
sys.modules['__main__'] seems to have to be reset to the original
__main__; otherwise os (and I guess other globals) is bound to None !
I'm not sure if this is normal or I missed something basic.

Regards,
George


PS: I reposted the script at
http://rafb.net/paste/results/3f1PIZ70.html.

 

[Jorgen Grahn]

[regarding module unittest]

What's the best way of creating a test.py which
- aggregates the tests from all the test_*.py modules?
- doesn't require me to enumerate all the test classes in test.py
(forcing each module to define test_foo.theSuite or someting would
be OK though)
- retains the ability to select tests and verbosity (-q, -v) from the
command line?

Thanks for all the input. Three weeks later I stumble across this thread
again and notice I didn't report what I ended up with. I ended up doing the
thing I wanted to avoid in the first place: hardcoding everything.

import unittest

import test_bibdb
import test_citefmt
import test_person
import test_refsfmt

suite = unittest.TestSuite()
suite.addTest(test_bibdb.suite())
suite.addTest(test_citefmt.suite())
suite.addTest(test_person.suite())
suite.addTest(test_refsfmt.suite())

if __name__ == "__main__":
unittest.TextTestRunner(verbosity=2).run(suite)

My test modules are, after all, a fairly fixed set, and if I want to run
specific tests, or with a specific verbosity, I can execute the individual
test modules.

BR,
/Jorgen