Ordering tests in a testsuite

[Ulrich Eckhardt]

Hello!

I'm currently working on a testsuite using Python's unittest library. This
works all good and fine, but there's one thing where I haven't seen an
elegant solution to yet, and that is the ordering. Currently, it takes all
classes and orders them alphabetically and then takes all test functions
therein and runs those alphabetically, too. However, sometimes it doesn't
make sense to run test_bar() if test_foo() already failed, because they
basically build upon each other. However, test_bar() is still run first,
and test_foo() second.

What I sometimes do is to simply number them, like test_1_foo() and
test_2_bar(), but that seems ugly. I'm not even looking for a way to
express complicated relations between tests, but is there a less ugly way
to explicitly order them?

Cheers!

Uli

 

[Peter Otten]

I'm currently working on a testsuite using Python's unittest library. This
works all good and fine, but there's one thing where I haven't seen an
elegant solution to yet, and that is the ordering. Currently, it takes all
classes and orders them alphabetically and then takes all test functions
therein and runs those alphabetically, too. However, sometimes it doesn't
make sense to run test_bar() if test_foo() already failed, because they
basically build upon each other. However, test_bar() is still run first,
and test_foo() second.

What I sometimes do is to simply number them, like test_1_foo() and
test_2_bar(), but that seems ugly. I'm not even looking for a way to
express complicated relations between tests, but is there a less ugly way
to explicitly order them?

You can pass unittest.main() a custom test loader which in turn can define a
custom sortTestMethodsUsing() method. Here's a fancy example:

import unittest

def _toposort(data):
# adapted from http://code.activestate.com/recipes/577413-topological-
sort/
for k, v in data.items():
v.discard(k) # Ignore self dependencies
extra_items_in_deps = reduce(set.union, data.values()) -
set(data.keys())
data.update((item, set()) for item in extra_items_in_deps)
while True:
ordered = set(item for item,dep in data.items() if not dep)
if not ordered:
break
for item in sorted(ordered):
yield item
data = dict((item, (dep - ordered)) for item,dep in data.items()
if item not in ordered)

def _get_name(func):
try:
return func.__name__
except AttributeError:
pass
assert isinstance(func, str)
return func

class Cmp(object):
def __init__(self):
self._deps = {}
def depends(self, f, names):
k = _get_name(f)
assert k not in self._deps
self._deps[k] = set(_get_name(n) for n in names)
return f
def make_loader(self):
lookup = dict((name, index) for index, name
in enumerate(_toposort(self._deps)))
def compare(a, b):
return cmp(lookup[a], lookup)
class TestLoader(unittest.TestLoader):
pass
loader = TestLoader()
loader.sortTestMethodsUsing = compare
return loader
def depends_on(self, *names):
def d(f):
return self.depends(f, names)
return d


c = Cmp()
depends_on = c.depends_on

class A(unittest.TestCase):
@depends_on("test_beta")
def test_alpha(self):
pass
@depends_on("test_gamma")
def test_beta(self):
pass
def test_gamma(self):
pass
@depends_on(test_gamma)
def test_delta(self):
pass
@depends_on(test_alpha, test_beta)
def test_epsilon(self):
pass

if __name__ == "__main__":
unittest.main(testLoader=c.make_loader())

Peter

 

[Mike Kent]

But sometimes you just wanna do it the way you wanna do it. If you
name your tests like 'test_01_yadda' and test_02_whatever', then they
will be run in the order you want, as given by the numbers.

 

[Steven D'Aprano]

That's a mistake. If the success of ‘test_bar’ depends on the result of
‘test_foo’, then it's not an independent test and therefore isn't a unit
test.

I don't think that is what Ulrich means to imply. I don't think he means
that test_bar uses the result of test_foo, but only that if test_foo
fails then test_bar has no hope of succeeding.

To give an example, suppose you have:

class MyClass:
def __init__(self, cheese):
self.cheese = cheese

class MyTests(unittest.TestCase):
def test_create(self):
# Test that instances can be created correctly.
self.assert_raises(TypeError, MyClass)
x = MyClass("cheddar") # will fail if MyClass can't be created
def test_cheese(self):
# Test that instances have a cheese attribute.
x = MyClass("swiss")
self.assertEquals(x.cheese, "swiss")


If test_create fails, then so will test_cheese, not because the later
test is dependent on the first, but because creation is more fundamental
than attribute access. I think the OP wants to skip test_cheese in the
even that test_create fails.