Unittest - How do I code lots of simple tests

[Paul Moore]

One of the things I really dislike about Unittest (compared, say, to a
number of adhoc testing tricks I've used in the past, and to Perl's
"standard" testing framework) is that the testcase-as-a-class model
tends to imply a relatively high granularity in testing.

A good example of this comes from "Dive Into Python"
(http://diveintopython.org) section 7.3. Here, the author has written
a module which converts numbers to Roman numerals. The test case is
then

class KnownValues(unittest.TestCase):
knownValues = ( (1, 'I'), ... # 50-plus specific cases

def testToRomanKnownValues(self):
"""toRoman should give known result with known input"""
for integer, numeral in self.knownValues:
result = roman.toRoman(integer)
self.assertEqual(numeral, result)

Now, to my mind, the big problem here is that this *isn't* one test,
but rather 50 (or more). OK, the same checks are done if everything
succeeds, but

1. If a test fails, the rest are skipped! If there's a pattern to the
failures (the code handles numbers ending in 4 and 9 wrongly, for
example) it's much easier to find if all of the checks are
reported.
2. Psychologically, "52 tests succeeded" is a much bigger boost than
"1 test succeeded" (albeit this test covered 52 cases). And it's
not just psychology - we really *did* test 52 distinct conditions.

The only way I can see of producing the "true" number of tests is via
some form of ugly hack like

test_values = ((1, 'I'), ...)

class KnownValues(unittest.TestCase):
pass

for arabic, roman in test_values:
def test(self):
result = roman.toRoman(arabic)
self.assertEqual(roman, result)
setattr(KnownValues, 'test_%s_%s' % (arabic, roman), test)

But I can't really see that as the "right approach".

Can anyone suggest a more reasonable way of running this sort of
table-driven test via unittest?

Maybe I should use doctest instead, but to be honest, I prefer the
overall infrastructure of unittest (for real tests). It's just this
particular issue that really bugs me...

Paul.

 

[Peter Hansen]

Can anyone suggest a more reasonable way of running this sort of
table-driven test via unittest?

Why not just extend self.assertEqual() and use your own check, with
additional logic as required to increment counters or add items
to the list of passing tests. Then put a final check of the number
of passing tests or something like that at the end to make sure
things worked overall.
For example:

class KnownValues(unittest.TestCase):
def setUp(self):
self.passCount = 0

def checkValue(self, expected, result):
if expected == result:
self.passCount += 1
else:
# left as exercise to the reader, but pass would work...

def testToRomanKnownValues(self):
for integer, numeral in self.knownValues:
result = roman.toRoman(integer)
self.checkValue(numeral, result)
self.assertEqual(len(self.knownValues), self.passCount)

No, you don't get the psychologically affirming "52 tests passed!"
without changes to the TestRunner, but I assume the non-cosmetic part
of this is more your concern right now...

-Peter

 

[David Goodger]

Take a look at Docutils' test/DocutilsTestSupport.py for ideas. The
CustomTestSuite and CustomTestCase classes provide support for named
data-driven tests. Most of Docutils' tests are data-driven.

> unittest is not written with subclassing in mind, except for the
> limited subclassing that is documented. (And it uses
> double-underscore variables, like it's just *trying* to piss me off!
> Double-underscore variables are so arrogant and patronizing.

All very true. Double-underscores ought to be banned from the
standard library. They inevitably get in the way because no matter
how well a class is written, somebody is going to want to subclass it
in a way the original author never considered.

 

[Jeremy Fincher]

1. If a test fails, the rest are skipped! If there's a pattern to the
failures (the code handles numbers ending in 4 and 9 wrongly, for
example) it's much easier to find if all of the checks are
reported.

That's true, but most of the time when I test, I prefer that behavior.
Many of my asserts depend on the success of the assert prior to
them, so I want the test to fail as soon as one of the asserts has
failed.

It'd be nice to have both possibilities, though.

Jeremy

 

[Miki Tebeka]

Hello Paul,

test_values = ((1, 'I'), ...)

class KnownValues(unittest.TestCase):
pass

for arabic, roman in test_values:
def test(self):
result = roman.toRoman(arabic)
self.assertEqual(roman, result)
setattr(KnownValues, 'test_%s_%s' % (arabic, roman), test)

But I can't really see that as the "right approach".

On reason that it won't do what you want
All the tests will check the last value in test_values. (Something to
do with binding rules, can't recall how to solve)

Try:
--- i2r.py ---
#!/usr/bin/env python
from unittest import TestCase, makeSuite, main

class Roman:
def toRoman(self, i):
return { 1 : "I",
2 : "II",
3 : "III",
4 : "IV",
5 : "V"}
roman = Roman()

class KnownValues(TestCase):
pass

test_values = ((1, "I"), (2, "II"), (3, "III"), (4, "IV"), (5, "V"))
for a, r in test_values:
def test(self):
print a, r
result = roman.toRoman(a)
self.assertEqual(r, result)
setattr(KnownValues, "test_%s_%s" % (a, r), test)

test_suite = makeSuite(KnownValues, "test_")

if __name__ == "__main__":
main()
--- i2r.py ---

$ python i2r.py
5 V
..5 V
..5 V
..5 V
..5 V
..
----------------------------------------------------------------------
Ran 5 tests in 0.000s

OK


HTH.
Miki

 

[Peter Otten]

Hello Paul,

test_values = ((1, 'I'), ...)

class KnownValues(unittest.TestCase):
pass

for arabic, roman in test_values:
def test(self):
result = roman.toRoman(arabic)
self.assertEqual(roman, result)
setattr(KnownValues, 'test_%s_%s' % (arabic, roman), test)

But I can't really see that as the "right approach".

On reason that it won't do what you want
All the tests will check the last value in test_values. (Something to
do with binding rules, can't recall how to solve)

Try:
--- i2r.py ---
#!/usr/bin/env python
from unittest import TestCase, makeSuite, main

class Roman:
def toRoman(self, i):
return { 1 : "I",
2 : "II",
3 : "III",
4 : "IV",
5 : "V"}
roman = Roman()

class KnownValues(TestCase):
pass

test_values = ((1, "I"), (2, "II"), (3, "III"), (4, "IV"), (5, "V"))
for a, r in test_values:
def test(self):

Change the above line to

def test(self, a=a, r=r):

or you will perform the test five times with (5, "V").

print a, r
result = roman.toRoman(a)
self.assertEqual(r, result)
setattr(KnownValues, "test_%s_%s" % (a, r), test)

test_suite = makeSuite(KnownValues, "test_")

if __name__ == "__main__":
main()
--- i2r.py ---

I like the idea, once the little error is removed. In general, I think the
unit test code should be as simple as possible. Otherwise we would need
unit tests for unit tests for...

Peter

 

[Duncan Booth]

But I can't really see that as the "right approach".

Can anyone suggest a more reasonable way of running this sort of
table-driven test via unittest?

Ok, how about the file below.
It uses a metaclass to generate dynamic tests from a table. I deliberately
wrote the tests so that one fails, when run it will tell you that
test_roman_v failed so you can see that the names get handled properly.

Output:
D:\temp>tabletest.py --verbose
testMe (__main__.MyTests) ... ok
test_roman_i (__main__.MyTests) ... ok
test_roman_v (__main__.MyTests) ... FAIL
test_roman_x (__main__.MyTests) ... ok

======================================================================
FAIL: test_roman_v (__main__.MyTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:\temp\tabletest.py", line 21, in test
self.assertEquals(roman, 'x')
File "D:\Python23\lib\unittest.py", line 302, in failUnlessEqual
raise self.failureException, \
AssertionError: 'v' != 'x'

----------------------------------------------------------------------
Ran 4 tests in 0.020s

FAILED (failures=1)


Note that the metaclass itself is reusable. The factory
function tableDrivenTests, although defined within the class is a function,
not a method, and cannot access any members of the class (since the class
does not yet exist at the time it is called). The table itself either has
to be created inside the tableDrivenTests function, or global.
The factory function simply returns a dictionary of functions which are
added to the class, note that the keyname in the dictionary is important so
far as the unittest code is concerned, not the original name of the
function.

Also be sure to pass parameters into the test function using default
parameters as nested scopes will get the values left at the end of the loop
(so you might end up with lots of tests that all do the same thing). I
mention that here because I did exactly that writing the code.

---- begin tabletest.py ----
class MetaTableTest(type):
def __new__(metacls, name, bases, dict):
factory = dict['tableDrivenTests']
dict.update(factory())
return super(MetaTableTest, metacls).__new__(metacls, name, bases,
dict)

import unittest

class MyTests(unittest.TestCase):
__metaclass__ = MetaTableTest


def tableDrivenTests():
'''Return a dictionary of additional test functions'''
knownValues = (1,'i'), (5, 'v'), (10, 'x')
table = {}
for arabic, roman in knownValues:
def test(self, arabic=arabic, roman=roman):
if arabic==1:
self.assertEquals(roman, 'i')
else:
self.assertEquals(roman, 'x')

table['test_roman_%s' % roman] = test
return table

def testMe(self):
self.assert_(True)


if __name__=='__main__':
unittest.main()

---- end tabletest.py ----

 

[Paul Moore]

Why not just extend self.assertEqual() and use your own check, with
additional logic as required to increment counters or add items
to the list of passing tests. Then put a final check of the number
of passing tests or something like that at the end to make sure
things worked overall.
For example:

class KnownValues(unittest.TestCase):
def setUp(self):
self.passCount = 0

def checkValue(self, expected, result):
if expected == result:
self.passCount += 1
else:
# left as exercise to the reader, but pass would work...

def testToRomanKnownValues(self):
for integer, numeral in self.knownValues:
result = roman.toRoman(integer)
self.checkValue(numeral, result)
self.assertEqual(len(self.knownValues), self.passCount)

No, you don't get the psychologically affirming "52 tests passed!"
without changes to the TestRunner, but I assume the non-cosmetic part
of this is more your concern right now...

I don't see the remainder of the problem as "non-cosmetic". The error
report I get (or rather the information it offers) is

1 test failed - pass count is 25 instead of 52.

But that doesn't tell me *which* tests failed.

The key point here is that I'm NOT running one test - I really am
running 52 distinct tests. OK, they are all very similar, differing
only in the data - but for pity's sake, isn't that what an object
oriented structure is supposed to make easy???

Paul.

 

[Anthony Briggs]

I don't see the remainder of the problem as "non-cosmetic". The error
report I get (or rather the information it offers) is

1 test failed - pass count is 25 instead of 52.

But that doesn't tell me *which* tests failed.

Are you running the tests verbosely? eg. with a -v argument under
UNIX, or as specified in the docs
<http://www.python.org/doc/current/lib/minimal-example.html>? I get
the following output when using Miki's code (with a deliberate error
thrown in, and the print statement commented out):

bash-2.05b$ ./test.py -v
test_1_I (__main__.KnownValues) ... ok
test_2_II (__main__.KnownValues) ... ok
test_3_II (__main__.KnownValues) ... FAIL
test_4_IV (__main__.KnownValues) ... ok
test_5_V (__main__.KnownValues) ... ok

======================================================================
FAIL: test_3_II (__main__.KnownValues)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./test.py", line 21, in test
self.assertEqual(r, result)
File "/usr/local/lib/python2.3/unittest.py", line 292, in failUnlessEqual
raise self.failureException, \
AssertionError: 'II' != 'III'

----------------------------------------------------------------------
Ran 5 tests in 0.055s

FAILED (failures=1)
bash-2.05b$

The key point here is that I'm NOT running one test - I really am
running 52 distinct tests. OK, they are all very similar, differing
only in the data - but for pity's sake, isn't that what an object
oriented structure is supposed to make easy???

Well, you're testing one aspect of the code. It's really just a
question of how you think about your tests.

Anthony

 

[Paul Moore]

Take a look at Docutils' test/DocutilsTestSupport.py for ideas. The
CustomTestSuite and CustomTestCase classes provide support for named
data-driven tests. Most of Docutils' tests are data-driven.

Urk. That's hairy stuff. Thanks for the pointer, I'll do some research.

But I still think that this sort of thing should be easy :-(

Paul.

 

[Peter Hansen]

Can anyone suggest a more reasonable way of running this sort of
table-driven test via unittest?

[snip]
No, you don't get the psychologically affirming "52 tests passed!"
without changes to the TestRunner, but I assume the non-cosmetic part
of this is more your concern right now...

I don't see the remainder of the problem as "non-cosmetic". The error
report I get (or rather the information it offers) is

1 test failed - pass count is 25 instead of 52.

But that doesn't tell me *which* tests failed.

The key point here is that I'm NOT running one test - I really am
running 52 distinct tests. OK, they are all very similar, differing
only in the data - but for pity's sake, isn't that what an object
oriented structure is supposed to make easy???

Well, look at it this way. Using the built-in assertEquals() and
similar functions is a way of explicitly asking for a test method to
abort, and for the framework to continue on with the next test
method. If you don't want that behaviour, nothing's forcing you
to use assertEqual().

Instead, just write up your own comparison routine, which doesn't
abort, and have verbose output for each failure. Something I've
done repeatedly in the past is to have a routine which says simply
(upon failure) "case %s: expected %s, result %s" and then substitute
in the test case input, the expected value, and the actual result.

The part that is "cosmetic" is insisting that this has to result
in the framework reporting the as individual "test" failures. To
do that, you need more extensive modifications, because unittest
has a clear, simple definition of what constitutes a test, and
individual comparisons inside a testMethod() are not it... it's the
whole test method that is a test.

After all, just because a test has two self.assertEquals() and a single
self.assert_() doesn't necessarily mean it's *three* tests.

As Anthony B. wrote, you're testing one "aspect" or something... don't
think of tests as calls to assertXxxx() methods, think of them as
collections of such calls.

-Peter

 

[Emile van Sebille]

David Goodger <[email protected]> wrote in message

Urk. That's hairy stuff. Thanks for the pointer, I'll do some research.

But I still think that this sort of thing should be easy :-(

If you find unnittest too heavy handed, you may want to take a look at
doctest. Essentially, doctest will test code found in docstrings and
compare results. This allows you to paste an interactive session into your
module.

HTH,

Emile van Sebille
(e-mail address removed)

 

[Garth T Kidd]

Take a look at Docutils' test/DocutilsTestSupport.py for ideas. The

CustomTestSuite and CustomTestCase classes provide support for named
data-driven tests. Most of Docutils' tests are data-driven.

Ha! You beat me to it. And here I was ready to post my article on self-
loading test suites:

http://www.pycs.net/users/0000088/stories/7.html

Apart from the default TestLoader not picking up TestSuite subclasses,
which can be fixed with a modified main test running script, it all
works pretty well.

All very true. Double-underscores ought to be banned from the
standard library. They inevitably get in the way because no matter
how well a class is written, somebody is going to want to subclass it
in a way the original author never considered.

Amen to that. I complain in my article about having to hack around
unittest's double-underscored attributes. Worse, I never found a way to
call double-underscored methods. Bah humbug.

Back to tests, though; I imagine it wouldn't take much to merge a few of
these wheel implementations and produce a new, improved unittest.

Regards,
Garth.