Creating unit tests on the fly

[Roy Smith]

I've got a suite of unit tests for a web application. There's an
(abstract) base test class from which all test cases derive:

class BaseSmokeTest(unittest.TestCase):

BaseSmokeTest.setUpClass() fetches a UR (from a class attribute
"route", which must be defined in the derived classes), and there's a
number of test methods which do basic tests like checking for
reaonable-looking HTML (parsed with lxml), scanning the server log
file to make sure there's no error messages or stack dumps, etc.

Many of the test cases are nothing more than running the base test
methods on a particular route, i.e.

class Test_Page_About(BaseSmokeTest):
route = 'page/about'

Now, I want to do something a little fancier. I want to get a
particular page, parse the HTML to find anchor tags containing
additional URLs which I want to test. It's easy enough to pull out
the anchors I'm interested in with lxml:

selector = CSSSelector('div.st_info .st_name > a')
for anchor in selector(self.tree):
print anchor.get('href')

I can even create new test cases from these on the fly with something
like:

newClass = type("newClass", (BaseSmokeTest,), {'route': '/my/newly/
discovered/anchor'})

(credit to http://jjinux.blogspot.com/2005/03/python-create-new-class-on-fly.html
for that neat little trick). The only thing I don't see is how I can
now get unittest.main(), which is already running, to notice that a
new test case has been created and add it to the list of test cases to
run. Any ideas on how to do that?

I suppose I don't strictly need to go the "create a new TestCase on
the fly" route, but I've already got a fair amount of infrastructure
set up around that, which I don't want to redo.

 

[Raymond Hettinger]

I can even create new test cases from these on the fly with something
like:

 newClass = type("newClass", (BaseSmokeTest,), {'route': '/my/newly/
discovered/anchor'})

(credit tohttp://jjinux.blogspot.com/2005/03/python-create-new-class-on-fly.html
for that neat little trick).  The only thing I don't see is how I can
now get unittest.main(), which is already running, to notice that a
new test case has been created and add it to the list of test cases to
run.  Any ideas on how to do that?

The basic unittest.main() runner isn't well suited to this task. It
flows in a pipeline of discovery -> test_suite -> test_runner.

I think you're going to need a queue of tests, with your own test
runner consuming the queue, and your on-the-fly test creator running
as a producer thread.

Writing your own test runner isn't difficult. 1) wait on the queue
for a new test case. 2) invoke test_case.run() with a TestResult
object to hold the result 3) accumulate or report the results 4)
repeat forever.

Raymond

twitter: @raymondh

 

[Roy Smith]

I have found the ‘testscenarios’ library very useful for this: bind a
sequence of (name, dict) tuples to the test case class, and each tuple
represents a scenario of data fixtures that will be applied to every
test case function in the class.

<URL:http://pypi.python.org/pypi/test-scenarios>

You (the OP) will also find the ‘testing-in-python’ discussion forum
<URL:http://lists.idyll.org/listinfo/testing-in-python> useful for this
topic.

That link doesn't work, I assume you meant

http://pypi.python.org/pypi/testscenarios/0.2

This is interesting, and a bunch to absorb. Thanks. It might be what
I'm looking for. For the moment, I'm running the discovery then doing
something like

class_name = 'Test_DiscoveredRoute_%s' % cleaned_route_name
g = globals()
g[class_name] = type(class_name, bases, new_dict)

on each discovered route, and calling unittest.main() after I'm done
doing all that. It's not quite what I need however, so something like
testscenarios or raymondh's test queue idea might be where this needs to
go.

 

[Roy Smith]

I think you're going to need a queue of tests, with your own test
runner consuming the queue, and your on-the-fly test creator running
as a producer thread.

Writing your own test runner isn't difficult. 1) wait on the queue
for a new test case. 2) invoke test_case.run() with a TestResult
object to hold the result 3) accumulate or report the results 4)
repeat forever.

OK, this is working out pretty nicely. The main loop is shaping up to
to be something like:

def go(self):
result = unittest.TestResult()
while not self.queue.empty():
route, depth = self.queue.get()
test_case = self.make_test_case(route)
suite = unittest.defaultTestLoader. \
loadTestsFromTestCase(test_case)
suite.run(result)
if result.wasSuccessful():
print "passed"
else:
for case, trace in result.failures:
print case.id()
d = case.shortDescription()
if d:
print d
print trace
print '------------------------------------'

It turns out there's really no reason to put the test runner in its own
thread. Doing it all in one thread works fine; make_test_case() passes
self.queue to the newly created TestCase as part of the class dict, and
one of the test methods in my BaseSmokeTest pushes newly discovered
route onto the queue. Perhaps not the most efficient way to do things,
but since most of the clock time is spent waiting for the HTTP server to
serve up a page, it doesn't matter, and this keeps it simple.

Thanks for your help!

PS: After having spent the last 6 years of my life up to my navel in
C++, it's incredibly liberating to be creating classes on the fly in
user code