Creating unit tests on the fly

Discussion in 'Python' started by Roy Smith, Apr 8, 2011.

  1. Roy Smith

    Roy Smith Guest

    I've got a suite of unit tests for a web application. There's an
    (abstract) base test class from which all test cases derive:

    class BaseSmokeTest(unittest.TestCase):

    BaseSmokeTest.setUpClass() fetches a UR (from a class attribute
    "route", which must be defined in the derived classes), and there's a
    number of test methods which do basic tests like checking for
    reaonable-looking HTML (parsed with lxml), scanning the server log
    file to make sure there's no error messages or stack dumps, etc.

    Many of the test cases are nothing more than running the base test
    methods on a particular route, i.e.

    class Test_Page_About(BaseSmokeTest):
    route = 'page/about'

    Now, I want to do something a little fancier. I want to get a
    particular page, parse the HTML to find anchor tags containing
    additional URLs which I want to test. It's easy enough to pull out
    the anchors I'm interested in with lxml:

    selector = CSSSelector('div.st_info .st_name > a')
    for anchor in selector(self.tree):
    print anchor.get('href')

    I can even create new test cases from these on the fly with something
    like:

    newClass = type("newClass", (BaseSmokeTest,), {'route': '/my/newly/
    discovered/anchor'})

    (credit to http://jjinux.blogspot.com/2005/03/python-create-new-class-on-fly.html
    for that neat little trick). The only thing I don't see is how I can
    now get unittest.main(), which is already running, to notice that a
    new test case has been created and add it to the list of test cases to
    run. Any ideas on how to do that?

    I suppose I don't strictly need to go the "create a new TestCase on
    the fly" route, but I've already got a fair amount of infrastructure
    set up around that, which I don't want to redo.
    Roy Smith, Apr 8, 2011
    #1
    1. Advertising

  2. On Apr 8, 12:10 pm, Roy Smith <> wrote:
    > I can even create new test cases from these on the fly with something
    > like:
    >
    >  newClass = type("newClass", (BaseSmokeTest,), {'route': '/my/newly/
    > discovered/anchor'})
    >
    > (credit tohttp://jjinux.blogspot.com/2005/03/python-create-new-class-on-fly.html
    > for that neat little trick).  The only thing I don't see is how I can
    > now get unittest.main(), which is already running, to notice that a
    > new test case has been created and add it to the list of test cases to
    > run.  Any ideas on how to do that?


    The basic unittest.main() runner isn't well suited to this task. It
    flows in a pipeline of discovery -> test_suite -> test_runner.

    I think you're going to need a queue of tests, with your own test
    runner consuming the queue, and your on-the-fly test creator running
    as a producer thread.

    Writing your own test runner isn't difficult. 1) wait on the queue
    for a new test case. 2) invoke test_case.run() with a TestResult
    object to hold the result 3) accumulate or report the results 4)
    repeat forever.

    Raymond

    twitter: @raymondh
    Raymond Hettinger, Apr 8, 2011
    #2
    1. Advertising

  3. Roy Smith

    Roy Smith Guest

    In article <>,
    Ben Finney <> wrote:

    > Raymond Hettinger <> writes:
    >
    > > I think you're going to need a queue of tests, with your own test
    > > runner consuming the queue, and your on-the-fly test creator running
    > > as a producer thread.

    >
    > I have found the ‘testscenarios’ library very useful for this: bind a
    > sequence of (name, dict) tuples to the test case class, and each tuple
    > represents a scenario of data fixtures that will be applied to every
    > test case function in the class.
    >
    > <URL:http://pypi.python.org/pypi/test-scenarios>
    >
    > You (the OP) will also find the ‘testing-in-python’ discussion forum
    > <URL:http://lists.idyll.org/listinfo/testing-in-python> useful for this
    > topic.


    That link doesn't work, I assume you meant

    http://pypi.python.org/pypi/testscenarios/0.2

    This is interesting, and a bunch to absorb. Thanks. It might be what
    I'm looking for. For the moment, I'm running the discovery then doing
    something like

    class_name = 'Test_DiscoveredRoute_%s' % cleaned_route_name
    g = globals()
    g[class_name] = type(class_name, bases, new_dict)

    on each discovered route, and calling unittest.main() after I'm done
    doing all that. It's not quite what I need however, so something like
    testscenarios or raymondh's test queue idea might be where this needs to
    go.
    Roy Smith, Apr 9, 2011
    #3
  4. Roy Smith

    Roy Smith Guest

    In article
    <>,
    Raymond Hettinger <> wrote:

    > I think you're going to need a queue of tests, with your own test
    > runner consuming the queue, and your on-the-fly test creator running
    > as a producer thread.
    >
    > Writing your own test runner isn't difficult. 1) wait on the queue
    > for a new test case. 2) invoke test_case.run() with a TestResult
    > object to hold the result 3) accumulate or report the results 4)
    > repeat forever.


    OK, this is working out pretty nicely. The main loop is shaping up to
    to be something like:

    def go(self):
    result = unittest.TestResult()
    while not self.queue.empty():
    route, depth = self.queue.get()
    test_case = self.make_test_case(route)
    suite = unittest.defaultTestLoader. \
    loadTestsFromTestCase(test_case)
    suite.run(result)
    if result.wasSuccessful():
    print "passed"
    else:
    for case, trace in result.failures:
    print case.id()
    d = case.shortDescription()
    if d:
    print d
    print trace
    print '------------------------------------'

    It turns out there's really no reason to put the test runner in its own
    thread. Doing it all in one thread works fine; make_test_case() passes
    self.queue to the newly created TestCase as part of the class dict, and
    one of the test methods in my BaseSmokeTest pushes newly discovered
    route onto the queue. Perhaps not the most efficient way to do things,
    but since most of the clock time is spent waiting for the HTTP server to
    serve up a page, it doesn't matter, and this keeps it simple.

    Thanks for your help!

    PS: After having spent the last 6 years of my life up to my navel in
    C++, it's incredibly liberating to be creating classes on the fly in
    user code :)
    Roy Smith, Apr 11, 2011
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. ChrisC
    Replies:
    5
    Views:
    5,667
    rochnet
    Nov 1, 2009
  2. Replies:
    8
    Views:
    881
  3. Peter Maas
    Replies:
    4
    Views:
    331
    Edvard Majakari
    Mar 8, 2005
  4. dayo
    Replies:
    11
    Views:
    328
    Ilya Zakharevich
    Dec 16, 2005
  5. Brian

    Fly outmenu on the fly

    Brian, Apr 8, 2005, in forum: Javascript
    Replies:
    0
    Views:
    96
    Brian
    Apr 8, 2005
Loading...

Share This Page