unittest: new reporting category "skipped"

[Remy Blank]

Hello unittest users,

In a project I am working on, I have a series of tests that have
to be run as root, and others as a normal user.

One solution is to separate the tests into two different files,
and run the right one. I don't like this too much, as I prefer
to group tests by "function".

Another solution is to build the test suite differently, depending
on the user running it. However, this requires marking the individual
tests with a "category", and have a custom test loader instantiate
only the right ones. I don't have a good solution for this marking.

A third solution, and the one I would like to have feedback about,
is to add the notion of a "skipped" test to unittest. Currently,
test results are given in terms of tests having either succeeded,
failed, or generated an error. What about adding a fourth result
type, skipped, that is also reported at the end of the suite?

I imagine something like this:

class AnythingTest(unittest.TestCase):
def testSomethingAsRoot(self):
self.skipIf(os.geteuid() != 0, "Must be root")

self.assertEqual(...)
...

def testSomethingAsNormalUser(self):
self.skipIf(os.geteuid() == 0, "Must be normal user")

self.assertEqual(...)
...

skipIf() would throw a skippedTestException that would be caught
in TestCase.__call__(). TestResult would be extended with an
addSkipped() function. The reporting in text mode could be as follows:

FAILED (failures=1, skipped=7)

or

OK (skipped=7)

Comments? Ideas?

If the echo is positive, I will make an initial implementation and
provide a patch for feedback.

-- Remy


Remove underscore and suffix in reply address for a timely response.

 

[Roy Smith]

In a project I am working on, I have a series of tests that have
to be run as root, and others as a normal user.

Having just completed a project which required testing as root, I can
sympathize with you completely! If at all possible, try to avoid
getting painted into that particular corner.

Is it possible that by building the appropriate test scaffolding, you
could run the tests as a normal user? For example, let's say the reason
you need to be root is because you are reading a file which is mode 600.
You could then do something like this:

class myFile:
"""Stub replacement for file class. Flesh this out enough to
supply whatever functionality your unit tests require."""

def __init__ (self, filename):
self.data = """this is line one
this is line two
this is line three"""

def read (self):
for line in self.data:
yield line

def myFileFactory (filename):
return myFile (filename)

file = myFileFactory

It's not always possible or practical to do that, but it's worth
thinking about.

Assuming this isn't going to work for you, then I do like your idea of
having a way to skip tests. Sounds like you want to subclass
TestResult, and add a addSkipped() method. You'd then probably need to
also subclass the appropriate TestRunner class to report the skipped
tests.

Another way would be to subclass TestSuite and override addTest(), but
my fear here is that skipped tests would just get quietly dropped. You
really want them to be tracked and alerted in the final report. You're
done testing when you get no failures and no skipped tests. Unless you
have a way to report the skips up to the top level TestRunner, it's too
easy to lose track of the skips.

Your idea of signaling skips with exceptions seems reasonable. You
might want to have more than one kind:

class SkippedTest
class NotRoot (SkippedTest)
class MissingTestResource (SkippedTest)
class TakesTooLong (SkippedTest)

Of course, the unit test gurus would probably be horrified at the whole
idea of skipping tests. A test either passes or fails, and you run
every test, every time. Turn all the dials up to ten, and all that.
It's your project, you decide what makes sense.

 

[George Yoshida]

Hello unittest users,

A third solution, and the one I would like to have feedback about,
is to add the notion of a "skipped" test to unittest. Currently,
test results are given in terms of tests having either succeeded,
failed, or generated an error. What about adding a fourth result
type, skipped, that is also reported at the end of the suite?

I imagine something like this:

class AnythingTest(unittest.TestCase):
def testSomethingAsRoot(self):
self.skipIf(os.geteuid() != 0, "Must be root")

self.assertEqual(...)
...

def testSomethingAsNormalUser(self):
self.skipIf(os.geteuid() == 0, "Must be normal user")

self.assertEqual(...)
...

skipIf() would throw a skippedTestException that would be caught
in TestCase.__call__(). TestResult would be extended with an
addSkipped() function. The reporting in text mode could be as follows:

FAILED (failures=1, skipped=7)

or

OK (skipped=7)

Comments? Ideas?

If the echo is positive, I will make an initial implementation and
provide a patch for feedback.

Check out TestSkipped.
It's already implemented as a part of utility functions for
unittest module and heavily used in Lib/test directory.
Here's a simple example:

from test.test_support import TestSkipped

class AnythingTest(unittest.TestCase):
def testSomethingAsRoot(self):
if os.geteuid() != 0:
raise TestSkipped, "Must be root"

self.assertEqual(...)
...

def testSomethingAsNormalUser(self):
if os.geteuid() == 0:
raise TestSkipped, "Must be normal user"
self.skipIf(os.geteuid() == 0, "Must be normal user")

self.assertEqual(...)
...

George

 

[Bernhard Herzog]

I imagine something like this:

class AnythingTest(unittest.TestCase):
def testSomethingAsRoot(self):
self.skipIf(os.geteuid() != 0, "Must be root")

self.assertEqual(...)
...

def testSomethingAsNormalUser(self):
self.skipIf(os.geteuid() == 0, "Must be normal user")

self.assertEqual(...)
...

skipIf() would throw a skippedTestException that would be caught
in TestCase.__call__(). TestResult would be extended with an
addSkipped() function. The reporting in text mode could be as follows:

FAILED (failures=1, skipped=7)

or

OK (skipped=7)

Amazing, this is practically what I implemented in Thuban (see sig for
URL). It wasn't very difficult to do that by deriving some custom
classes from the ones in unittest. Basically, skipped tests raise a
SkipTest exception which is recorded by the test result class not as an
error but as a skipped test. The relevant code is in test/support.py in
which can be accessed with viewcvs as
http://www.intevation.de/cgi-bin/vi...rev=1.19&content-type=text/vnd.viewcvs-markup

If the echo is positive, I will make an initial implementation and
provide a patch for feedback.

I think this would be useful to have.

Bernhard

 

[Remy Blank]

Having just completed a project which required testing as root, I can
sympathize with you completely! If at all possible, try to avoid
getting painted into that particular corner.

Is it possible that by building the appropriate test scaffolding, you
could run the tests as a normal user? For example, let's say the reason
you need to be root is because you are reading a file which is mode 600.
You could then do something like this:
(snip)

It's not always possible or practical to do that, but it's worth
thinking about.

Yes, well, the whole idea of my project (which I'll call mockfs) is
to provide a framework to do exactly that. However, for it to be
useful, it has to behave as closely as possible as an original
filesystem. So the idea was to write a test suite that can be run
either on the real filesystem (in a chroot environment), or on the
mock filesystem. Both should pass identically. And for running it on
the real filesystem, well, you need to be root for some tests.

-- Remy


Remove underscore and suffix in reply address for a timely response.

 

[Remy Blank]

Check out TestSkipped.
It's already implemented as a part of utility functions for
unittest module and heavily used in Lib/test directory.

That is interesting. From a first look, however, it seems that the
tests don't use the unittest framework, right? I'll study them
anyway, good ideas are always good to have.

Thanks for the pointer.
-- Remy


Remove underscore and suffix in reply address for a timely response.

 

[Jeremy Fincher]

Hello unittest users,

In a project I am working on, I have a series of tests that have
to be run as root, and others as a normal user.

In my project, I have many tests that require a network connection,
and many others which don't.

One solution is to separate the tests into two different files,
and run the right one. I don't like this too much, as I prefer
to group tests by "function".

I agree.

class AnythingTest(unittest.TestCase):
def testSomethingAsRoot(self):
self.skipIf(os.geteuid() != 0, "Must be root")

self.assertEqual(...)
...

def testSomethingAsNormalUser(self):
self.skipIf(os.geteuid() == 0, "Must be normal user")

self.assertEqual(...)
...

With all the argument that's happened on this list regarding
decorators and conditional expressions and whatnot, I had been
wondering whether anyone here came up with good ideas anymore. But
you've certainly given a counterexample for that notion: your idea is
golden.

If the echo is positive, I will make an initial implementation and
provide a patch for feedback.

Even if it doesn't get accepted into Python proper, I'll take a copy
and put it with my project -- I already provide my own unittest.py
that I hacked to show the total number of asserts (I was curious one
day, and it's turned out to be a remarkably useful thing to have,
since my tests-upon-tests can hide unreaonable numbers of assertions
which otherwise would go unnoticed (except as a general slowness of
the test suite).

Jeremy

 

[Remy Blank]

Even if it doesn't get accepted into Python proper, I'll take a copy
and put it with my project -- I already provide my own unittest.py

I was wondering, is it worth modifying unittest.py directly and
provide a patch, or should I just extend its functionality in a
separate file, and provide that one?

that I hacked to show the total number of asserts (I was curious one
day, and it's turned out to be a remarkably useful thing to have,
since my tests-upon-tests can hide unreaonable numbers of assertions
which otherwise would go unnoticed (except as a general slowness of
the test suite).

Interesting idea. What do you use the assertion count for?

-- Remy


Remove underscore and suffix in reply address for a timely response.

 

[Jeremy Fincher]

I was wondering, is it worth modifying unittest.py directly and
provide a patch, or should I just extend its functionality in a
separate file, and provide that one?

I just modified it directly. What format the Python people prefer for
their patches is their business

Interesting idea. What do you use the assertion count for?

Supybot has tests for its plugins, but since some plugins are rather
central to the bot, they're loaded during all the plugin tests. When
I increased the number of these central plugins, I found that the
tests ran more slowly. After adding the assertion count, I discovered
that each central plugin I added resulted in about 1,000 more
assertions. What had happened was that during the tests for each
plugin, the tests for each central plugin were being re-run, resulting
in a significantly more time consuming test run.

Nowadays, I just use it as a general measure of how many tests we
have, and as a bragging point on our website

Jeremy