unittest vs py.test?

[Roy Smith]

I've used the standard unittest (pyunit) module on a few projects in the
past and have always thought it basicly worked fine but was just a little
too complicated for what it did.

I'm starting a new project now and I'm thinking of trying py.test
(http://codespeak.net/py/current/doc/test.html). It looks pretty cool from
the docs. Is there anybody out there who has used both packages and can
give a comparative review?

 

[Raymond Hettinger]

[Roy Smith]

I've used the standard unittest (pyunit) module on a few projects in the
past and have always thought it basicly worked fine but was just a little
too complicated for what it did.

I'm starting a new project now and I'm thinking of trying py.test
(http://codespeak.net/py/current/doc/test.html). It looks pretty cool from
the docs. Is there anybody out there who has used both packages and can
give a comparative review?

I've used both and found py.test to be effortless and much less verbose.
For more complex testing strategies, py.test is also a winner. The generative
tests are easier to write than crafting a similar strategy for unittest.

py.test does not currently integrate well with doctest; however, that will
likely
be the next feature to be added (per holger's talk at PyCon).

For output, unittest's TextTestRunner produces good looking, succinct output
on successful tests. For failed tests, it is not bad either. In contrast,
py.test
output is more highly formatted and voluminous -- it takes a while to get used
to.

unittest users have to adapt to the internal structure of the unittest module
and
become familiar with its class structure (test fixture, test case, test suite,
and test
runner objects). py.test does a good job of hiding its implementation.

py.test is relatively new and is continuing to evolve. Support tools like a
GUI test runner are just emerging. In contrast, unittest is based on a proven
model and the code is mature.

unittest module updates come up in distinct releases, often months or years
apart. py.test is subject to constant update by subversion. Personally, I like
the continuous updates, but it could be unsettling if you're depending on it
for production code.


Raymond Hettinger

 

[Nigel Rowe]

I've used the standard unittest (pyunit) module on a few projects in the
past and have always thought it basicly worked fine but was just a little
too complicated for what it did.

I'm starting a new project now and I'm thinking of trying py.test
(http://codespeak.net/py/current/doc/test.html). It looks pretty cool
from
the docs. Is there anybody out there who has used both packages and can
give a comparative review?

Have you seen Grig Gheorghiu's 3 part comparison of unittest, and py.test?

http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-1-unittest.html
http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-2-doctest.html
http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-3-pytest-tool.html

 

[bearophileHUGS]

Nigel Rowe>Have you seen Grig Gheorghiu's 3 part comparison of
unittest, and py.test?<

Very interesting articles, thank you. Testing seems something still in
quick development.

For small functions the doctests are useful, but py.test has some
advantages. Probably something even better that py.test can be
designed, taking some ideas from the doctests, or vice versa :-]
In py.test I see a couple of features useful for the Python language
too:

The raises:
py.test.raises(NameError, "self.alist.sort(int_compare)")
py.test.raises(ValueError, self.alist.remove, 6)
(A try can probably do something similar)

And the improved error messages:
"When it encounters a failed assertion, py.test prints the lines [3-4
lines?] in the method containing the assertion, up to and including the
failure. It also prints the actual and the expected values involved in
the failed assertion."

http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-3-pytest-tool.html

Such things can help avoid (just for simple situations/functions!)
testing frameworks in the first place, so you can use just the normal
Python code to test other code.

Bye,
Bearophile

 

[Roy Smith]

Have you seen Grig Gheorghiu's 3 part comparison of unittest, and py.test?

http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-1-unittest.html
http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-2-doctest.html
http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-3-pytest-tool.html

Just finished reading them now. Thanks for the pointer, they make an
excellent review of the space.

One thing that worries me a little is that all three seem to have
advantages and disadvantages, yet none is so obviously better than the
others that it stands out as the only reasonable way to do it. This means
some groups will adopt one, some will adopt another, and the world will
become fragmented.

 

[Grig Gheorghiu]

In my mind, practicing TDD is what matters most. Which framework you
choose is a function of your actual needs. The fact that there are 3 of
them doesn't really bother me. I think it's better to have a choice
from a small number of frameworks rather than have no choice or have a
single choice that might not be the best for your specific environment
-- provided of course that this doesn't evolve into a PyWebOff-like
nightmare

Grig

 

[Raymond Hettinger]

One thing that worries me a little is that all three seem to have
advantages and disadvantages, yet none is so obviously better than the
others that it stands out as the only reasonable way to do it. This means
some groups will adopt one, some will adopt another, and the world will
become fragmented.

Worry is a natural thing for someone with "panix" in their email address ;-)

FWIW, the evolution of py.test is to also work seemlessly with existing tests
from the unittest module.

the world diversifies, the world congeals,


Raymond Hettinger

 

[Jeremy Bowers]

FWIW, the evolution of py.test is to also work seemlessly with existing tests
from the unittest module.

Is this true now, or is this planned?

I read(/skimmed) the docs for py.test when you linked to the project, but
I don't recall seeing that. Certainly some of the features made me drool
but I have an investment in unittest. If I can relatively easily port them
over, I'd love to use py.test. (I don't care about a small per-file
change, it'd probably be one I can automate anyhow. But I can't afford to
re-write every test.) I didn't see anything like this in the docs, but I
may have missed it.

That'd be cool.

 

[Grig Gheorghiu]

From what I know, the PyPy guys already have a unittest-to-py.test
translator working, but they didn't check in the code yet. You can send
an email to py-dev at codespeak.net and let them know you're interested
in this functionality.

Grig

 

[Colin J. Williams]

In my mind, practicing TDD is what matters most. Which framework you
choose is a function of your actual needs. The fact that there are 3 of
them doesn't really bother me. I think it's better to have a choice
from a small number of frameworks rather than have no choice or have a
single choice that might not be the best for your specific environment
-- provided of course that this doesn't evolve into a PyWebOff-like
nightmare

Grig

Grig,

Many thanks for your helpful essays.

unittest seems rather heavy. I don't like mixing tests with
documentation, it gives the whole thing a cluttered look.
Py.test is the more appealing but it doesn't appear to be
ready for packaging yet.

Thanks,

Colin W.

 

[Peter Hansen]

unittest seems rather heavy. I don't like mixing tests with
documentation, it gives the whole thing a cluttered look.

unittest can really be rather light. Most of our
test cases are variations on the following, with
primarily application-specific code added rather than
boilerplate or other unittest-related stuff:

import unittest

class TestCase(unittest.TestCase):
def test01(self):
'''some test....'''
self.assertEquals(a, b)

def test02(self):
'''another test'''
self.assertRaises(Error, func, args)


if __name__ == '__main__':
unittest.main()


That's it... add testXX() methods as required and
they will be executed in sorted order (alphabetically)
automatically when you run from the command line.
The above might look excessive in comparison to the
test code, but add some real code and the overhead
quickly dwindles to negligible.

I'm a little puzzled why folks so often consider this
particularly "heavy". No need to deal with suites,
TestResult objects, etc, as others have suggested,
unless you are trying to extend it in some special
way.

-Peter

 

[Roy Smith]

unittest can really be rather light. Most of our
test cases are variations on the following, with
primarily application-specific code added rather than
boilerplate or other unittest-related stuff:

import unittest

class TestCase(unittest.TestCase):
def test01(self):
'''some test....'''
self.assertEquals(a, b)

Well, right there the "extra" stuff you needed to do (vs. py.test) was
import unittest, inherit from it, and do "self.assertEquals" instead of
just plain assert. But (see below), that's not the big thing that attracts
me to py.test.

I'm a little puzzled why folks so often consider this
particularly "heavy". No need to deal with suites,
TestResult objects, etc, as others have suggested,
unless you are trying to extend it in some special
way.

In all but the most trivial project, you're going to have lots of tests.
Typically, each class (or small set of closely related classes) will go in
one source file, with a corresponding test file. You'll probably have
stuff scattered about a number of different directories too. That means
you need to build some infrastructure to find and run all those various
tests.

One way would be futzing with suites (which I still haven't completely
figured out). Another way would be building a hierarchical series of
dependencies in Make (or whatever build tool you use) to run your tests.
The latter is what I usually do. The idea that I can just type "python
py.test" at the top level and have it find and run everything for me just
blows me away convenience-wise.

I also like the idea that I just stick print statements into my tests and
the output automagically goes away unless the test fails. I'm a firm
believer that unit tests should NOT PRODUCE ANY OUTPUT unless they fail.
I'm working with a system now where the unit tests not only produce reams
of output, but it's also rigged to keep going in the face of failure.
Trying to find the actual error in the output is a nightmare.

 

[Raymond Hettinger]

[Peter Hansen]

unittest can really be rather light. Most of our
test cases are variations on the following, with
primarily application-specific code added rather than
boilerplate or other unittest-related stuff:

import unittest

class TestCase(unittest.TestCase):
def test01(self):
'''some test....'''
self.assertEquals(a, b)

def test02(self):
'''another test'''
self.assertRaises(Error, func, args) . . .
I'm a little puzzled why folks so often consider this
particularly "heavy".

unittest never felt heavy to me until I used py.test. Only then do you realize
how much boilerplate is needed with unittest. Also, the whole py.test approach
has a much simpler object model.

BTW, the above code simplifies to:

from py.test import raises
assert a == b
raises(Error, func, args)


Raymond Hettinger

 

[Peter Hansen]

BTW, the above code simplifies to:

from py.test import raises
assert a == b
raises(Error, func, args)

This is pretty, but I *want* my tests to be contained
in separate functions or methods. The trivial amount
of extra overhead that unittest requires fits with
the way I want to write my tests, so it basically
represents zero overhead for me.

The above doesn't look like it would scale very
well to many tests in terms of maintaining some
semblance of structure and readability.

And once you add some functions or whatever to do
that, I'm still unclear on how the one or two lines
of extra code that unittest requires represents
an amount of code that really merits the label "heavy".

As for Roy's comments: I use a small internally
developed driver script which uses os.walk to find
all the files matching tests/*_unit.py or tests/story*.py
in all subfolders of the project, and which runs them
in separate processes to ensure none can pollute
the environment in which other tests run. I can
dispense with the unittest.main() call, but I like
to be able to run the tests standalone. I guess
with py.test I couldn't do that...

If py.test provides a driver utility that does
effectively this, well, that's nice for users. If
it doesn't run them as separate processes, it wouldn't
suit me anyway.

Still, it sounds like it does have a strong following
of smart people: enough to make me want to take a
closer look at it to see what the fuss is about.

-Peter

 

[Roy Smith]

As for Roy's comments: I use a small internally
developed driver script which uses os.walk to find
all the files matching tests/*_unit.py or tests/story*.py
in all subfolders of the project, and which runs them
in separate processes to ensure none can pollute
the environment in which other tests run. I can
dispense with the unittest.main() call, but I like
to be able to run the tests standalone. I guess
with py.test I couldn't do that...

Actually, I believe it does. I'm just starting to play with this, but it
looks like you can do:

py.test test_sample.py

and it'll run a single test file. I imagine you could use your os.walk
fixture in combination with this to run each test in its own process if you
wanted to.

 

[Raymond Hettinger]

[Peter Hansen]

If py.test provides a driver utility that does
effectively this, well, that's nice for users. If
it doesn't run them as separate processes, it wouldn't
suit me anyway.

Still, it sounds like it does have a strong following
of smart people: enough to make me want to take a
closer look at it to see what the fuss is about.

FWIW, py.test scales nicely. Also, it takes less time to try
it out or read the docs than discuss it to death on a newsgroup.
The learning curve is minimal.


Raymond Hettinger

 

[Raymond Hettinger]

[Peter Hansen]

This is pretty, but I *want* my tests to be contained
in separate functions or methods.

In py.test, those would read:

def test1():
assert a == b

def test2():
raises(Error, func, args)

Enclosing classes are optional.


Raymond

 

[Bengt Richter]

[Peter Hansen]

If py.test provides a driver utility that does
effectively this, well, that's nice for users. If
it doesn't run them as separate processes, it wouldn't
suit me anyway.

Still, it sounds like it does have a strong following
of smart people: enough to make me want to take a
closer look at it to see what the fuss is about.

FWIW, py.test scales nicely. Also, it takes less time to try
it out or read the docs than discuss it to death on a newsgroup.
The learning curve is minimal.

Is there a package that is accessible without svn?

Regards,
Bengt Richter

 

[Roy Smith]

Is there a package that is accessible without svn?

That seems to be its weak point right now.

Fortunately, you can get pre-built svn clients for many platforms
(http://subversion.tigris.org/project_packages.html#binary-packages), and
from there you just have to run a single command (svn get URL). Still, the
py.test folks would be doing themselves a favor if they made it available
with more common tools.

 

[Peter Hansen]

Actually, I believe it does. I'm just starting to play with this, but it
looks like you can do:

py.test test_sample.py

and it'll run a single test file.

Well, my driver script can do that too. I just meant
I could do "test_sample.py" and have it run the test
any time, if I wanted, and I was mainly trying to
show that even the __name__ == '__main__' part of
my example was not essential to the use of unittest.
Comparing apples to apples, so to speak, since it
looks like you don't use an import to get access
to py.test.

As near as I can tell, other than requiring an import
statement, and a class statement, there is zero
additional overhead with unittest versus py.test,
given the way I want to structure my tests (in
functions or methods). Is that true? If so, I stand
by my claim that the difference in "weight" between
the two is much much less than some have claimed.

-Peter