best python unit testing framwork

[Brendan Miller]

What would heavy python unit testers say is the best framework?

I've seen a few mentions that maybe the built in unittest framework
isn't that great. I've heard a couple of good things about py.test and
nose. Are there other options? Is there any kind of concensus about
the best, or at least how they stack up to each other?

Brendan

 

[J Kenneth King]

What would heavy python unit testers say is the best framework?

I've seen a few mentions that maybe the built in unittest framework
isn't that great. I've heard a couple of good things about py.test and
nose. Are there other options? Is there any kind of concensus about
the best, or at least how they stack up to each other?

Brendan

I find nose to be the best. It's simple, easy, and doesn't sacrifice
power. All good things if you value your time.

 

[Roy Smith]

What would heavy python unit testers say is the best framework?

I've seen a few mentions that maybe the built in unittest framework
isn't that great. I've heard a couple of good things about py.test and
nose. Are there other options? Is there any kind of concensus about
the best, or at least how they stack up to each other?

Brendan

I've been using unittest for years. I'm happy with it. I've looked at a
couple of the other ones (I don't remember exactly which ones) and came to
the conclusion that they didn't seem to provide enough advantage over
unittest to make it worth switching.

Just my opinion.

 

[Timothy Grant]

What would heavy python unit testers say is the best framework?

I've seen a few mentions that maybe the built in unittest framework
isn't that great. I've heard a couple of good things about py.test and
nose. Are there other options? Is there any kind of concensus about
the best, or at least how they stack up to each other?

Brendan

I wanted to like unittest but couldn't. So I started using py.test. I
even wrote a plugin for TextMate to interface with py.test. If I had
known about Nose I would likely have used it instead because it is
built on top of the standard module. Now I have many thousands of
lines of py.test code so am not likely to make the change.

 

[Roy Smith]

I wanted to like unittest but couldn't. So I started using py.test. I
even wrote a plugin for TextMate to interface with py.test. If I had
known about Nose I would likely have used it instead because it is
built on top of the standard module. Now I have many thousands of
lines of py.test code so am not likely to make the change.

A lot of people like or dislike various unit test frameworks for all sorts
of reasons. But, the bottom line is still that the best unit test
framework is the one which gets you to use it every time you write a line
of code.

 

[Timothy Grant]

A lot of people like or dislike various unit test frameworks for all sorts
of reasons. But, the bottom line is still that the best unit test
framework is the one which gets you to use it every time you write a line
of code.

When I first learned about Unit Testing seven years ago, I thought to
myself "what a colossal waste of time." Now if I don't have tests I
really start to worry. I am a write the tests first kind-of-guy, and
my code and my ability to maintain my code have both improved
dramatically.

 

[James Harris]

What would heavy python unit testers say is the best framework?

I've seen a few mentions that maybe the built in unittest framework
isn't that great. I've heard a couple of good things about py.test and
nose. Are there other options? Is there any kind of concensus about
the best, or at least how they stack up to each other?

You don't mention what you want from testing so it's hard to say which
is "best" as it depends on one's point of view.

For example, I had a requirement to test more than just Python
programs. I wanted to test code written in any language. None of the
frameworks I found supported this so I wrote my own tester. It
interacts with programs via file streams - principally stdin, stdout
and stderr though others can be added as needed.

One nice by-product is that test code does not bloat-out the original
source which remains unchanged. And test cases can be written before,
alongside or after the code as desired.

So, YMMV.

 

[Brendan Miller]

You don't mention what you want from testing so it's hard to say which
is "best" as it depends on one's point of view.

For example, I had a requirement to test more than just Python
programs. I wanted to test code written in any language. None of the
frameworks I found supported this so I wrote my own tester. It
interacts with programs via file streams - principally stdin, stdout
and stderr though others can be added as needed.

I was speaking to unit tests (tests of individual classes and
functions in isolation of the rest of the program) and a test driven
development approach. I find those to be much more useful, and good
for pinpointing bugs and guiding development. In contrast regression
tests tend to be slow, and are the programmatic equivalent of kicking
the tires and seeing if they fall off.

I've also seen regression tests lead to a "sweep the bugs under the
rug mentality" where developers will code to prevent errors from
crashing the regression test, e.g. by catching and swallowing all
exceptions without fixing the underlying problem. It's easy to fool
regression tests since what it does works at such a high level that
most aspects of program correctness can't be directly checked. This is
very frustrating to me because it actually leads to lower code
quality.

One nice by-product is that test code does not bloat-out the original
source which remains unchanged.

That's the main reason most people don't write unit tests. It forces
them to properly decouple their code so that parts can be used
independently of one another. Adding such things to messy ball of mud
code after the fact is an enourmous pain in the butt. Thankfully,
since python is duck typed and doesn't require lots of boilerplate
writing interfaces and abstract factories (since the class object
itself acts as an abstract factory), writing properly decoupled code
in the first place doesn't look nearly as hard as in C++ or Java.

 

[James Harris]

I was speaking to unit tests (tests of individual classes and
functions in isolation of the rest of the program) and a test driven
development approach. I find those to be much more useful, and good
for pinpointing bugs and guiding development. In contrast regression
tests tend to be slow, and are the programmatic equivalent of kicking
the tires and seeing if they fall off.

I agree with your points but would add

1. We should write smaller modules than we generally do. It promotes
clear thinking and encourages the production of reusable code. In
Python it is easy to add a __main__ section to interface to the
classes or functions defined in the code. Such code can then be tested
as mentioned. The functions or classes can still be imported into and
used in other code as needed.

2. The example given wasn't intended to be the only way forward but to
highlight that what makes a test package good depends on your
requirements (which I don't think you mentioned). In my case the
desire to test arbitrary languages was a killer blow to various Python-
only frameworks which I wasn't too impressed with anyway. (At least I
saw none that looked worth investing the time to learn.) Similarly an
option for you is to write your own tester. It may be a better fit
than someone else's package and not too hard to do. On the other hand
a pre-written package may be better - depending on what you are
looking for.

I've also seen regression tests lead to a "sweep the bugs under the
rug mentality" where developers will code to prevent errors from
crashing the regression test, e.g. by catching and swallowing all
exceptions without fixing the underlying problem. It's easy to fool
regression tests since what it does works at such a high level that
most aspects of program correctness can't be directly checked. This is
very frustrating to me because it actually leads to lower code
quality.

If you have dodgy developers then much is wrong, not just testing. If
the team works well, IMHO, production of test cases should be a
welcome part of the development process. When is it better to find
bugs: before or after shipping to a client? Test cases can be put
together in various ways. For example, if one has the code and chooses
to work this way white-box testing can lead to generation of test
cases. On the other hand if an organisation prefers it can produce
test cases from the spec - or they can do both. It's their choice but
in any case the developers and testers should work together.

That's the main reason most people don't write unit tests. It forces
them to properly decouple their code so that parts can be used
independently of one another. Adding such things to messy ball of mud
code after the fact is an enourmous pain in the butt. Thankfully,
since python is duck typed and doesn't require lots of boilerplate
writing interfaces and abstract factories (since the class object
itself acts as an abstract factory), writing properly decoupled code
in the first place doesn't look nearly as hard as in C++ or Java.

You could still import one Python module into another to run test
cases such as

-----
import test_support
import <program_to_be_tested>

<tests of program_to_be_tested using test_support>
-----

where the tests can interact directly with the components of the code
under test. The test support code can handle things such as collating
results and reporting as well as providing code to directly support
certain tests, handle exceptions etc.

I wouldn't be surprised if good testing code were in many cases to be
significantly longer than the code being tested. Keeping the tests in
a separate file keeps the code being tested shorter which can only
improve clarity. It also allows either the developer or someone else
to write the tests. Putting code and testing code in the same file
almost requires one person to write both parts.

A final thought: if you have dodgy developers who engineer their code
to avoid tests you're not going to have much success in getting them
to write good test code (though, hopefully, the mental process of
writing test cases should help them improve the code they write).

At the end of the day all I'm saying is that the best test framework
for you really depends on your requirements, rather than what others
think is "best" or not.

 

[Kay Schluehr]

What would heavy python unit testers say is the best framework?

I've seen a few mentions that maybe the built in unittest framework
isn't that great.

The UT frameworks follow the same principles and are all alike more or
less. Of course doctest makes a difference and implies another
approach to testing but whether you use unittest, nosetest or py.test
is mostly a matter of taste. Of course taste matters a lot for various
people and when they have to use CamelCased method names like setUp or
assertEqual in Python they feel alienated and reminded that Java is
out there.

Personally I use unittest.py for a pretty obvious reason: other people
can simply run the test scripts without prior installation of a
testframework.

Kay