unit testing guidelines

P

Phlip

I find debugging a bit too restrictive. I can't just use Undo to make the
bug go >poof<.

Imagine if you had such a button on your debugger! You would hit it all the
time!

You have such a button; it's just a little more expensive than raw code. The
cost savings - no more debugging - overwhelmingly offsets that cost.
More fundamentally, TDD is Design Methodology, Not a Testing Methodology.

It just happens to use Unit tests as its means of describing the design,
much like RUP uses UML.

Indeed, some TDD practitioners are starting to call it BDD - as in
http://www.google.co.uk/search?hl=en&q=behaviour+driven+development&btnG=Google+Search&meta=

And some call it Test First Programming, because TDD is position to replace
the hideous name "eXtreme Programming".

And it doesn't create "unit tests", which are a different topic entirely.

The failure of a unit test implicates only one unit - such as the Ariane V
engine controller.

The failure of a _Developer_ Test implicates the developer's last edit. Time
to hit Undo.

That's not exhaustive.

TDD done well will reduce the _odds_ that you need exhaustive unit testing.
 
I

Ian Collins

Andrew said:
I'd clarify that with 'TDD done *Correctly will give you 100% execution
coverage'

*Correctly = Write 1 failing Testcase,
Write only enough code to make test Pass,
Refactor to Remove Duplication,
Repeat

More commonly referred to as Red, Green, Refactor.



Always starting with the test first, only allows for 100%.
I was using the OP's definition of "test coverage". It might just be
me, but I've always had testers or users (normally testers) find some
bizarre use case that wasn't catered for in the original user stories or
unit tests.
 
P

Phlip

Ian said:
I was using the OP's definition of "test coverage". It might just be me,
but I've always had testers or users (normally testers) find some bizarre
use case that wasn't catered for in the original user stories or unit
tests.

That's why, regardless of your unit testing strategy, you work to lower the
cost of acceptance tests, so anyone can write them, and they come up with
all sorts of things.

Hence all of XP is driven by tests.
 
A

Andrew McDonagh

Ian said:
I was using the OP's definition of "test coverage". It might just be
me, but I've always had testers or users (normally testers) find some
bizarre use case that wasn't catered for in the original user stories or
unit tests.

But we are talking about unit testing here - developers write and run
unit tests.

Users/testers don't unit test - they Acceptance (integration, System) Test.

Everyone runs the acceptance tests.
 
N

n2xssvv g02gfr12930

Jacob said:
I have compiled a set og unit testing
recommendations based on my own experience
on the concept.

Feedback and suggestions for improvements
are appreciated:

http://geosoft.no/development/unittesting.html

Thanks.
At no point do you suggest that both the code design and testing should
be subject to independent review, preferably by another programmer and a
user. This can often help spot potential problems early on. The
developer can frequently miss something, if only because their too close
and as the cliche goes "You can't see the wood for the trees".

Hope you approve, from an unemployed C++ developer

JB
 
A

Adam Maass

Jacob said:
I understand your objection, but this is actually one of the
mechanisms that have helped me found some of the hardest to
trace and most subtle errors in the code. It has proven to be
extremely helpful. Also, it gives me lots of confidence
knowing that my test suite of several thousand tests
are executed every hour with different input each time.
It is like adding another dimension to unit testing.

There is sometimes value in testing on large numbers of random inputs. But
this isn't *unit* testing; it's more akin to a system or stress test. It's
something you hope your QAs will do for you; test on inputs that you weren't
necessarily expecting and see what breaks. Unit testing is about the
correctness of code for known inputs. If you come across a failure for a
novel set of inputs in system or stress testing, by all means, take that
input and add it to your unit test suite.

Note that test frameworks can be used both for unit tests as well as other
kinds of tests. (Simply because it's called 'JUnit', for example, does not
necessarily mean that all the test cases are, in fact, unit tests.)

-- Adam Maass
 
J

Jacob

Adam said:
There is sometimes value in testing on large numbers of random inputs. But
this isn't *unit* testing; it's more akin to a system or stress test. It's
something you hope your QAs will do for you; test on inputs that you weren't
necessarily expecting and see what breaks. Unit testing is about the
correctness of code for known inputs.

Which definition of unit testing is this? I have searched the
net but hasn't been able to find any backing for this?

If I write a method void setLength(double length), who define
the input "necesserily expected", and why isn't this the entire
double range? I'd claim the latter and to cover as many inputs
as possible I use the random trick.

I don't have a problem with defining this kind of testing
differently, for instance "stress testing", but on the other
hand there isn't really any more "stress" is calling
setLength(1.23076e+307) than setLength(2.0) as long as the
method accepts a double as input?

And why do you care about "known" input as long as the
actual (failing) input can be traced afterwards anyway?

You define this as a unit test:

for (int i = 0; i < 1000; i++)
testMyIntMethod(i);

while this is not:

for (int i = 0; i < 1000; i++)
testMyIntMethod(getRandomInt())

even if an error on input=42 will produce identical error reports
in both cases. Only the latter will (eventually) reveal the
error for input=-100042.

Also, if I have a setLength() method which cover the "typical"
input cases just fine, but is in general crap (a common scenario),
then a testSetLength() method that verifies that setLength() work
fine for "typical" input isn't worth a lot. What you need is a test
method that test the non-typical inputs. From a black-box perspective
you don't really know what is typical or non-typical, so why not just
throw a random number genrator at it?
 
B

Ben Pope

Jacob said:
Which definition of unit testing is this? I have searched the
net but hasn't been able to find any backing for this?

If I write a method void setLength(double length), who define
the input "necesserily expected", and why isn't this the entire
double range? I'd claim the latter and to cover as many inputs
as possible I use the random trick.

The programmer specifies the preconditions. If the preconditions are
not met, there is no reason for it to produce valid results. Random is
not repeatable, and is not predictable.
I don't have a problem with defining this kind of testing
differently, for instance "stress testing", but on the other
hand there isn't really any more "stress" is calling
setLength(1.23076e+307) than setLength(2.0) as long as the
method accepts a double as input?

No, but the point is that when you unit test, you need to make informed
choices about the inputs you choose. It's usually wise to throw in a
couple of "normal", "everyday" values, but also explicitly check
boundary cases and out of range.
And why do you care about "known" input as long as the
actual (failing) input can be traced afterwards anyway?

Repeatability. It's no use relying on randomness to thoroughly test.
You have to design your test cases.
You define this as a unit test:

for (int i = 0; i < 1000; i++)
testMyIntMethod(i);

Not really. What are you testing? That it doesn't crash? Presumably
you need to check the output against an array of 1000 pre-computed
values? Not much fun.
while this is not:

for (int i = 0; i < 1000; i++)
testMyIntMethod(getRandomInt())

How can you possibly check the output is correct for a random input?
even if an error on input=42 will produce identical error reports
in both cases. Only the latter will (eventually) reveal the
error for input=-100042.

I don't understand, are you checking for crashing?
Also, if I have a setLength() method which cover the "typical"
input cases just fine, but is in general crap (a common scenario),
then a testSetLength() method that verifies that setLength() work
fine for "typical" input isn't worth a lot. What you need is a test
method that test the non-typical inputs. From a black-box perspective
you don't really know what is typical or non-typical, so why not just
throw a random number genrator at it?

You know your preconditions. You know your postconditions.

If anything can happen on invalid input, then no point in testing. If
you want a default output, exception or whatever for out-of-range, then
check it with a unit test. You get to choose your input and you get to
check your output.

Randomness just doesn't cut it, and I don't understand how you can check
the output is correct, without knowing the input.

Ben Pope
 
J

Jacob

Ben said:
Randomness just doesn't cut it, and I don't understand how you can check
the output is correct, without knowing the input.

You *do* know the input!

Consider testing this method:

double square(double v)
{
return v * v;
}

Below is a typical unit test that verifies that the
method behaves correctly on typical input:

double v = 2.0;
double v2 = square(v); // You know the input: It is 2.0!
assertEquals(v2, 4.0);

The same test using random input:

double v = getRandomDouble();
double v2 = square(v); // You know the input: It is v!
assertEquals(v2, v*v);

If the test fails, all the details will be in the error
report.

And this method actually *do* fail for a mjority of all
possible inputs (abs of v exceeding sqrt(maxDouble)).
This will be revealed instantly using the random approach.

For an experienced programmer the limitation of square()
might be obvious so border cases are probably covered
sufficiently in both the code and the test. But for more
complex logic this might not be this apparent and throwing
in random input (in ADDITION to the typical cases and all
obvious border cases) has proven quite helpful, at least
to me.
 
T

Tom Leylan

Jacob:

You've chosen a trivial example where your assert can compute the results of
the Square() function you are calling. That is hardly a typical situation
or there would be no reason for the function to have been created.

double v = getRandomDouble();
double v2 = AccountBalance( v );
assertEquals( v2, ? );

So explain how you get the value to type in ? given you don't know what the
input will be. Perhaps you would do the computations you read about in the
AccountBalance() method inline to see if those and yours matched?
 
J

Jacob

Tom said:
You've chosen a trivial example where your assert can compute the results of
the Square() function you are calling. That is hardly a typical situation
or there would be no reason for the function to have been created.

double v = getRandomDouble();
double v2 = AccountBalance( v );
assertEquals( v2, ? );

So explain how you get the value to type in ? given you don't know what the
input will be. Perhaps you would do the computations you read about in the
AccountBalance() method inline to see if those and yours matched?

I chose a fairly typical example of a basic unit requiring
unit testing and I proved that by using random input it
easily identified an error that otherwise could slip through.

I never said that using random input was useful in all
cases and perhaps it isn't in your specific example. On the
other hand, how do you know what goes into "?" given you
know the input? There must be some sort of reasoning behind
your result as well.

Below is a different example which might not be as trivial
as my previous. It "proves" that encoding + decoding (according
to some procedure) of any string should give back the original
string:

String text = getRandomString(0,1000000); // 0 - 1MB
String encoded = Encoder.encode(text);
String decoded = Encoder.decode(encoded);
assertEquals(text, decoded);
 
A

Andrew McDonagh

Below is a different example which might not be as trivial
as my previous. It "proves" that encoding + decoding (according
to some procedure) of any string should give back the original
string:

String text = getRandomString(0,1000000); // 0 - 1MB
String encoded = Encoder.encode(text);
String decoded = Encoder.decode(encoded);
assertEquals(text, decoded);

This only proves that the encoding & decoding scheme is the same. I can
make this test pass like this...

public String encode(String text) {
return text;
}

public String decode(String encoded) {
return encoded;
}

Here, the unit test is testing every line of code, yet its a worthless
implementation.

The unit test is not testing the encoding mechanism as it should be
implemented as its using the decode() for the assertion.

In this case, its prudent/better to have separate unit tests for both
the encode & decode, an doing the opposite conversion locally within the
unit test itself as this prevents errors of implementation but more
importantly, forces the implementation to use the correct algorithms.

public void testEncoder() {
String text = getRandomString(0,1000000); // 0 - 1MB
String encoded = Encoder.encode(text);
String decoded = MD5.decode(encoded);
assertEquals(text, decoded);
}

public void testDecoder() {
String text = getRandomString(0,1000000); // 0 - 1MB
String encoded = MD5.encode(text);
String decoded = Encoder.decode(encoded);
assertEquals(text, decoded);
}

Here we have separated the logic needed for the test from the unit under
test.
 
T

Tom Leylan

Forgive me but you are terming it "fairly typical" and it isn't typical of
anything I have seen. So rather than you or I decide (since we don't agree)
let others weigh in on how typical a function is that an easily-contrived
formula produces the exact same answer. Show me your assertEquals for
IsPrime() for instance.

The discussion hasn't been (as I read it) that random input is of no value
in all cases. It was illustrated by others that "unit tests" imply one does
know the answer and that random input means you can rarely know the answer.
To answer your question as to how one might know the value of ? it would be
"computed" by whatever method was required. A test for IsPrime() would be
fed known prime and non-prime values safely knowing which ones should return
true and which should return false. A test for AccountBalance() would
similarly have inputs and outputs which have been determined to test the
functionality.

I think Andrews response does a great job of pointing out how relying on the
two functions to test each other is a mistake.
 
A

Alex Hunsley

Andrew said:
This only proves that the encoding & decoding scheme is the same. I can
make this test pass like this...

public String encode(String text) {
return text;
}

public String decode(String encoded) {
return encoded;
}

Here, the unit test is testing every line of code, yet its a worthless
implementation.

No, an encoding that doesn't do any encoding is still an encoding. It
just happens to be the 'identity' encoding (in the same way that 1 is
the identity for multiplication and 0 is the identity for addition). If
an encoding and decoding function are presented as a pair of opposite
functions, then you are perfectly justified in testing that one is the
inverse of the other:

Decode(Encode(x)) = x

Testing that an encoding is *any good* by whatever means you judge
'good' is entirely a different matter to testing the reversibility of an
encode/decode pair (and harder to do, to boot, although testing that the
encoding did something, anything, to the data is trivial).
 
A

Andrew McDonagh

Jacob said:
The test is still useful. It can't prove that the code is correct,
but if it fails, it can prove that the code is wrong.

But it cant tell you which of the collaborating en/de coding methods is
the cause.
 
J

Jacob

Andrew said:
Here, the unit test is testing every line of code, yet its a worthless
implementation.

The test is still useful. It can't prove that the code is correct,
but if it fails, it can prove that the code is wrong.

And as stated several times already: The test comes in ADDITION to
the test for typical cases and all obvious boundary cases, which in
this particular case would have been written quite differently.
 
J

Jacob

Tom said:
Forgive me but you are terming it "fairly typical" and it isn't typical of
anything I have seen.

The most typical methods around are getters and setters which
are even less complex than the square example I used previously:

String name = getRandomString(0,1000);
A.setName(name);
assertEquals(A.getName(), name);

They are not the most interesting ones to test, but they should
still be tested, and using random input increase the test coverage.
Show me your assertEquals for IsPrime() for instance.

Not the best example I could come up with, but it indicates
the principle:

for (int i = 0; i < 1000; i++) {
int v1 = getRandomInt();
if (isPrime(v1)) {
for (int j = 0; j < 1000; j++) {
int v2 = getRandomInt();
if (isPrime(v2)) {
assertNotEquals(v2 % v1, 0);
assertNotEquals(v1 % v2, 0);
}
}
}
}

Again: It doesn't prove that isPrime() is correct, but it may be able
to prove that it is wrong.
 
J

Jacob

Andrew said:
But it cant tell you which of the collaborating en/de coding methods is
the cause.

Given an explicit input for which the operation fails should
give you enough information to be able to track this down yourself.
 
A

Alex Hunsley

Andrew said:
But it cant tell you which of the collaborating en/de coding methods is
the cause.

But it is telling you that you have a problem, which is a good start.
You can then invetigate further, or back up with other complementary tests.
 
T

Tom Leylan

Of course one can insert getRandomString() into a test when the actual
string value has no known limits. Put it into your US State or US Zip
Get/Set... Any field which validates it's entry should fail upon assignment
and your assertion doesn't get a chance to run does it? Again nobody
claimed that there is anything wrong with random string testing. It was
pointed out that it shouldn't form the basis of one's unit tests.

It seems to me the purpose of unit testing is to verify that values known to
be good, pass and that values known to be bad, fail. If a bad value makes
it through it can be added to the test suite. That is different than
"generating an additional random value."

So you should continue to include random string tests (to your unit tests)
and I (and a few others) will probably recommend against it. There is no
problem with differing viewpoints.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads


Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,905
Latest member
Kristy_Poole

Latest Threads

Top