A simple unit test framework

[Ian Collins]

Yes - I've done it. Short release cycles - I invented them. Management
still fsck's up all the time, every time.

You should either educate them, or be them!

 

[Ian Collins]

first and second.

I don't see a practical distinction between tester, developer and
designer. While the overall design of the system needs an "architect",
the job of the architect is to provide a framework that inevitably needs
extensibility.

The "architect" is the development team.

Automated acceptance tests (some call them *Customer* acceptance) tests
are a critical part of any product development process and someone has
to design and develop them. That someone can either be the customer, or
testers working as their proxy.

 

[Gianni Mariani]

You should either educate them, or be them!

Be them is the next challenge. Drowning in BS... I tell ya.

 

[Gianni Mariani]

Ian Collins wrote:
....

The "architect" is the development team.

Automated acceptance tests (some call them *Customer* acceptance) tests
are a critical part of any product development process and someone has
to design and develop them. That someone can either be the customer, or
testers working as their proxy.

The customer is always ill defined.

The products I have worked on most recently are consumer applications
(p2p). Product success is adoption by millions of customers. There is
no hard and fast "acceptance" test. The marketing team had no idea, I
ended up resigning but that's another story.

G

 

[Ian Collins]

Ian Collins wrote:
....

The customer is always ill defined.

That's why it is essential to have proxy. For a consumer product, that
should be the internal product owner.

 

[Gianni Mariani]

That's why it is essential to have proxy. For a consumer product, that
should be the internal product owner.

Finding a good dev is costly. Finding a good marketing/customer proxy
.... priceless.

The only place I have ever found marketeers that knew what they were
doing was at WebTV (before it became MS).

Most of the successful companies I have worked with (other than WebTV)
were a success despite themselves.

There was one failed company that failed because the market disappeared
(web radio). Yes, it still exists but there is no money (not much
money) in it.

Other failed companies were mainly because the product that was
requested was never the product that the company needed and management
was never capable of comprehending what they could really do with the
product they had.

Developing software is relatively easy. A very good project manage once
said, software is just a set of decisions (I know I messed that one up).
The inference being the more quickly you make the right decisions, the
quicker the product is developed. Hence, if your management team is
incapable of making decisions, they're not doing their job. Similarly,
if your dev team is incapable of making decisions, it's not doing their
job. I've never been accused of not making decisions.

 

[Pete Becker]

There are plenty of situations where a Monte Carlo test isn't
appropriate or even possible. A good test developer has the knack of
thinking like a user, where a developer thinks like a developer.

That's only a part of testing. A good tester thinks about how to make
the code fail, and aggressively tries to break it.

--

-- Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com)
Author of "The Standard C++ Library Extensions: a Tutorial and
Reference." (www.petebecker.com/tr1book)

 

[James Kanze]

James Kanze wrote:

Common misconception.

1. Testability of code is a primary objective. (i.e. code that can't be
tested is unfit for purpose)

2. Any testing (MT or not) is about a level of confidence, not absoluteness.

I have discovered that MT test cases that push the limits of the code
using random input does provide sufficient coverage to produce a level
of confidence that makes the target "testable".

Then write it up, and publish it, because you have obviously
struck on something new, which no one else has been able to
measure. But you must mean something else by Monte Carlo
testing than has been meant in the past. Because the
probability of finding an error by just throwing random data at
a problem is pretty low for any code which has passed code
review.

If you consider what happens when you have multiple processors
interacting randomly in a consistent system, you end up testing more
possibilities than can present themselves in a more systematic system.
However, with threading, it's not really systematic because external
events cause what would normally be systematic to be random. Now
consider what happens in a race condition failure. This normally
happens when two threads enter sections of code that should be mutually
exclusive. Usually there are a few thousand instructions in your test
loop (for a significant test). The regions that can fail are usually
10's of instructions, sometimes 100's.

The regions that can fail are often of the order of 2 or 3
machine instructions. In a block of several million. And in
some cases, the actual situation an only occur less often than
that: there is a threading error in the current implementation
of std::basic_string, in g++, but I've yet to see a test program
which will trigger it.

If you are able to push randomness, how many times do you need
to reschedule one thread to hit a potential problem.

Practically an infinity.

Given cache latencies, pre-emption from other threads, program
randomness (like memory allocation variances) you can achieve
pretty close to full coverage of every possible race condition
in about 10 seconds of testing. There are some systematic
start-up effects that may not be found, but you mitigate that
by running automated testing. (In my shop, we run unit tests
on the build machine around the clock - all the time.)

So perhaps, after a couple of centuries, you can say that your
code is reliable.

So that leaves us with the level of confidence point. You can't achieve
perfect testing all the time, but you can achieve high level of
confidence testing all of the time.

Certainly. Testing can only prove the existence of errors.
Never the absense. Well run shops don't count on testing,
because of this. (That doesn't mean that they don't test. Just
that they don't count on testing, alone, to ensure quality.)

It does require a true multi processor system to test adequately. I
have found a number of problems that almost always fail on a true MP
system that hardly ever fail on a SP system. Very rarely have I found
problems on 4 processor or more systems that were not also found on a 2
processor system, although, I would probably spend the money on a 4 core
CPU for developer systems today just to add more levels of confidence.

In practice, I have never seen a failure in the wild that could not be
discovered with a properly crafted MC+MT test.

And I have.

[...]

I have met very few customers that know what a spec is even if it
smacked them up the side of the head. Sad. Inevitably it leads to
pissed off customer.

Regrettably, it is often up to the vendor to define what the
customer actually needs. But that doesn't mean not writing
specifications.

 

[James Kanze]

Gianni Mariani wrote:

Welcome to the club!

Any agile process (XP, Scrum or whatever) is ideal for this situation.

If your goal is to rip off the customer, yes.

This situation is one of the main reasons these processes have evolved.
The sort release cycles keep the customer in the loop and give them the
flexibility to change their mind without incurring significant rework
costs.

Prototyping and interacting with the customer are nothing new.
They've been part of every software development process for the
last 30 or 40 years.

 

[James Kanze]

Short release cycles - I invented them.

You must be significantly older than me, then, because they were
the rule when I was learning software development, in the
1970's. (In fact, they were the rule where my father worked,
in the 1960's.)

 

[James Kanze]

James Kanze wrote:

We certainly did - field defect reports and the internal cost of
correcting them.

So what were the before and after measures?

You should at least publish this, since to date, all published
hard figures (as opposed to annecdotal evidence) goes in the
opposite sence. (For that matter, the descriptions of eXtreme
Programming that I've seen didn't provide a means of actually
measuring anything.)

Have you tried it? Not having to hold code reviews was one of the
biggest real savings for us.

Yes, we tried it. It turns out the effective code (and design)
review is the single most important element of producing quality
code. (I have done projects where we eliminated testing
entirely. Without any really significant loss of quality. But
the process we had in place was already very well developped; I
wouldn't recommend such as a general rule.)

That's your opinion and you are entitled to it.

Actually, it's not an opinion. It's a result of concrete
measurement.

 

[James Kanze]

James Kanze wrote:

Look up TDD.

I'm familiar with the theory. Regretfully, it doesn't work out
in practice.

Actually, it does meet the requirements by definition since the test
case demonstrates how the requirements should be met.

Bullshit. I've seen just too many cases of code which is wrong,
but for which no test suite is capable of reliably triggering
the errors.

See my "log"ging example.

You mean where you confused the mathematical function "log()"
with a function used to log error messages?

 

[James Kanze]

We will have to agree to disagree on this.

There's nothing to disagree with. It's a basic definition. If
a MC test finds the error, and a hand crafted test doesn't, the
hand crafted test isn't well designed or carefully done.

I have one anecdotal evidence which suggests that no-one is capable of
truly foreseeing the full gamut of issues that can be found in a well
designed MC test.

I have more than anecdotal evidence that there are significant
errors which will slip through MC testing. Admittedly, the
most significant ones also slip through the most carefully
crafted tests as well. It is, in fact, impossible to write a
test for them which will reliably fail.

This is why no shop serious about quality will rely solely on
testing.

A pass on an MC test raises the level of confidence which is always a
good thing.

Certainly. It's just the often, generating enough random data
is more effort than doing things correctly, and beyond a very
low level, doesn't raise the confidence level very much. If we
take Pete's example of a log() function, testing with a million
random values doesn't really give me much more confidence than
testing with a hundred, and both give significantly less
confidence than a good code review, accompanied by testing with
a few critical values. (Which values are critical, of course,
being determined by the code review.)

If I read between the lines here, I think you're saying that we need
test developers to conceive every kind of possible failure. I have yet
to meet anyone who could do that consistently and I have been developing
software for a very long time.

The probability of a single programmer missing something is
close to 1, I agree. The probability of several programmers
missing the same thing, on the other hand, is close to 0. And
the probability of a random test hitting the single input value
for which the code doesn't work is 1/N, where N is the number of
input values. If N is small, exhaustive testing is obviously a
perfect solution. In most cases, however, N is large enough
that a significant sampling by random selection is simply not
possible in a reasonable time.

I don't think your premise (if I read it correctly) is achievable.

It seems to work in practice. At least one company is (or was)
at SEI level 5, and even companies at level 3 regularly turn out
software with less than one error per 100 KLoc, going into
integration testing.

I lean toward getting making the computer do as much work as possible
because it is much more consistent than a developer.

The problem is that the computer only does what you tell it to
do. If you don't tell it to test such and such a feature, it
won't. If you don't tell it what the critical values are
(limits, etc.), then it is unlikely that it will hit them by
chance.

 

[Phlip]

Unless the test are written first!

When their focus is on getting the code to pass tests, of course the tests
will be stronger, and the code will be "weaker", in the math sense.

The fun hits the fan when you make a pro tester think they must defend their
job from TDD. If I were a pro tester, doing the sick soak tests and metrics
stuff, I would naturally prefer my developers use TDD, so I don't waste time
installing the test rig and making all the code accessible.

 

[Phlip]

If your goal is to rip off the customer, yes.

I fail to see how a process that gives the customer what they ask for in 1
week iterations could get away with that. Explain?

Prototyping and interacting with the customer are nothing new.

Nobody said they were. However, most students learning software engineering
never hear of them in any textbook, and many bosses like to insulate their
developers from their clients, to "control the message". So the more ways we
can find to advocate direct collaboration, the better.

I'm familiar with the theory. Regretfully, it doesn't work out
in practice.

And if I asked why you think that, I suspect I'd be treated to a game of
"move the goal post"...

 

[Phlip]

Nope. Test driven design cannot account for the possibility that a
function will use an internal buffer that holds N bytes, and has to
handle the edges of that buffer correctly. The specification says
nothing about N, just what output has to result from what input.

You slipped from "write tests before code" to "write tests based only on the
specifications before writing code".

Even under high amnesia situations, that N-byte buffer must be one function
call away from its test driver. This improves the odds you notice it and
question if production code can overflow it. And if you expressly treat the
test cases as documenting the code, then you should write tests which
document that buffer.

All methodologies still require review...

 

[Phlip]

I never underestimate the ingenuity of good testers.

Ian, did someone play "fallacy of excluded middle" on you?

;-)

 

[Phlip]

So Pete will pass your first test with "return 1;".

How many more tests do you expect to write, before you are sure that
Pete's code is always no more than one unit off in the last decimal?

That technique is "fake it til you make it". It's not the core principle of
TDD, but sometimes it's surprisingly powerful. You pass a test with obtusely
simple code, then write the next test to force that code to upgrade in the
correct direction. And you frequently refactor to simplify the code, even if
it's fake.

The surprising part comes when you stop faking. You pass your new test cases
either by trivially extending the existing design, or even by doing nothing.
This system can rapidly produce the Open Closed Principle for a given set of
inputs.

If, for a given situation, it's _not_ surprisingly powerful, you _will_
learn that very early!

 

[Phlip]

When I was a QA manager I'd bristle whenever any developer said they'd
"let testers" do something. That's simply wrong. Testing is not an adjunct
to development. Testing is a profession, with technical challenges that
differ from, and often exceed in complexity, those posed by development.
The skills required to do it well are vastly different, but no less
sophisticated, than those needed to write the product itself. Most
developers think they can write good tests, but in reality, they simply
don't have the right skills, and test suites written by developers are
usually naive. A development manager who gives testers "complete freedom"
is missing the point: that's not something the manager can give or take,
it's an essential part of effective testing.

Nobody said to fire all the testers. You made that up, then replied to your
presumption.

 

[Phlip]

The most successful teams I worked with were teams that wrote their own
tests.

That's why the testers are not far away, in time, space, or timezone. They
should be in the same lab.