Requesting critique of a C unit test environment

[Ark Khasin]

[First, I apologize for cross-posting. I just think a wider audience can
critique from different vantage points.]

Unit testing is an integral component of both "formal" and "agile"
models of development. Alas, it involves a significant amount of tedious
labor.

There are test automation tools out there but from what limited exposure
I've had, they are pricey, reasonably buggy, and require compiler/target
adaptation.

Out of my frustration with two out of two of them came my own. Its
instrumentation approach is based solely on profound abuse of the C
preprocessor (and in this respect it is equally applicable to C++).

I would like to ask to evaluate the approach
- whether it has gaping holes in ideology or implementation
- whether in your opinion it has merits

A preliminary draft description is at
http://www.macroexpressions.com/dl/C code unit testing on a shoestring.pdf

A reference implementation (with a C99 accent) with a runnable example is at
http://www.macroexpressions.com/dl/maestra.zip

Please reply to a newsgroup or via email as you find convenient.
Thank you for your anticipated feedback,

-- Ark

 

[Ian Collins]

[First, I apologize for cross-posting. I just think a wider audience can
critique from different vantage points.]

Unit testing is an integral component of both "formal" and "agile"
models of development. Alas, it involves a significant amount of tedious
labor.

There are test automation tools out there but from what limited exposure
I've had, they are pricey, reasonably buggy, and require compiler/target
adaptation.

Out of my frustration with two out of two of them came my own. Its
instrumentation approach is based solely on profound abuse of the C
preprocessor (and in this respect it is equally applicable to C++).

I would like to ask to evaluate the approach
- whether it has gaping holes in ideology or implementation
- whether in your opinion it has merits

Why not just use one of the free frameworks such as CppUnit?

It works well with both C (with a little fiddling like you do in your
paper for "static") and C++. I'm sure the same applies for other
frameworks.

 

[Ark Khasin]

Ian Collins wrote:

Why not just use one of the free frameworks such as CppUnit?

It works well with both C (with a little fiddling like you do in your
paper for "static") and C++. I'm sure the same applies for other
frameworks.

Ian,
Thank you for your response.

Please correct me if I am wrong, but AFAIK CppUnit doesn't provide a
code execution trace, so it's pretty darn hard to prove code coverage.
[There must be reasons why testing tools vendors command big money.]

Also, if I use C in non-C++ compatible way (e.g. tentative definitions),
my source won't even compile for CppUnit.

And finally there is a port issue (it's an embedded type talking ). I
am proposing something that requires only the compiler.

Regards,
Ark

 

[Ian Collins]

Ian Collins wrote:

Why not just use one of the free frameworks such as CppUnit?

It works well with both C (with a little fiddling like you do in your
paper for "static") and C++. I'm sure the same applies for other
frameworks.

Ian,
Thank you for your response.

Please correct me if I am wrong, but AFAIK CppUnit doesn't provide a
code execution trace, so it's pretty darn hard to prove code coverage.
[There must be reasons why testing tools vendors command big money.]

If you develop your software test first, you get all the code coverage
you need.

Also, if I use C in non-C++ compatible way (e.g. tentative definitions),
my source won't even compile for CppUnit.

If you mean K&R style prototypes, don't use them. Write and compile
your tests in C++ and your code in C. Don't attempt to compile your C
with a C++ compiler.

And finally there is a port issue (it's an embedded type talking ). I
am proposing something that requires only the compiler.

Shouldn't matter for unit testing, develop and test on a hosted system.
If you require bits of the target environment, mock (simulate) them.

 

[Ark Khasin]

Ian Collins wrote:

Please correct me if I am wrong, but AFAIK CppUnit doesn't provide a
code execution trace, so it's pretty darn hard to prove code coverage.
[There must be reasons why testing tools vendors command big money.]

If you develop your software test first, you get all the code coverage
you need.

Test first is a nice model but not of a universal applicability.
Besides, I need to demonstrate test coverage to the certifying/auditing
entity.
OTOH, I wonder if the proposed instrumentation can be made a part of
CppUnit. I think, there is nothing in either that would prohibit it.

> Write and compile
your tests in C++ and your code in C. Don't attempt to compile your C
with a C++ compiler.

Right. It just didn't occur to me

Shouldn't matter for unit testing, develop and test on a hosted system.
If you require bits of the target environment, mock (simulate) them.

The farthest I can go away from the target is a software simulator of
the instruction set. Same compiler, same version, perhaps, more
"memory". I think I am not alone in this...

-- Ark

 

[Phlip]

Besides, I need to demonstrate test coverage to the certifying/auditing
entity.

It sounds like learning what their requirements are and meeting them is more
important than guessing if TDD will incidentally meet their requirements.

Test first is a nice model but not of a universal applicability.

You can keep the TDD thing a secret...

 

[Phlip]

Testing is not an intgeral component of formal methods intended to
reduce testing.

Colin Paul Gloster

Why would a formal method intend to reduce a Good Thing??

 

[Ian Collins]

Ian Collins wrote:

Please correct me if I am wrong, but AFAIK CppUnit doesn't provide a
code execution trace, so it's pretty darn hard to prove code coverage.
[There must be reasons why testing tools vendors command big money.]

If you develop your software test first, you get all the code coverage
you need.

Test first is a nice model but not of a universal applicability.
Besides, I need to demonstrate test coverage to the certifying/auditing
entity.

You are not alone in that, I'd suggest you take this to a TDD list for
advice.

The farthest I can go away from the target is a software simulator of
the instruction set. Same compiler, same version, perhaps, more
"memory". I think I am not alone in this...

Why?

 

[Colin Paul Gloster]

|--------------------------------------------------------------------------------------|
|"[..] |
| |
|Unit testing is an integral component of [..] "formal" [..] |
|models of development. [..] |
| |
|[..]" |
|--------------------------------------------------------------------------------------|

Testing is not an intgeral component of formal methods intended to
reduce testing.

Regards,
Colin Paul Gloster

 

[=?ISO-8859-1?Q?Erik_Wikstr=F6m?=]

Why would a formal method intend to reduce a Good Thing??

Testing is used to find errors, while formal methods are used to prove
that there are no errors, at least that's the goal. So if you can prove
that there are no errors why test for them?

 

[Richard Heathfield]

Erik Wikström said:

Testing is used to find errors, while formal methods are used to prove
that there are no errors, at least that's the goal. So if you can
prove that there are no errors why test for them?

"Beware of bugs in the above code; I have only proved it correct, not
tried it." - Donald E Knuth.

 

[Phlip]

Testing is used to find errors, while formal methods are used to prove
that there are no errors, at least that's the goal. So if you can prove
that there are no errors why test for them?

I use testing as a formal method, to prevent errors. I have not tried
the "proof" systems (and please don't try to tell the mathematicians I used
to hang out with that they are really "proofs").

You are describing writing and running tests in isolation from the
development process. Don't do that.

 

[Richard]

Erik Wikström said:



"Beware of bugs in the above code; I have only proved it correct, not
tried it." - Donald E Knuth.

Proof of Correctness depends very much on how and when "correct" is
used. It is a crock of shit in most day to day SW development.

 

[Flash Gordon]

Ian Collins wrote, On 27/08/07 08:14:

Why?

There are many possible *valid* reasons for this. One is that if you are
not using the same version of the same compiler with the same switches
then the code you are testing is not the same as the code that will be
run. Since compilers *do* have bugs it is possible that the bug will be
triggered in the real environment but not in the test environment unless
you ensure that they are the same.

If I was doing QA for a product I would insist than you either use the
same version of the same compiler or you provide testing to the same
level that the deliverable SW requires of *both* the compiler used for
test *and* the compiler used for the final SW. The more critical the SW,
the more insistent I would be on this, and the more testing you would
have to do on the compilers, for safety critical SW this would probably
kill the project dead if you did not use identical SW to build for test
and build for delivery. BTW, I *have* rejected SW and documentation at
review, and even told developers that there was no point in putting it
in for review because I would fail it.

 

[Ben Bacarisse]

Erik Wikström said:



"Beware of bugs in the above code; I have only proved it correct, not
tried it." - Donald E Knuth.

But this was a "by hand" proof in 1977. A machine assisted proof of
the actual code could be expected to inspire a little more confidence.

 

[Richard Heathfield]

Ben Bacarisse said:

But this was a "by hand" proof in 1977. A machine assisted proof of
the actual code could be expected to inspire a little more confidence.

Why? Presumably the machine that is doing the assisting is itself a
computer program. What makes you think the assistance program is
correct?

 

[Ian Collins]

Ian Collins wrote, On 27/08/07 08:14:

There are many possible *valid* reasons for this. One is that if you are
not using the same version of the same compiler with the same switches
then the code you are testing is not the same as the code that will be
run. Since compilers *do* have bugs it is possible that the bug will be
triggered in the real environment but not in the test environment unless
you ensure that they are the same.

But one has to differentiate between developer unit testing (the subject
of this post) and QA (customer) acceptance testing. The former can be
performed on any environment the developer chooses, the later must be
run on the target.

If I was doing QA for a product I would insist than you either use the
same version of the same compiler or you provide testing to the same
level that the deliverable SW requires of *both* the compiler used for
test *and* the compiler used for the final SW. The more critical the SW,
the more insistent I would be on this, and the more testing you would
have to do on the compilers, for safety critical SW this would probably
kill the project dead if you did not use identical SW to build for test
and build for delivery. BTW, I *have* rejected SW and documentation at
review, and even told developers that there was no point in putting it
in for review because I would fail it.

Again, this is different from developer unit testing, I don't think
anyone would be daft enough to release a product that hadn't been
through acceptance testing on the target platform.

 

[Ben Bacarisse]

Ben Bacarisse said:


Why? Presumably the machine that is doing the assisting is itself a
computer program. What makes you think the assistance program is
correct?

What do you test your software with if not more software?

If you think that a machine assisted proof would not inspire "a little
more" confidence than a hand proof, then I won't try to persuade you
(it was a modest enough claim) but the fact that a proof system is
software does not invalidate the method any more than testing is
invalidated by being done in software.

 

[Flash Gordon]

Ian Collins wrote, On 27/08/07 21:51:

But one has to differentiate between developer unit testing (the subject
of this post) and QA (customer) acceptance testing.

You have missed out internal formal testing which in many environments
is far more complete than acceptance testing. For example, I've worked
on projects where a formal test literally takes a week to complete but
the customer acceptance testing takes only a few hours.

Unit test can also be formal, and in a lot of environments, including
the afore mentioned safety critical projects, you are *required* to
perform formal unit tests.

The former can be
performed on any environment the developer chooses,

Informal testing can be run in any environment the developer has
available. Formal testing, which is the only sort of testing that you
can guarantee will be available and working for those maintaining later,
is another matter.

the later must be
run on the target.

All formal testing, whether unit testing or testing at a higher level
has to be run on code compiled with the correct compiler, although not
always on an identical target.

Again, this is different from developer unit testing, I don't think
anyone would be daft enough to release a product that hadn't been
through acceptance testing on the target platform.

Acceptance testing has very little to do with proving whether the system
works, it is just to give the customer some confidence. The real worth
while formal testing has to be completed *before* doing customer
acceptance testing and done with the correct compiler. At least, this is
the case in many environments, including all the projects where I have
been involved in QA, and on the safety critical project I was involved in.

If your customer acceptance testing is sufficient to prove the SW is
sufficiently correct then your customer has either very little trust in
your company or a lot of time to waste. If your customer acceptance
testing is the only testing done with the correct compiler and it is not
sufficient to prove your SW is sufficiently correct then your SW is not
tested properly. At least, not according to any standard of testing I
have come across.

 

[Richard Heathfield]

Ben Bacarisse said:

What do you test your software with if not more software?

A rolling pin. Any software that can withstand the pastry test is likely
to be able to withstand anything else too.

If you think that a machine assisted proof would not inspire "a little
more" confidence than a hand proof, then I won't try to persuade you
(it was a modest enough claim)

Yes, on reflection I see that I'm guilty of (accidentally) extending
your claim, which was indeed modest enough.

<snip>