How to use CPPUnit effectively?

[Gianni Mariani]

You can test the logic without running in an MT environment.

Same with testing something like a signal handler or interrupt service
routine, you don't have to send an event to test the code. You know the
underlying OS/hardware will deliver the event, your job is to build the
code that processes it.

That's the point. In a limited number of cases, you can't.

Show me what your timer test code looks like ?

 

[Ian Collins]

That's the point. In a limited number of cases, you can't.

Can't what? Build the code, or test it?

Code that provides the *lowest level* threading functionality has to be
tested in an MT environment, but a) I'd argue this doesn't constitute
unit testing and b) how often do you write that type of code, once per
project? Once per platform? Once?

If you are developing an event handler, decouple the processing logic
from whatever connects it to the source of the event. Care to provide
an example of where this can't be done?

Show me what your timer test code looks like ?

I don't have any, but I'm sure the OS developer does.

 

[Gianni Mariani]

Can't what? Build the code, or test it?

Appreciate the implications of the design.

Code that provides the *lowest level* threading functionality has to be
tested in an MT environment, but a) I'd argue this doesn't constitute
unit testing and b) how often do you write that type of code, once per
project? Once per platform? Once?

It depends on the asynchronous nature of the problem you're trying to solve.

If you are developing an event handler, decouple the processing logic
from whatever connects it to the source of the event. Care to provide
an example of where this can't be done?

Sure, the Austria C++ thread pool design unit test. Maybe the Austria
C++ timer unit test. Both of these tests must show how you deal with
the inherent multi threaded nature of the service you're designing
otherwise you're not really seeing all the design issues.

I don't have any, but I'm sure the OS developer does.

Actually you make my point, if they had a unit test before they coded
they probably would have changed the design. The O.S. timer facilities
are very difficult to use correctly. For example, most OS timer
services come back with a signal. This means you can't do anything that
takes a resource. About the only thing you can do safely is set a flag
or call some other "signal safe" OS calls (like write a byte to a pipe).

The code below shows what the Austria timer unit test "kernel" looks
like. Event(...) gets called back in the "timer service" thread. It
doesn't look like it but it is inherently multi threaded !


class MyTimerClient
: public TimerClient_Basic
{

TimerLog & m_log;
int m_client_number;

public:

MyTimerClient( TimerLog & io_log, int i_client_number )
: m_log( io_log ),
m_client_number( i_client_number )
{
}

protected:

virtual bool Reschedule(
const TimeStamp & i_current_time,
TimeStamp & o_reschedule_time
) = 0;

private:

virtual bool Event(
const TimeStamp & i_current_time,
TimeStamp & o_reschedule_time
)
{
m_log.push_back( m_client_number, ThreadSystemTime() );
return Reschedule( i_current_time, o_reschedule_time );
}

};

 

[Phlip]

That's the point. In a limited number of cases, you can't.

CppUnit and its ilk can be used for QA testing, but that's not what we've
been discussing. We are discussing TDD, where we write tests that help force
our lines of code to exist. This helps to refactor our designs as they grow.

If we are not writing a kernel, we don't need to TDD its event systems. We
won't refactor our kernel, so gaps in our test coverage are very low risk.

So you can safely rely on a policy "wait till we have a bug there". (Note
this is the policy that test-free programs already follow - for everything!)

Show me what your timer test code looks like ?

TEST_(TestDialog, SetTimer)
{
CButton slowButton = m_aDlg.GetDlgItem(IDC_SLOW_OPERATION);
slowButton.SendMessage(BM_CLICK);
BOOL thereWasaTimerToKill = m_aDlg.KillTimer(0);
CPPUNIT_ASSERT(thereWasaTimerToKill);
}

In Win32 (under WTL), we test that our BM_CLICK handler sets a timer by
killing that timer.

I'm aware that doesn't answer the exact question, but I already had this
answer available!

 

[jolz]

And if 1000 passes succedes that means that everything is ok? I don't

I tink that would be a bad test.

example of bad test removed. I can write bad tests too...

Sure it is bad. But how would look a good test that shows that c1 and
c2 may be equal? My point is that such a test doesn't exist.

You can't test every situation all the time, nobody argues that.

Somebody does:
1. > You should write tests for each new line (or two) of the code.
Never write

new code without a failing test.

2. > If you can't think of the test, how will you think of the line of
code?

I'm just trying to explain that "You can't test every situation all the
time", not that all tests are evil.
I also think that there are more situations that can't be tested than
those that can, but of course I may be wrong here.

 

[jolz]

Fight them, because they are bad.

And exactly why?

Fix the function so you can
time-slice its algorithm.

That is exactly what OS does. I don't want to rewrite part of OS
because I'm afraid of threads. I want to use code written by OS's
developers and focus on my own application.

we
shouldn't need them, and should resist adding them until we explore all
possible alternatives.

Again, why? Developer's only goal is not to aviod threads, espacially
if they are the best solution.

The question here is simple: Why and how does communication block your GUIs
thread?

- bacause it accesses disk which is slow
- bacause it uses network (not everybody work on windows and even then
not everybody uses networking with messages - Internet Explorer you
mentioned uses now on my computer 9 threads even if it doesn't do
anything currently)
- because I'v just executed sql which will finish in 5 minutes
- because my chess computer really trying to beat me
- because my fractal algorithm for some reason refuses to finish in 2
seconds
- because I'm converting a movie from divx to xvid
....
....

Threading violates encapsulation

thread 1: eatBreakfast()
thread 2: watchTV()

How does it violates encapsulation?
I have eggs on breakfast so:
private:
eatEggs()
But how do I do test it?
I know:
public:
eatEggs()
Fortunatelly in c++ I can make friends with eggs, but now I have test
reference in my code. Good thing is that it doesn't do anything wrong.
So test may break encapsulation (and may not), propelly designed thread
shouldn't.

A test here and there that can't fail is mostly harmless.

Obviously. So is scratching behind my ear. It also doesn't do anything
good.

To continue your digression here, one might ask what we expect to do without
tests when we change 1 global variable

I ment setting in the test. Global variables in a program if they exist
should be changed, possibly in an automated test so there will be no
surprises ater change from const int days_in_current_year from 365 to
366. Or possibly to 543. Its just a number, it should work.

Yes, because developers are running all the tests before integrating their
code. If any test fails, they don't submit. So the same tests must pass on
every machine, many times a day.

Test that last few days should pass on developer machine many times a
day?

What's so special about GUIs?

- it's main purpose is to beeing seen
- it requires user input which may not be predicted

 

[Marco]

For beginners to C++ Unit Tests the following article may be helpful:
Chuck Allison: The Simplest Automated Unit Test Framework That Could
Possibly Work - http://www.ddj.com/dept/cpp/184401279

Why beginners? I recommend this tool for all that don't want a GUI.

 

[Gianni Mariani]

CppUnit and its ilk can be used for QA testing, but that's not what we've
been discussing. We are discussing TDD, where we write tests that help force
our lines of code to exist. This helps to refactor our designs as they grow.

That is specifically what I talk about.

Hiding the MT nature of a service does not help you design it properly !

If your interface design is inherently MT then show me how to use it in
an MT environment, otherwise I can't evaluate how well it works.

If we are not writing a kernel, we don't need to TDD its event systems. We
won't refactor our kernel, so gaps in our test coverage are very low risk.

I disagree. I think this is why I have seen so many abortions of MT
code, because they don't take into account the big 3:

1. resource deadlock (not just mutexes)
2. reliable destruction
3. reliable event management

There are API's I have come across (WININET is one) that you cannot
close properly.

Also, you cannot return a handle from an MT call, it must be made
available to the caller *before* you return.

The point is, of the few classes where asynchronism is critical, you
cannot design it properly unless you take into account the MT nature of
the interface. If you think you can, then show me.

So you can safely rely on a policy "wait till we have a bug there". (Note
this is the policy that test-free programs already follow - for everything!)

After I'm done writing the code, I usually write at least one stress
test which usually finds a couple of bugs and occasionally bugs in the
interface itself.

TEST_(TestDialog, SetTimer)
{
CButton slowButton = m_aDlg.GetDlgItem(IDC_SLOW_OPERATION);
slowButton.SendMessage(BM_CLICK);
BOOL thereWasaTimerToKill = m_aDlg.KillTimer(0);
CPPUNIT_ASSERT(thereWasaTimerToKill);
}

Not a very useful timer interface, how can I reset the timer ? Can I
safely delete my client and see the timer get deregistered
automatically? Can I set the timer to an absolute time or an interval?

I'd like an interface that is as independent as possible of the O.S.
interfaces since it's a fundamental too.

Compare it to the Austria C++ timer test and I think you'll see what I mean.

I'm aware that doesn't answer the exact question, but I already had this
answer available!

The point is, you can't design something using TDD if you don't show how
it's done. If your interface is dealing with MT issues, there really is
little point in not doing an MT unit test for the purposes on TDD.

 

[Ian Collins]

The point is, you can't design something using TDD if you don't show how
it's done. If your interface is dealing with MT issues, there really is
little point in not doing an MT unit test for the purposes on TDD.

I think we'll have to agree to disagree on this point, I still think you
can't write an MT unit test in a TDD context. Why? Because it's all
but impossible to write a single pass MT test that has a 100%
predictable outcome. There are too many variables, how busy the machine
is, which thread starts first, is there more than one CPU available when
the test runs... So you end up getting unexpected and often
unrepeatable test failures.

Sure you can cure a lot of these inconsistencies by binding everything
to one CPU, but you aren't doing an MT test and may as well mock the
threading layer.

But you can TDD the logic of an MT object and use a separate Monte Carlo
type acceptance stress test for the MT functionality.

One side effect I found doing this was I could easily adapt objects
(like a message queue) from single threaded to MT and event models.
Same logic, different engine.

 

[Phlip]

I think we'll have to agree to disagree on this point, I still think you
can't write an MT unit test in a TDD context. Why?

http://groups.google.com/group/comp.object/browse_frm/thread/34c96ea7802f37b0/c371f34966e6c5e1

Hence

http://www.xprogramming.com/xpmag/acsDiningPhilosophers.htm

 

[Phlip]

Hiding the MT nature of a service does not help you design it properly !

You are correct that TDD is _a_ design technique, but it's not the only one.
MT requires more than emergent design. (Not much more, as Ron Jeffries's
paper noted, for one narrow situation.)

I disagree. I think this is why I have seen so many abortions of MT code,
because they don't take into account the big 3:

1. resource deadlock (not just mutexes)
2. reliable destruction
3. reliable event management

Right. That's neither TDD's fault, nor a good target for TDD's power.

Not a very useful timer interface, how can I reset the timer ? Can I
safely delete my client and see the timer get deregistered automatically?
Can I set the timer to an absolute time or an interval?

I didn't invent the Win32 timer interface. And we note with sadness that
even high-end OS APIs are often not designed for testing.

The point is, you can't design something using TDD if you don't show how
it's done.

Yes you can, for known algorithms, and for the majority of new code. Legacy
code considerations always interfere. MT is just one of the culprits. Any
kind of pre-existing code will taint the purity of your emergent design. All
this is still no excuse not to TDD your code, and not to emerge your own
designs.

If your interface is dealing with MT issues, there really is little point
in not doing an MT unit test for the purposes on TDD.

Can I get such a test to fail in a way that directs me to write new code?

Regardless of my reluctance to TDD the MT stuff, I suspect the answer is
yes!

 

[Gianni Mariani]

I think we'll have to agree to disagree on this point, I still think you
can't write an MT unit test in a TDD context. Why? Because it's all
but impossible to write a single pass MT test that has a 100%
predictable outcome. There are too many variables, how busy the machine
is, which thread starts first, is there more than one CPU available when
the test runs... So you end up getting unexpected and often
unrepeatable test failures.

I thought you argued earlier that unit tests for the purposes of design?
As for your unpredictability argument, umm what are you talking about.
MT code had better have a predictable outcome otherwise there is no
point in MT at all.

But you can TDD the logic of an MT object and use a separate Monte Carlo
type acceptance stress test for the MT functionality.

Yes. I've always seen it happen that way, TDD unit test followed by a
more rigorous test. What I am saying is that in the limited number of
interfaces that are MT specific, the TDD unit test case/s should
encompass the complexities of MT, otherwise you're design is likely to
come out incomplete.

 

[Gianni Mariani]

You are correct that TDD is _a_ design technique, but it's not the only one.
MT requires more than emergent design. (Not much more, as Ron Jeffries's
paper noted, for one narrow situation.)

Yes. The only point I am trying to defend is "in the limited number of
cases where the design is inherently MT, the TDD test case will help you
design better if it demonstrates how to manage it's MT nature of the
design". As for how to design as a whole, that a far more involved
discussion.

Right. That's neither TDD's fault, nor a good target for TDD's power.

The point I'm trying to make is that this can be considered unless the
TDD test case shows what how the problem is solved.

I didn't invent the Win32 timer interface. And we note with sadness that
even high-end OS APIs are often not designed for testing.

"Testability" is a primary deliverable.

Yes you can, for known algorithms, and for the majority of new code. Legacy
code considerations always interfere. MT is just one of the culprits. Any
kind of pre-existing code will taint the purity of your emergent design. All
this is still no excuse not to TDD your code, and not to emerge your own
designs.

That's what wrapping code in abstractions is for.

Can I get such a test to fail in a way that directs me to write new code?

Yes. I tend to make less mistakes today but just the act of writing the
test case usually helps me realize the more elegant ways of doing things.

Regardless of my reluctance to TDD the MT stuff, I suspect the answer is
yes!

MT TDD unit tests should be limited to just the designs that are MT in
nature. e.g. Timers, asynchronous comms layers, low level MT facilities
like threadpools.

I might be wrong but I think we're not saying anything different.

 

[Ian Collins]

I thought you argued earlier that unit tests for the purposes of design?

They are, the design of the logic.

As for your unpredictability argument, umm what are you talking about.
MT code had better have a predictable outcome otherwise there is no
point in MT at all.

I was referring to tests that start threads. On my last project we
started with several of these, by the end they were removed and the
treading layer tested in its own Monte Carlo type test harness.

Yes. I've always seen it happen that way, TDD unit test followed by a
more rigorous test. What I am saying is that in the limited number of
interfaces that are MT specific, the TDD unit test case/s should
encompass the complexities of MT, otherwise you're design is likely to
come out incomplete.

Maybe I should revisit TDD and threads. I've been bitten too many times
over the years by tests with multiple threads that I've grown accustomed
to using a different test style for that limited number of interfaces.
It may also be that I've implemented them so many times, I don't have to
design any more!

 

[Gianni Mariani]

Gianni Mariani wrote: ....
Maybe I should revisit TDD and threads. I've been bitten too many times
over the years by tests with multiple threads that I've grown accustomed
to using a different test style for that limited number of interfaces.
It may also be that I've implemented them so many times, I don't have to
design any more!

They are definitely limited in number. e.g. I would not do an MT TDD
test for a matrix library (as an example), even if the library used
threads to parallelize. Unless of course it needed an asynchronous
termination interface to terminate long running computations reliably.

A nice threading or pool interface makes life easier.

 

[Phlip]

"Testability" is a primary deliverable.

Yay! Now watch me try to bill hours for it! (To a client familiar with
hack-n-slash web programming ;-)

I might be wrong but I think we're not saying anything different.

I noticed that a while ago but I kept going...

 

[VJ]

VJ wrote:




I don't understand the brief amount of documentation I read.

Given test<3>, do I guess right that we have to increment that 3 by hand,
each time we

I want to smoke crack while coding, and having to remember what number I'm
up to interferes with my lifestyle. I can't even think of a macro that fixes
this (with a compile-time constant).

So does cxxtest really make me increment that number? Or should I, like,
read more of the documentation before passing judgement?

Not sure what you are talking about - I have not read their documentation

You call a function with specific parameters, and check what it outputs

Does a unit test library you are using increment that number for you?

 

[Phlip]

http://cxxtest.sourceforge.net/guide.html

Not sure what you are talking about - I have not read their documentation

I will start again from scratch.

I'm looking for a pattern called "Test Collector".

cxxtest implements it using a Perl script:

cxxtestgen.pl --error-printer -o runner.cpp MyTestSuite.h

That reads the source, pulls out every function named /test*/, and writes
runner.cpp to call them all.

(Parenthetically, I propose that going to another language just to avoid
macros is kind'a pathetic...)

That is still better than the CppUnit system of making you write the name of
each test case twice. (For TDD, test cases must be _absurdly_ easy to add!)

I seem to recall recently reading the documentation of a test runner that
uses templates to do Test Collector. Maybe not; I can't find them on the
above page. No worries.

 

[Gianni Mariani]

Yay! Now watch me try to bill hours for it! (To a client familiar with
hack-n-slash web programming ;-)

That's ok, keep on milking when they need all those urgent customer
fixes ... sad

 

[Gianni Mariani]

Phlip wrote:
....

I seem to recall recently reading the documentation of a test runner that
uses templates to do Test Collector. Maybe not; I can't find them on the
above page. No worries.

Austria C++'s unit test framework uses the Austria C++ generic factory
to collect all the unit tests into one place at startup. One of the
nice things about this is that you can place your unit tests entirely in
their own anonymous namespace so you can more-or-less cut-n-paste a test
from one file to the other and not have to worry about namespace
collisions. Depending on your debugger, you may or may not want to do this.