Testing in C++

[Pete Becker]

No, it isn't. But it is a definition of regression. If you add test to
the end you can easily derive what the correct meaning should be from
the definition of the component words. Since regression means to move
back, then regression test must mean to test what was previously tested.

So "integral calculus" is all about basic arithmetic, since "integral"
in mathematics means "of or denoted by an integer." Parsing individual
words does not tell you what a compound word means.

Now, this is the common use, it fits the definition of the component
words as they are used in the English language. If you want to change
that definition to mean something else and then claim that the other use
is somehow wrong be my guest; the fact that you find one book that
coincides with your use of little import. Your claim of authority
likewise does not mean you are correct; I fully know who I am arguing
with, I own your book on TR1, and I still say your definition is flawed.

I think I will stick with common use as based on the English language
myself. It's how most people use the words:

"[Regression testing] is a quality control measure to ensure that the
newly modified code still complies with its specified requirements and
that unmodified code has not been affected by the maintenance activity."

http://www.webopedia.com/TERM/R/regression_testing.html

I don't dispute that this is common usage. I do regret the loss of a
previously useful term.

--

-- Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com)
Author of "The Standard C++ Library Extensions: a Tutorial and
Reference." (www.petebecker.com/tr1book)

 

[Noah Roberts]

Well, back in the day, we had unit tests, integration tests, acceptance
tests, and regression tests.


Yes, and that is apparently taken to mean that every test is a
regression test.

Regression test is running tests from previous versions that still apply
(ie are not being changed in the current iteration). In other words,
you have a product that parses XML. It's old and doesn't do schema.
You decide to add that feature. You build acceptance and integration
tests that test your new feature. You also want to make sure that it
still parses all the stuff it did before so your run a
_regression_suite_ at times additional to your new tests. It may be
that you do them both every time but then again you might have a lot of
old features so maybe you want to speed things up a bit by skipping the
regression and only running it once a night.

 

[Ian Collins]

That's not true. The added test tests what the developer or the tester
thinks caused the problem, after distilling the original defect report.
You still need to run the original code now and then, in case the
problem was more complex than the developer realized.

I think we are arguing at cross purposes here, what are tests run
against if not the original code?

Real code, yes. Real-world code, no. Most tests are fairly short,
intended to isolate a particular part of a specification. That's
important, but it's also important to incorporate real users in the
testing cycle, for example, through a beta test. They typically don't
find anywhere near as many problems as the internal testers do (back
when I managed Borland's compiler testing, over 90% of the defects were
found by internal testers), but the ones they find are often killers.

I see, you are talking about real world test cases. I agree these are a
must, but I tend to disagree with your description of tests. While
individual tests may cover individual user stories, specification
clauses or whatever, the test suite as a whole covers way more. If the
tests are designed by or with the customer, the application should be
being tested to ensure it does what it says on the tin in the way the
users expect.

The ability to achieve a very high test coverage with minimal labour
overhead is the biggest improvement offered with fully automated test
frameworks.

 

[Ian Collins]

Well, back in the day, we had unit tests, integration tests, acceptance
tests, and regression tests.

I see the trend these days, especially where one of the various agile
methodologies is being followed for the latter three being collapsed
into acceptance tests.

Yes, and that is apparently taken to mean that every test is a
regression test.

If every test catches the regression of the feature it tests, it is.

 

[Noah Roberts]

So "integral calculus" is all about basic arithmetic, since "integral"
in mathematics means "of or denoted by an integer." Parsing individual
words does not tell you what a compound word means.

There are other meanings to the word "integral", only one is about
integers. Integral in this case is a function that is derived through a
process called "integration" or, "the act or instance of combining into
an integral whole."

The dictionary has several definitions for the word including:
"consisting or composed of parts that together constitute a whole,"
which seems an apt description of what an integral function is.

If you want to stick to only those that are labeled "mathematics" there
is, "any of several analogous quantities. Compare improper integral,
line integral, multiple integral, surface integral."

In general, yes you can find the meaning of a phrase through analysis of
its composite words. If you couldn't communication would be difficult
if not impossible. Integral calculus is no exception.

 

[Ian Collins]

True, it is faster but still, running your regression suite can take
hours. Ours is parallel processed on many different PC's and still
takes 3+. So it's more expensive than running a set of tests for what
you're working on.

That's why you have the unit tests for what you are working on.

I didn't say it isn't an important step to add a new test for the bug.
What I'm saying is that "regression testing" means something different
and is equally important.

That's where we disagree, I see "regression testing" as an integral part
of "acceptance testing". Do you have separate regression and acceptance
tests, or a combined suite?

Eventually for sure. Immediately, not necessarily. It might be put in
the stack of immediate tests that run in every integration since it is
now something of current importance.

I see. we didn't do that very often. Once we had identified unit tests
that illuminated a bug, these were used to fix it and the validation was
performed during regular acceptance suite runs. This proved (at least
for us), to be the most effective approach.

But there may be a division between what you're currently working on and
the vast supply of tests in your regression. The real point is that a
true complete test includes the regression and not just the new stuff.

I agree, we just differ in approach.

 

[Pete Becker]

I think we are arguing at cross purposes here, what are tests run
against if not the original code?

Yes, sorry: by "the original code" I was referring to the failing code
submitted by the customer.

I see, you are talking about real world test cases. I agree these are a
must, but I tend to disagree with your description of tests. While
individual tests may cover individual user stories, specification
clauses or whatever, the test suite as a whole covers way more. If the
tests are designed by or with the customer, the application should be
being tested to ensure it does what it says on the tin in the way the
users expect.

The ability to achieve a very high test coverage with minimal labour
overhead is the biggest improvement offered with fully automated test
frameworks.

Frameworks do reduce the labor of running tests, but they do not in
themselves improve test coverage (except in the sense that you can run
more tests in a given time). Test coverage is the result of design and
implementation.

--

-- Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com)
Author of "The Standard C++ Library Extensions: a Tutorial and
Reference." (www.petebecker.com/tr1book)

 

[Pete Becker]

Regression test is running tests from previous versions that still apply
(ie are not being changed in the current iteration). In other words,
you have a product that parses XML. It's old and doesn't do schema. You
decide to add that feature. You build acceptance and integration tests
that test your new feature. You also want to make sure that it still
parses all the stuff it did before so your run a _regression_suite_ at
times additional to your new tests. It may be that you do them both
every time but then again you might have a lot of old features so maybe
you want to speed things up a bit by skipping the regression and only
running it once a night.

Tests from previous versions that still apply test current requirements.
Separating them from newly written tests that also test current
requirements is artificial.

--

-- Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com)
Author of "The Standard C++ Library Extensions: a Tutorial and
Reference." (www.petebecker.com/tr1book)

 

[Pete Becker]

In general, yes you can find the meaning of a phrase through analysis of
its composite words. If you couldn't communication would be difficult
if not impossible.

The meaning of modifiers is often determined by context, which a
dictionary doesn't understand. For example, if you look up "unit" in the
dictionary, its definition could lead you to believe that a "unit test"
is any test that you apply to your application, which is, after all, a
"unit".

--

-- Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com)
Author of "The Standard C++ Library Extensions: a Tutorial and
Reference." (www.petebecker.com/tr1book)

 

[Ian Collins]

Yes, sorry: by "the original code" I was referring to the failing code
submitted by the customer.

Ah, I see now, you were working with a compiler test suite.

 

[Pete Becker]

Ah, I see now, you were working with a compiler test suite.

Whoops, sorry: you're right, that's why I fell into that terminology.
So: the customer's test case isn't necessarily the same as the test case
you end up writing when you make the fix. The latter will often be
smaller and faster.

--

-- Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com)
Author of "The Standard C++ Library Extensions: a Tutorial and
Reference." (www.petebecker.com/tr1book)

 

[Noah Roberts]

The meaning of modifiers is often determined by context, which a
dictionary doesn't understand. For example, if you look up "unit" in the
dictionary, its definition could lead you to believe that a "unit test"
is any test that you apply to your application, which is, after all, a
"unit".

How so? Since unit is quite explicitly singular and meaning a single
part and applications are composed of several distinct modules and
components. I imagine someone on the outside of development might fall
into a definition like that but those that understand what composes an
application shouldn't. The first time I heard the term there was little
doubt to its meaning; it was almost completely self explainitory.

 

[Pete Becker]

How so? Since unit is quite explicitly singular and meaning a single
part and applications are composed of several distinct modules and
components. I imagine someone on the outside of development might fall
into a definition like that but those that understand what composes an
application shouldn't. The first time I heard the term there was little
doubt to its meaning; it was almost completely self explainitory.

Okay, I gave you too much leeway, and you dodged the issue. So let me
say it again:

The definition of "unit" could lead you to believe that a "unit test" is
any test that you apply to any of the executables that constitute your
application.

--

-- Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com)
Author of "The Standard C++ Library Extensions: a Tutorial and
Reference." (www.petebecker.com/tr1book)

 

[Noah Roberts]

Okay, I gave you too much leeway, and you dodged the issue. So let me
say it again:

The definition of "unit" could lead you to believe that a "unit test" is
any test that you apply to any of the executables that constitute your
application.

No, I didn't dodge the issue. I didn't say applications where composed
of several executables.

Yes, it could lead you to believe that unit test means test the whole
program but you would have to use an incomplete definition of unit and
have an incomplete knowledge of program development to fairly make that
assumption since unit quite explicitly means a single part and
applications are made up of several distinct pieces. I'm fairly certain
the first time you heard the phrase it didn't require a lot of
explanation either.

 

[Phlip]

Yes, it could lead you to believe that unit test means test the whole
program but you would have to use an incomplete definition of unit

To avoid arguments over definitions, select definitions that are distinct,
useful, and concise. "Distinct" means that any two engineers could apply the
definition (whether or not they agree it is useful), and get the same
results.

For example, a good definition of a "unit test" is this: "A test whose
failure implicates a well-defined unit of the code, requiring very little
search to find the bug".

The definition applies to a process - a test failing - and not to a static
analysis of the test's code. The definition could include multiple
executables, if the test's failure is still easy to search out.

The point of the definition is not to achieve semiotic perfection, it is to
avoid excessive searching when a unit test fails.

 

[Glen Dayton]

Talk about testing, not about tools. For example, have each member of
the class write a complete set of test cases for a program that reads
three numbers from stdin and tells you whether a triangle with sides of
those lengths is equilateral, isosceles, scalene, or impossible. Then
compare the solutions, and talk about testing principles that help
refine the set of test cases.

Unit testing is merely one aspect of software testing, but it
probably merits a complete course on its own.

Because unit testing should be light in time to encourage its
use at build time, I mostly recommend not using a framework at
all. The unit test program merely needs to report success or
failure to the build environment, and should be silent on
successes and verbose on failures.

Legacy code needs careful treatment. Most beginning programmers
will need to deal with existing code in the real world. They
will need to know about breaking dependencies, creating mock
objects, and test coverage.

The limitations of unit testing should also be covered. Unit
tests at build time just don't do the same job as Monte Carlo
testing with randomized loads (useful for multi-threaded
testing), but you don't want to want every build for a million
trial MC run.

I like Michael Feather's book Working Effectively with Legacy
Code. It contains good material on unit testing for both new
code and legacy code.

 

[Ian Collins]

Unit testing is merely one aspect of software testing, but it probably
merits a complete course on its own.

Because unit testing should be light in time to encourage its use at
build time, I mostly recommend not using a framework at all. The unit
test program merely needs to report success or failure to the build
environment, and should be silent on successes and verbose on failures.

Using a framework takes a lot of the donkey work out of writing unit
test suites. Most frameworks are silent on successes and verbose on
failures.

 

[Noah Roberts]

I like Michael Feather's book Working Effectively with Legacy Code. It
contains good material on unit testing for both new code and legacy code.

Yes, it's a very good book.