Unit testing - Breaking bad habits

[james.bartow]

I'm interested in practicing at least some level of Test Driven
Design. The problem is that I'm so set in my ways that it's hard to
see the "right" way to do this. Particularly as I do a lot of CRUD-
style Web work, my code doesn't resemble the simple examples very
much, and Mock objects isn't where I'd prefer to start.

Here's an example app: It comes in three parts: (This is the
simplified version)
One is a Bean that pulls XML messages off a JMS queue, parses them,
and inserts the resulting data into a DB.
The second piece is a client jar that presents a DTO. When the DTO is
populated, it can be passed to a send() routine that converts the DTO
data into XML and drops it on the JMS queue.
The third piece is a simple web-based search form that searches the DB
for matching terms and displays them.

Where should/should I not use tests?
Testing that each of the above pieces "work" seems too high level for
unit testing, but many of the components are dependent on dynamic data
and/or DB.
I can test that a given DTO will generate the correct XML, but do I
want to test that in one large pass, or do I run a different test for
each DTO data element?
Likewise how do I test my XML parsing - I don't want to test that
XPath works, but testing that the data shows up in the DB seems too
late for unit testing.
And testing the web form just confuses me. I don't want to query the
DB, I don't want to test the webserver itself, but there's very little
logic to test outside of the framework. I know I can try higher-level
testing packages to test the full process, and I eventually want to,
but I'm trying to learn TDD here.

In this case it's a completed project, but understanding what I would
have done (and writing the tests) will make it easier to incorporate
testing from the start on the next project. Can anyone offer me some
direction? My googling has found only material that assumes you're
experienced with unit testing, and I feel like I'm missing some
essential "ah-ha!" moment. Books like "Pragmatic Unit Testing"
offered very good examples - for code that isn't much like mine.

Thanks in advance

 

[dcest61]

I'm interested in practicing at least some level of Test Driven
Design. The problem is that I'm so set in my ways that it's hard to
see the "right" way to do this. Particularly as I do a lot of CRUD-
style Web work, my code doesn't resemble the simple examples very
much, and Mock objects isn't where I'd prefer to start.

Here's an example app: It comes in three parts: (This is the
simplified version)
One is a Bean that pulls XML messages off a JMS queue, parses them,
and inserts the resulting data into a DB.
The second piece is a client jar that presents a DTO. When the DTO is
populated, it can be passed to a send() routine that converts the DTO
data into XML and drops it on the JMS queue.
The third piece is a simple web-based search form that searches the DB
for matching terms and displays them.

Where should/should I not use tests?
Testing that each of the above pieces "work" seems too high level for
unit testing, but many of the components are dependent on dynamic data
and/or DB.
I can test that a given DTO will generate the correct XML, but do I
want to test that in one large pass, or do I run a different test for
each DTO data element?
Likewise how do I test my XML parsing - I don't want to test that
XPath works, but testing that the data shows up in the DB seems too
late for unit testing.
And testing the web form just confuses me. I don't want to query the
DB, I don't want to test the webserver itself, but there's very little
logic to test outside of the framework. I know I can try higher-level
testing packages to test the full process, and I eventually want to,
but I'm trying to learn TDD here.

In this case it's a completed project, but understanding what I would
have done (and writing the tests) will make it easier to incorporate
testing from the start on the next project. Can anyone offer me some
direction? My googling has found only material that assumes you're
experienced with unit testing, and I feel like I'm missing some
essential "ah-ha!" moment. Books like "Pragmatic Unit Testing"
offered very good examples - for code that isn't much like mine.

Thanks in advance

I'm used to writing individual tests that roughly correspond
to "interesting" methods. I guess it is something of a judgment call as to
when a method becomes "interesting", but at a minimum this includes methods
that cannot be verified fairly easily by inspection. It's also not a bad
rule of thumb to test methods that incorporate error handling. As you've
said yourself, you obviously don't want to write tests that check library
code.

In your first scenario, I wouldn't see much need to test the XML parsing
separately from putting the data into the DB. Just write a test for each
varying type of XML message, including possible erroneous inputs. If you
have your own findX() methods to retrieve DB data test those by
themselves...for the XML message parsing just use straight hardwired SQL
queries to return results that you can assert against.

For the second piece, the interesting bit is DTO -> XML. You mention "given
DTOs", so more than one...it can't hurt to have a test for each type. I
myself would have XML test output already prepared that can be compared to
the DTO->XML converter; the exact details vary. You're probably not going
to be too worried about testing whether or not JMS works.

The third piece - testing a simple form - is not as difficult as it sounds.
You're basically testing the server-side code that grabs the form inputs
and decides whether or not they are sensible. You'll end up doing some
paper and pencil work here, but look at each form input and think
creatively...what combinations of inputs can you come up with? What can
conceivably be put into a text input? This is in fact black box testing.
HttpUnit is an example of a testing framework that can be used for this.

Hope this helps.

AHS

 

[Lew]

I'm used to writing individual tests that roughly correspond
to "interesting" methods. I guess it is something of a judgment call as to
when a method becomes "interesting", but at a minimum this includes methods
that cannot be verified fairly easily by inspection. It's also not a bad
rule of thumb to test methods that incorporate error handling. As you've
said yourself, you obviously don't want to write tests that check library
code.

Unit tests should be written for all public and protected methods.

Use fake data or stubs for extrinsic elements, e.g., the database.

 

[Dirk Michaelsen]

Unit tests should be written for all public and protected methods.

Lew, I disagree here. It doesn't make any sense to test such simple
things like getters and setter or any CRUD operations on the database,
especially when you use some ORM framework. That's just a waste of
(rare) time. You should use a test coverage tool instead to decide if
the coverage level satisfies you (and your boss).

But I agree that you have to test any business logic on any
application layer higher than DAOs. I am doing test first development
here almost exclusively.

Use fake data or stubs for extrinsic elements, e.g., the database.

Another option is to use something like Springs
AbstractTransactionalSpringContextTests. You can arrange your test
data in the real database in the onSetUp method of the test case and
the benefit is, that the test class does a rollback after the test has
run.

Dirk

 

[Daniel Pitts]

Unit tests should be written for all public and protected methods.

Use fake data or stubs for extrinsic elements, e.g., the database.

For Test Driven Development, you write a test to verify a single
use-case, and then write the minimum amount of code to make the test
pass. Once it does, then you refactor.

This allows you to test the overall functionality of the system, without
worrying about whether you have a specific test for all public/protected
methods. It also helps prevent creating public methods (and
corresponding tests) that don't further your goal.

Now, if only I actually followed this advice

 

[Lew]

For Test Driven Development, you write a test to verify a single
use-case, and then write the minimum amount of code to make the test
pass. Once it does, then you refactor.

This allows you to test the overall functionality of the system, without
worrying about whether you have a specific test for all public/protected
methods.  It also helps prevent creating public methods (and
corresponding tests) that don't further your goal.

If I read you correctly, that results in public methods that support
use cases, and no other public methods, so the test suite ends up
covering all public methods. Is that right?

 

[Lew]

Lew, I disagree here. It doesn't make any sense to test such simple
things like getters and setter

Getters and setters are properties. I agree with you to a degree about this,
although I see absolutely no harm in testing them. However, I stand by my
opinion wrt non-property methods.

or any CRUD operations on the database, especially when you use some ORM framework.

I disagree here. These are critical operations and should be tested, albeit
not necessarily unit-tested. Since data operations are the heart of such
applications' back ends, they'd better work.

That's just a waste of (rare) time. You should use a test coverage tool instead to decide if
the coverage level satisfies you (and your boss).

?

First you say don't test, then you say do test. Which?

 

[Dirk Michaelsen]

First you say don't test, then you say do test. Which?

come on Lew, you had a bad day? If you would have read the next
paragraph of my original post you really wouldn't have to ask that
question ;-)

I test business logic and almost everything above the DAO layer. And
when I test business logic I automatically test the DAO layer,
especially when I use Springs transactional test case to rollback my
database changes after the test has completed.

I intensively test my applications but I'm not eager to reach a test
coverage of 100%.

Dirk

 

[Arved Sandstrom]

Getters and setters are properties. I agree with you to a degree about
this, although I see absolutely no harm in testing them. However, I stand
by my opinion wrt non-property methods.

And I stand by my characterization of methods as either being "interesting"
or not. Simple getters and setters are not, therefore I don't test them. I'm
also not going to test other simple methods that are clearly correct by
inspection.

I disagree here. These are critical operations and should be tested,
albeit not necessarily unit-tested. Since data operations are the heart
of such applications' back ends, they'd better work.

I test this stuff at what I think is a fairly common level. My business
logic is not normally aware of JPA, just as it wouldn't be aware of JDBC -
there is a data access layer. So a single JUnit/JUnit test may add three
POJOs/POCOs of class A to the "repository" (a DAL class), and then do a
"find" through that same repository on a subset of those objects. That way
I'm not testing the ORM itself, but rather my use of it.
[ SNIP ]

AHS

 

[Lew]

...

I test this stuff at what I think is a fairly common level. My business
logic is not normally aware of JPA, just as it wouldn't be aware of JDBC -
there is a data access layer. So a single JUnit/JUnit test may add three
POJOs/POCOs of class A to the "repository" (a DAL class), and then do a
"find" through that same repository on a subset of those objects. That way
I'm not testing the ORM itself, but rather my use of it.

This is consistent with my recommendation.

 

[Daniel Pitts]

If I read you correctly, that results in public methods that support
use cases, and no other public methods, so the test suite ends up
covering all public methods. Is that right?

Right, but the Unit tests test one unit of use, which may be several
public method calls in succession. Granted it might be better to
condense this use into a single method, but the there are times when you
need to have a series of calls instead of one.
The point is that you don't need to have a one-to-one
test-to-public-method in order to be complete. You might need multiple
tests for one public method. You might have a particular public method
is called from a test, but the test is concerned with a use-case
centered around a different public method.

It is /likely/ that you will end up with a one-to-one, but not
necessarily required or sufficient.

Keep in mind, that what I'm talking about is specifically TDD testing.
TDD is different than correctness/robustness testing. In order to
properly test correctness and robustness, you should have a test for
each edge-case, and a typical case. TDD can be considered "black box",
where you know the use-cases you care about, but GIGO applies. As oppose
to a full correctness test which is a clear-box test to verify that all
possible execution paths produce expected results.

 

[Mark Space]

Keep in mind, that what I'm talking about is specifically TDD testing.
TDD is different than correctness/robustness testing. In order to
properly test correctness and robustness, you should have a test for
each edge-case, and a typical case. TDD can be considered "black box",
where you know the use-cases you care about, but GIGO applies. As oppose
to a full correctness test which is a clear-box test to verify that all
possible execution paths produce expected results.

The problem that I see with TTD and other similar methodologies, is that
they want you to "refactor" as necessary when you add new tests and new
use cases. Well, that's not too hard for the second one, you only have
one previous test to refactor.

For the third test, you have 2. For the fourth test/use case you add,
you have 3 previous cases to refactor. Etc. So this methodology seems
to cause (n-1)! refactorings, which is pretty nasty if you ask me.

What ever happened to good old planning ahead? Maybe I misunderstand
how TTD really works, but it seems like it's going to run into a brick
wall eventually.

 

[Daniel Pitts]

The problem that I see with TTD and other similar methodologies, is that
they want you to "refactor" as necessary when you add new tests and new
use cases. Well, that's not too hard for the second one, you only have
one previous test to refactor.

For the third test, you have 2. For the fourth test/use case you add,
you have 3 previous cases to refactor. Etc. So this methodology seems
to cause (n-1)! refactorings, which is pretty nasty if you ask me.

What ever happened to good old planning ahead? Maybe I misunderstand
how TTD really works, but it seems like it's going to run into a brick
wall eventually.

Refactoring in TDD is used only to clean up after you add stuff. When
you add, you aren't supposed to care about duplicate code or patterns or
design, only about adding the one small feature. Then, once you have
"proved" the feature (Tests pass), do a small round of clean-up. That
doesn't mean you have to refactor every test, only the part of the
design that is affected by the code that you just added.

Also, refactoring tools make it easy to clean up large code-bases in
O(1) human-time (for example, rename method does all the search/replace
for me.) The unit tests help verify that the refactoring tool did its
job appropriately.

 

[Mark Space]

Refactoring in TDD is used only to clean up after you add stuff. When

for me.) The unit tests help verify that the refactoring tool did its
job appropriately.

You switched here from what sounded like system test to unit test, so
I'm still concerned that I completely miss the point.

At first, TDD sounded like the tests where based on use cases. "The
user selects New->File from the menu bar." That would be kinda chunky
right there to write a new test case for, but that aside, that's what I
think of as a Use Case. Something relatively coarse and high-level.
But it seems like TDD would fail utterly at this level with out some
other design step(s).

Now if TDD is something that an individual developer uses to implement
his or her small part of a design -- something that's already been
decomposed into manageable chunks -- well that makes a lot more sense.

 

[Tom Anderson]

For Test Driven Development, you write a test to verify a single use-case,
and then write the minimum amount of code to make the test pass.

'User story', rather than 'use case' - user stories are more fine-grained
than orthodox use cases. Of course, both terms are used to mean a wide
range of things in the wild, so this is perhaps just pedantry.

However, a more significant point: the tests you write to police user
stories aren't unit tests, they're acceptance tests, aka functional tests.
They don't test individual components of the system, they test larger
aspects of the system end-to-end, implicitly testing many components and
their interactions. Unit tests are the smaller tests that just test one
component, or one behaviour of one component.

For instance, if have a ShoppingCart, a unit test might look like:

ShoppingCart cart = new ShoppingCart() ;
assertEquals(0, cart.numberOfProducts()) ;
assertEquals(0, cart.numberOfItems()) ;
Product p = new Product("2381497", "Pork pie (pack of 6)") ;
cart.add(p) ;
assertEquals(1, cart.numberOfProducts()) ;
assertEquals(1, cart.numberOfItems()) ;
cart.add(p) ;
assertEquals(1, cart.numberOfProducts()) ;
assertEquals(2, cart.numberOfItems()) ;

This can't possibly correspond to a user story, can it? How could there be
a user story about how objects behave? Users can't see objects!

Now, you could have an acceptance test analogue of this. You can imagine a
user story "users can add products to their cart, and when they do,
they're told how many items, and how many different products, are in it".
You might write a test for this using HtmlUnit like this (apologies for
those not familiar with HtmlUnit or the page design i have in my head - i
hope this makes some sense anyway):

WebClient client = new WebClient(BrowserVersion.FIREFOX_3) ;
HtmlPage page = client.getPage("http://localhost/store/cart") ;
assertEquals("0", page.getHtmlElementById("productcount").getTextContent()) ;
assertEquals("0", page.getHtmlElementById("itemcount").getTextContent()) ;
page = client.getPage("http://localhost/store/product/2381497") ;
assertEquals("Pork pie (pack of 6)", page.getHtmlElementById("productdescription").getTextContent()) ;
ClickableElement purchaseLink = page.getFirstByXpath("//[text() = 'Purchase']") ;
page = purchaseLink.click() ;
assertEquals("http://localhost/store/cart", page.getWebResponse().getUrl()) ;
assertEquals("1", page.getHtmlElementById("productcount").getTextContent()) ;
assertEquals("1", page.getHtmlElementById("itemcount").getTextContent()) ;
page = client.getPage("http://localhost/store/product/2381497") ;
ClickableElement purchaseLink = page.getFirstByXpath("//[text() = 'Purchase']") ;
page = purchaseLink.click() ;
assertEquals("http://localhost/store/cart", page.getWebResponse().getUrl()) ;
assertEquals("1", page.getHtmlElementById("productcount").getTextContent()) ;
assertEquals("2", page.getHtmlElementById("itemcount").getTextContent()) ;

At first glance, this looks very similar (okay, at first glance, it looks
completely different, but YKWIM). However, if you think about it, for this
test to pass, so much more is required than that ShoppingCart passes the
unit test above. For starters, the JSPs have to be working. Session
management has to be associating a cart with a user. Product lookup in the
database has to be working okay. The web server has to be set up right.
Lots of other under-the-hood details have to be right. Accordingly, if
this test fails, you can't immediately conclude that there's a problem
with Shopping Cart, as you can with the unit test. On the other hand, if
it passes, you can immediately conclude that a whole bunch of components
are doing okay, which you can't with the unit test.

Again, this is the way it is in XP orthodoxy (orthodoxpy?). Processes and
names vary wildly. Indeed, where i work, we didn't really have unit tests
for our last client project at all. The gurus decided that instead of
writing small-scale unit tests, we'd just write loads and loads of
acceptance-style tests, all coming in through the web interface, and so
testing some aspect of the system end-to-end, and use those in place of
unit tests.

This decision was made long before i joined, and i've never talked to them
properly about why, but i think the idea is that unit testing parts of
this monstrous system (built using a very complex and all-pervading
closed-source third-party app framework) in isolation would have been very
difficult and not very rewarding, and that the acceptance tests were
simple to write, and actually gave us much more thorough testing. The
downside is that they're slower (it takes several minutes to run the whole
lot - which is still not too bad) and don't automatically tell you where a
problem lies when one is detected. Still, it did the job, and we shipped
many releases with long green bars to the customer.

A cool byproduct of this came when we modded our test framework to log
what was happening - every link being clicked, every assertion being made
- and then did a bunch of funky javascript that used the log to draw a
graphical overlay on the pages being tested (live pages, from the actual
app) showing what was happening in a client-readable form. That meant that
for hundreds and hundreds of different actions on the site, we could show
them "you go here, click this, then verify that this field says this, then
that when you click on this element, this box appears and says this ...".

tom

 

[Tom Anderson]

I'm interested in practicing at least some level of Test Driven Design.
The problem is that I'm so set in my ways that it's hard to see the
"right" way to do this. Particularly as I do a lot of CRUD- style Web
work, my code doesn't resemble the simple examples very much, and Mock
objects isn't where I'd prefer to start.

I think this is actually a hard place to be in for unit testing. In the
post i've just made in this thread, i talk about how we did testing on a
web app, where we only wrote end-to-end tests. I think the theory there
applies a fortiori to your situation, where there's a paucity of objects
in the middle that are worth testing. You might find the best option is
just to test end-to-end: send some messages to the app, and then hit the
search form with some queries and verify what comes back. I'm not sure
what the client jar component does - some kind of GUI? I don't know how
you go about testing that, but i'm sure it can be done.

Alternatively, unit tests you could write:

Here's an example app: It comes in three parts: (This is the
simplified version)
One is a Bean that pulls XML messages off a JMS queue, parses them,
and inserts the resulting data into a DB.

Factor this so that DB access is through an interface which looks like:

interface DataStore {
public void addProduct(String id, String name, String description) ;
public void addSpecialOffer(String id, String productId, int percentOff, Date start, Date end) ;
}

Unit test 1: make an instance of this, hit it with some calls, then verify
that the database looks how it should (by going in directly with JDBC).

Now write an implementation of this which sits on top of your actual
database, so that the test can pass.

Write a mock implementation of the data store interface which just logs
all calls as objects in a set (the objects could be struct-like things, or
even strings, like "addproduct\t2381497\tPork pie (pack of 6)\tHalf a
dozen delicious, nutritious pork pies.").

Write a mock JMS Message. A pain, but not really difficult. It's really
only getText that needs a non-empty implementation; an IDE will write most
of this class for you.

Unit test 2: create a mock data store, set up the bean to use it, and fire
a message at it by calling its onMessage method directly - don't set up an
actual JMS queue. Retrieve the set of logged calls from the data store,
and make sure it looks how it should. Like:

Set<String> calls = mockStore.getCalls() ;
assertEquals(3, calls.size()) ;
assertTrue(calls.contains("addproduct\t2381497\tPork pie (pack of 6)\tHalf a dozen delicious, nutritious pork pies.")) ;
assertTrue(calls.contains("addspecialoffer\t2381497\t50\tThu Nov 06 12:00:00 GMT 2008\tThu Nov 13 12:00:00 GMT 2008") ;
assertTrue(calls.contains("addproduct\t8529034\tSausage roll\tA sausage roll made with only the highest-quality pork shoulder (the gristle, mostly; some bone).")) ;

Write the bean, so that the test passes.

Acceptance test 1: start a J2EE instance with a JMS queue plugged into
your bean, which is plugged into the database. Fire some messages at the
queue. Make JDBC calls to verify that the database looks how it should.

Unit test 1 is dispensible if the data store is really, really simple.
However, if it's doing anything clever, like inserting into several tables
for each call, then you should test it.

For all of these, you need to write a test for each different input - if
you have five things that can be stored, that's five versions of unit test
1. More, in fact, to test that you can store one thing after another etc.
Much of this will be cut-and-paste programming, or will involve factoring
out helper methods like storeProduct() and validateStoredProduct(). Ditto
for test 2 - a test for each message type, or for a representative sample
of message types if they're open-ended.

The second piece is a client jar that presents a DTO. When the DTO is
populated, it can be passed to a send() routine that converts the DTO
data into XML and drops it on the JMS queue.

Define two interfaces, DataSender and MessageSender, which are given a DTO
and a message respectively. The GUI makes a DTO and gives it to the
DataSender, which constructs a message and gives it ot the MessageSender,
which sends it to the server.

Mock up the DataSender interface to just store the DTO.

Unit test 1: set up the GUI with a mock DataSender and click some buttons
(somehow), ending with the send button. Verify that the DTO stashed looks
how it should.

Write the GUI, so the test passes.

Mock up the MessageSender to store the message.

Unit test 2: set up the DTO-to-XML converter with a mock MessageSender,
give it a DTO, and verify that the message it comes out with looks good.

Write the DTO-to-XML converter, so the test passes.

Unit test 3: set up a JMS queue. Set your MessageSender up to point to it.
Give it a message to send. Verify that you can then pull a correct message
off the queue. This might be hard, because of the involvement of JMS.

Write the MessageSender, so the test passes.

Functional test: set everything up, click on the GUI, read a message off
the queue and verify it.

The third piece is a simple web-based search form that searches the DB
for matching terms and displays them.

Define a search interface for the DB.

Unit test 1: poke some values into a database, connect the search
interface to it, fire off some queries, verify the results.

Implement the search interface, so the test passes.

Mock up the search interface to return preset answers to some set of
queries.

Unit test 2: using something like HtmlUnit, go to the search form, enter
your set queries, and verify that they're presented properly on the page.

Implement the web interface, so the test passes.

Functional test: poke values into the database, set up the whole
interface, do some queries.

I can test that a given DTO will generate the correct XML, but do I
want to test that in one large pass, or do I run a different test for
each DTO data element?

My googling has found only material that assumes you're experienced with
unit testing, and I feel like I'm missing some essential "ah-ha!"
moment.

I know what you mean.

Your starting point is being able to think "what am i trying to do here?"
at any point. If you can do that, you have something that can be tested.
Indeed, writing it as a test is a fantastic way of getting your idea of
what you're trying to do worked out precisely. The next step is to figure
out what grain you need to do that on - do you write tests for the goal of
a day's work, five minutes' work, what? That's a tricky problem, and one
that i don't have an answer to. Indeed, i think it's a matter of much
disagreement and debate.

One thing i will urge you to do with the utmost intensity is to Capture
Bugs With Tests. If at any point you realise you have a bug, of any
description, before you set about fixing it, write a test (or tests) that
captures it - that fails because of the bug, will pass when it's fixed,
and will fail again if you ever reintroduce the bug - at the finest
possible grain. Observing the granularity at which you end up writing
these tets may give you a clue to the grain at which you need to write
your other tests.

Books like "Pragmatic Unit Testing" offered very good examples - for
code that isn't much like mine.

Your system isn't really that different to 'normal' systems - you just
have to realise that a web interface, GUI or message bean interface is
just another interface that can be interacted with. You'll need funky
tools to do that interaction, but semantically, clicking the "add item"
button on a GUI is just like calling the addItem method on an object.

tom

 

[Tom Anderson]

The problem that I see with TTD and other similar methodologies, is that
they want you to "refactor" as necessary when you add new tests and new
use cases. Well, that's not too hard for the second one, you only have
one previous test to refactor.

For the third test, you have 2. For the fourth test/use case you add,
you have 3 previous cases to refactor. Etc. So this methodology seems
to cause (n-1)! refactorings, which is pretty nasty if you ask me.

Why are you refactoring tests? Or rather, why is refactoring your system
causing you to change test?

You'd only need to change a test if the interface to the system has
changed, or if the definition of what correct behaviour is has changed,
and changed in such a way that that test is affected. Hopefully, that
doesn't happen very often - certainly far less often than you refactor.
Refactorings should be changes to the internals of the system which are
invisible at the interface.

For this to be the case, you obviously need quite loose coupling, a
desirable state in itself. The pain of having to change lots of tests when
refactoring will thus also serve to drive you towards this!

tom

 

[Patricia Shanahan]

Why are you refactoring tests? Or rather, why is refactoring your system
causing you to change test?

You'd only need to change a test if the interface to the system has
changed, or if the definition of what correct behaviour is has changed,
and changed in such a way that that test is affected. Hopefully, that
doesn't happen very often - certainly far less often than you refactor.
Refactorings should be changes to the internals of the system which are
invisible at the interface.

For this to be the case, you obviously need quite loose coupling, a
desirable state in itself. The pain of having to change lots of tests
when refactoring will thus also serve to drive you towards this!

tom

Are we talking about system tests, or unit tests?

A unit test should be testing some unit, such as a class or method, and
will need to be changed to track refactoring affecting the unit it
tests. A system test tests some feature of the whole system, and should
change only for changes in the system's interfaces or specification.

Patricia

 

[Tom Anderson]

Are we talking about system tests, or unit tests?

A unit test should be testing some unit, such as a class or method, and
will need to be changed to track refactoring affecting the unit it
tests.

Yes, completely (if the refactoring changes the interface - it often
doesn't). But changes in one unit should only affect the tests for that
unit, and maybe a handful of others. I got the impression that Mark was
worried about refactoring leading to changes in millions of tests - or
(n-1)!, to be precise.

tom

 

[Harold Yarmouth]

I think this is actually a hard place to be in for unit testing. In the
post i've just made in this thread, i talk about how we did testing on a
web app, where we only wrote end-to-end tests.

Those end-to-end tests can sometimes be exceptionally simple.

In particular, today I performed an end-to-end test of a function of a
system by:

* Right click main class in class browser in NetBeans
* Pick "Run".
* Examine left half of resulting on-screen JFrame.
* Close the JFrame.
* Answer "Yes" to "Are you sure?" prompt.

Possibly the shortest test ever, short of "java -jar HelloWorld.jar",
and testing a whole lot more.

I doubt I'll have such a short-and-sweet test again in my lifetime,
though. This particular app, on startup, queries a DB with the last-used
query and uses it to populate a JList control with objects for the user
to manipulate. I'd coded the actual querying functionality (~1500LOC in
15 classes, most nested/inner in a Query class, with quite a bit of
delegation, inversion of control, and strategy pattern, plus conversion
of Query objects to a string representation and parsing of same) over a
period of 2 days and then changed the application code that populated
the JList to go through the Query class instead of using a hardcoded
particular query string sent to the backend. Essentially, Query was
written and dropped in place of a former mock object.

The testing exposed a grand total of three bugs, all of which had been
fixed roughly five minutes after test/debug cycles began.

One was a typo -- child1 and child1 instead of child1 and child2 (did
you just guess that maybe there are trees involved? ), one was a
subtler bug with the same consequences (presence of item X in collection
Y was used where presence of as many occurrences of X in Y as in Z was
what was needed -- fix was to add one line of code), and one was a
devilish bug in the output-sorting routine (every comparator derived
from a base class designed to fall back on a modification-date
comparator on otherwise-equal items -- including the modification-date
comparator, which therefore blew up with StackOverflowError. Fix was to
make the modification-date comparator a special case by putting a bit of
code in the base class; about four lines of code, as I wrapped the
problem line in if (!(this instanceof DateComp)) { and three lines }
else {, doSomethingElse() (not the real name, but DateComp is), and }.)

The first two bugs caused a devilish result -- nothing. The JList came
up empty, no exception trace, no other signs of incorrect behavior.
Silent failure is always the hardest thing to track down (except maybe
for inconsistent, intermittent, random failure, which usually boils down
to concurrency bugs in the case of a Java system, and is much more
common and has even more devilish causes in C/C++ systems). My first
resort was to suspect the code that optimized a tree somewhere -- an
unoptimized tree would work but be slow. I disabled the call to the
optimizer and it worked. Checked the part of the optimizer code relevant
to the query tested, starting with the prime suspect code, a bit that
decides if something is redundant and throws it out. There I immediately
spotted the typo, and saw by direct inspection that it would indeed
cause an empty JList at the frontend.

Reran the system, with the optimizer reenabled -- empty JList. Arrrgh!
Back to the same block of code, and found that the code actually
removing redundant items was removing every occurrence instead of one
per call. So when two parts of the query parse tree evaluate identical
and are logical-or'd together (think "return all widgets that are green
OR blue" -- so the output should be everything matching the first and
everything matching the second -- and then think the default initial
query if no saved file somehow wound up "return all widgets that are
green OR green") it would go to remove exactly one of them, but the
remove code would remove both (think "return all widgets that are
<null>", which query obviously matches no widgets).

Fixed it to remove only one item per remove and reran the app -- right
click main class, "Run", one glance, everything is working.

Now THAT is an end-to-end test.

(Next up: a user interface for querying. Right now, on shutdown it saves
the last-used query, on startup it retrieves it and uses it, and if it's
not found it uses a default query, though it still goes through Query
instead of directly invoking the backend with a hardcoded String
constant. The save/restore testing exposed a fourth bug in all that
code, which was a particularly stupid one: if (cached != null)
sb.append(cached); sb.append(cached = getFoo()); -- can you tell this
was refactored from an implementation that had a "return" inside the if
block? Only showed up as a parse error at startup after a sequence of
delete saved session state, successful startup, successful shutdown.
Fortunately the saved session state is a flat-ASCII file and it was easy
to see by directly inspecting it in a text editor that part of the query
was being repeated with no boolean operator between the repetitions,
causing a syntax error. But this bug was provoked only by a slightly
more involved test procedure, involving two successive startups from a
clean slate instead of one. On the other hand, on the second startup,
the error appeared before the JList or any of the rest of it did.)