Need help designing some JUnit tests

[Rhino]

news:[email protected]:

[ SNIP ]

My theory is rather weak so I'm not really up on the meanings of
"black box" and "clear box" testing, let alone the subtle differences
between them. Also, I'm not writing the code for anyone but myself
right now, except that I _would_ like to get one project's code
looking as professional as possible so that I could present it to a
prospective client or employer as a portfolio of what I can do. (And
then imitate that in future projects as well as gradually retrofit
other existing projects with what I've learned). With that in mind,
would a reasonable employer/client likely find it acceptable that I
just tested the methods I wrote and overrode myself in my classes or
are they going to see me as the biggest idiot since the development
of COBOL if I fail to observe these subtleties?

No word of a lie, just the fact that you're writing tests already gets
you some serious points with employers or clients. And the huge
majority of people that matter will be perfectly happy with you
writing tests for a class that focus on methods that are lexically in
the class. That's what I do myself.

I have to say it is a bit frightening that there are big brownie points
for doing any testing at all: you'd think that would be absolutely
essential to any serious employer or client. I was just hoping to be
doing a reasonable amount of testing to look like I know what I'm doing.
No testing at all didn't seem like an option in any universe I want to
live in

You perhaps misunderstood Eric - he didn't mean that there is a subtle
difference between black box and white box testing. There isn't -
those two techniques are quite different. He simply called the
specific point he was making subtle.

You're right, I did misunderstand him in exactly the way you said.

You don't need to be a testing god - people make FT careers out of
understanding testing. But you should know what black box and white
box testing are, for example. The Wikipedia page on Software Testing
is not a bad start, IMO. Just bear in mind that their terminology and
definitions are debatable on some specifics, but this doesn't detract
from the overall discussion (in particular the first few sentences
under Non-Functional Testing are whacked, try to gloss over them).

Good points. I'll review the Wikipedia definitions and try to remember
the highlights.

You'll end up doing a fair bit of reading and playing with code to get
a good handle on testing overall, but it won't take all that long to
get a good grip on the basics.

I will make a few personal recommendations, which shouldn't be taken
as complete. This is based on going on ten years of J2EE work, so
depending on what kind of coding you're doing YMMV...some.

1. Code reviews/inspections and static code analysis with tools _are_
a form of testing. Keep them in mind.

I see that you've given some good concrete suggestions for tools to use
in upcoming paragraphs. Are any of these static code analysis tools? If
not, can you suggest some good ones? I'd love to have some feedback on my
code via code reviews too but that's a little difficult to do. It's not
realistic to full (non-trivial) classes in the newsgroup so I just pick
and choose especially concerning problems and post snippets reflecting
them. I'm sure that you folks are all too busy to do detailed
walkthroughs of dozens of classes with hundreds of lines of codes in many
of them ;-) I must see if there is a local Java user group in this area
and see if I can develop a few buddies for doing code walkthroughs....

2. Integration tests are as important as unit tests. In a J2EE web
application these are indispensable - you may have hundreds or
thousands of unit tests all pass but still have integration tests not
pass. You'll hear a lot of people, myself included, refer to a
specific class of integration tests as functional tests - we mean
"application functions from the user perspective" when we use
"functional" in that regard.

Examples of integration/functional tests in a J2EE web app range all
the way from ensuring that good things happen in the logic when you
click that "Transmogrify" button all the way to doing a complete pass
through a use case and making sure that your "Issue Fishing License"
logic works in Angler Administrator 2010.

Myself I use Selenium IDE/RC in conjunction with the JUnit framework
to write these kinds of tests.

3. Code coverage - Huge, IMO. How can you know that your unit tests or
integration tests (or even human-tester-driven acceptance tests) are
doing any good unless you know how much of the code is actually being
exercised? Code coverage is very simple to do, and for starters you
can't go wrong investigating Emma or Cobertura. These simply
instrument the Java bytecode, such that when the bytecode is executed
(by any mechanism) coverage counts by line/branch/method/class/package
are written to HTML or XML reports.

4. Carefully consider the issue of test data - test SQL scripts, mock
data, in-memory databases (see http://www.mikebosch.com/?p=8) etc.

5. Your tests are themselves defective. The sad truth is that if the
core source code you just wrote is riddled with defects, then so
probably are your tests.

Agreed! However careful and thorough you think you are, you're bound to
have made a few unreasonable assumptions, been unaware of various gotchas
in the Java API, etc. etc.

Main take-away here is, be aware that just
because all your unit tests pass, some not insignificant percentage of
those results are wrong.

Sad but true....

As a side note, this is where the higher-level layer of integration
tests also helps - it can assist in identifying flawed unit tests.

Agreed. Those other levels of testing - and the other people we involve
in our testing - make sure that everything anyone on the team can think
of gets covered.

Thanks for all the great suggestions, Arved!

I've been holding off on learning about the other testing tools until I
had a bit more time but maybe I really should get at least basic
familiarity with the key ones NOW. Then I can include those test results
in my "portfolio" and show employers/clients that I have a serious
professional attitude, even if my chops are not entirely as fluent as I
would like yet....

 

[Rhino]

Arved Sandstrom wrote:
...

Although it is not 100% effective, there is some value in writing, and
running, the test before implementing the feature. Start testing a
method when it is just an empty stub. At that point, a test that passes
is either useless or has a bug in it.

Patricia

Fair enough. I saw an article just the other day that used a buzzword I've
already forgotten which proposed that this was the right way to develop
code: write test cases BEFORE you even start the code and then retest
frequently as the code evolves. I can see some merit in that, even if I've
forgotten the buzzword already ;-)

 

[Tom Anderson]

Fair enough. I saw an article just the other day that used a buzzword
I've already forgotten which proposed that this was the right way to
develop code: write test cases BEFORE you even start the code and then
retest frequently as the code evolves. I can see some merit in that,
even if I've forgotten the buzzword already ;-)

Probably 'test-driven development', aka TDD.

And it's no more of a buzzword than 'wearing a seat-belt'. It's just a
bloody good idea!

tom

 

[Rhino]

Speaking as someone who occasionally sits in on job interviews, I'm

delighted when we find a candidate who's written any tests at all.
Most of them haven't. We don't expect people to know more than we do
(it would be great if they did) and we all learned as we went along
and wish we understood it better.

As I said to Arved elsewhere in the thread, I find it frightening that
anyone could apply for a job as anything more than a trainee programmer
(one who didn't know anything about programming at all and was expecting
to be trained by the employer) without having some familiarity with
testing and, hopefully being able to demonstrate the familiarity. If
that's the norm, then I feel better about my chances of getting a job
programming in Java already

In the specific case of a utility class, what I look for is clear
Javadoc. I need to be able to decide, without wasting a lot of time,
whether a method does what I need or not, and how it handles corner
cases. For an example of how I like to see it done, go to

http://commons.apache.org/lang/api-release/org/apache/commons/lang/Stri
ngUtils.html

and look at, say, the definition for

center(java.lang.String, int)

I come pretty close to that standard in my better Javadoc comments. Thank
you, this gives me a good standard to emulate in ALL of my Javadoc
comments!

Notice that once you have a good definition of how the method should
behave, writing a unit test for it is straightforward.

Yss and no. I don't have a lot of trouble with what to test in routine
methods but, as you can see from the first part of the thread, the proper
testing for some methods, like Constructors and getInstance() methods
isn't quite that obvious to me. But it's becoming a lot more clearer as a
result of this thread

Notice also that testing a utility class is very different from
testing a program or an application framework.

I hear you. I've got a variety of different projects - servlets, midlets,
GUI-centric programs, as well as more standard Java classes - and I've
got a running question in the back of my mind about how to test some of
them. But I'll leave questions of that kind for another day. I can't
learn everything I don't know in a single day or a single newsgroup
thread ;-)

 

[Rhino]

Rhino wrote:
if (Foo == null) fail("Constructor failed to instantiate the
class");

Eric Sosman wrote:
Rhino, if you keep on spewing this sort of codecrap I'm going
to shove that horn of yours firmly up the orifice that spews.

Rhino, the source of Eric's irritation is that despite apologizing
profusely multiple times for posting uncompilable code, you did it
again anyway.

Apologies don't help much if you don't correct the behavior.

For what it's worth, I never imagined that anyone was actually going to
compile that code. I just provided that (sort-of) code for illustrative
purposes. [...]

Nobody was probable to compile the code, not. But a fragment
in full load of the errors is not the "illustrative," it is only
offuscatorio. In order to extract the meaning from a beautiful
insignificant piece of the code, the reading of the person must
make the changes and corrections to arrive to something images that
you could have in mind and are every probability that will supply
some different thing from what has meant and that its ulterior
observations therefore will be not at all useful you.

(In other words: "Illustrate" your thoughts with garbled
utterances, and people may form opinions about your thinking.)

I get your point now and it is very reasonable. I'll try to do better in
future!

Thanks for your patience and sorry for the confusion.

 

[Tom Anderson]

I would test it.

Suppose at some future time a programmer changes the Map implementation
e.g. from TreeMap to HashMap. That might be done in an attempt to
improve performance, or because of a future decision to standardize on
HashMap unless there is a specific reason to use a different
implementation. The future programmer could be yourself after you have
had time to forget why you used TreeMap.

Perhaps the method should be defined to return a SortedMap. If the test
says:

SortedMap<String, Locale> locales = LocaleHelper.getLocales(); // can't remember the actual name, sorry

Any attempt to return a non-Sorted Map, including redefining the method to
return a plain Map will be stopped in its tracks by the compiler.

Of course, it doesn't stop someone returning a TreeMap with a custom
comparator that does something bananas. Java's type system has its limits.

tom

 

[Arne Vajhøj]

As I said to Arved elsewhere in the thread, I find it frightening that
anyone could apply for a job as anything more than a trainee programmer
(one who didn't know anything about programming at all and was expecting
to be trained by the employer) without having some familiarity with
testing and, hopefully being able to demonstrate the familiarity. If
that's the norm, then I feel better about my chances of getting a job
programming in Java already

You would be surprised how many total clueless (wannabee) programmers
there are.

Arne

 

[Arne Vajhøj]

Fair enough. I saw an article just the other day that used a buzzword I've
already forgotten which proposed that this was the right way to develop
code: write test cases BEFORE you even start the code and then retest
frequently as the code evolves. I can see some merit in that, even if I've
forgotten the buzzword already ;-)

TDD

http://www.agiledata.org/essays/tdd.html

Arne

 

[Rhino]

So, if the constructor doesn't

I'm not very familiar with "assert". I'll look into it on my own and
add that code.

Followup question. I just read up on assert and it is apparently not
supposed to be used in public methods. Since it is my getInstance() method
that invokes the private constructor, the getInstance() is where I'd put
the 'assert' right? After all, my constructor is completely empty.
If I follow the rule about not using 'assert' in public methods, I'm not
sure how to reconcile you advice with the rule in the assertions article
(http://java.sun.com/j2se/1.4.2/docs/guide/lang/assert.html)....

That would go in the test case (as opposed to the class) and wouldn't
be redundant with the 'assert' you mentioned that should go in the
class itself?

Is the assertNotNull in the test case a reasonable substitute for doing the
assert in the getInstance()?

 

[Lew]

Followup question. I just read up on assert and it is apparently not
supposed to be used in public methods. Since it is my getInstance() method
Wrong.

that invokes the private constructor, the getInstance() is where I'd put
the 'assert' right? After all, my constructor is completely empty.
If I follow the rule about not using 'assert' in public methods, I'm not

That's not a rule. You are not reading the advice carefully.

You're not supposed to use 'assert' to validate arguments to a public method.
There are plenty of use cases for 'assert' in a public method.

sure how to reconcile you advice with the rule in the assertions article
(http://java.sun.com/j2se/1.4.2/docs/guide/lang/assert.html)....

Read the article again, more carefully.

The purpose of 'assert' is to prove that invariants hold. It's actually not
so far different from the JUnit 'assertXxx()' methods, where 'assert' proves
within the code itself that algorithmic invariants hold, and JUnit's
assertions prove that test invariants hold in a test class outside the code
itself. Both throw 'Error's when violated.

Is the assertNotNull in the test case a reasonable substitute for doing the
assert in the getInstance()?

Not really.

They don't really do the same thing, exactly.

 

[Arne Vajhøj]

On 21-05-2010 08:51, Arved Sandstrom wrote:

[lots of stuff that I completely agree with]

3. Code coverage - Huge, IMO. How can you know that your unit tests or
integration tests (or even human-tester-driven acceptance tests) are
doing any good unless you know how much of the code is actually being
exercised? Code coverage is very simple to do, and for starters you
can't go wrong investigating Emma or Cobertura. These simply instrument
the Java bytecode, such that when the bytecode is executed (by any
mechanism) coverage counts by line/branch/method/class/package are
written to HTML or XML reports.

Note that unit tests to achieve 100% code coverage is
not sufficient - one need to test every path through
the code (CCN).

Arne

 

[Tom Anderson]

What is the best way to test a method whose output is unpredictable
because that output is a moving target? For example, I have a method
that uses Locale.getAvailableLocales() to display all of the Locales on
the current JVM. It works fine but how do I write a JUnit test, given
that an upgrade to the JVM could introduce additional Locales?

I'm a bit late to the party here, and i lost track of the answers, but i'd
think about writing a test where the expectation was driven by the set of
locales available in the JVM:

@Test
public void getLocalesReturnsAllLocales() {
Collection<Locale> locales = RhinoUtils.getLocales();
Locale[] referenceLocales = Locale.getAvailableLocales();
assertEquals(referenceLocales.length, locales.length);
for (Locale locale: referenceLocales) {
assertTrue(locales.contains(locale));
}
}

To be honest, though, if all your method is doing is wrapping
Locale.getAvailableLocales, i wouldn't write it in the first place. The
code you don't write is the easiest code to test!

Lastly - and thanks for bearing with me this far - is it normal practice
to write JUnit tests for all methods that are in parent classes of the
class being tested? For example, while writing tests for one of my
classes, the wizard in Eclipse asked if I wanted to produce test methods
to test methods of Object like equals() and wait(). Should I be writing
tests for those methods as well as for those in my own class?

There's no point at all writing tests for wait and notify, because those
are final methods defined in Object, so nothing you do can change whether
they work properly or not.

If you haven't overridden equals from Object, then for a similar reason,
there's no need to test it. If you have overridden it, then of course you
need to test it.

The one time i can think of where you might want to test a non-overridden
method is if it depends on a method you have overridden (or an abstract
method which you've implemented). For example:

public class CollectionOfThings<E> {
public abstract E[] getAllThings();
public E getFirstThingInOrder() {
SortedMap<String, E> sortedThings = new TreeMap<String, E>();
for (E thing: getAllThings()) sortedThings.put(thing.toString(), thing);
return sortedThings.values().iterator().next();
}
}

public class CollectionOfLocales extends CollectionOfThings<Locale> {
public Locale[] getAllThings() {
return Locale.getAvailableLocales();
}
}

In that case, when testing CollectionOfLocales, you might want to test
getFirstThingInOrder. Like:

@Test
firstLocaleInOrderIsRight() {
assertEquals(
new Locale("ar", "AE"),
new CollectionOfLocales().getFirstThingInOrder());
}


The point would really be that the contract of CollectionOfLocales with
its subclasses requires that the things they provide have toString methods
which return strings which preserve the ordering of the things. You need
to test that this term of the contract is fulfilled by each subclass.

Whilst this case may seem trivial, note that a CollectionOfAges subclass
which supplied Integers would not meet the requirement, because the
ordering of Integers' toStrings do not reflect their numerical order. Such
a class would be intrinsically broken, and this unit test would catch
that. Fixing it would probably involve changing the contract between
CollectionOfThings and its subclasses, and the implementation of
getFirstThingInOrder.

Oh, also note that the code i wrote above for Locales is wrong. I only
discovered this when i actually compiled and ran it (or something a bit
like it, at least - not the exact code above). This illustrates the
importance of tests!

tom

 

[Tom Anderson]

But the same situation applies to things that aren't based on
convenience methods. A list of telephone area codes or the names of all
chiropractors in Ohio or any of a thousand other results from methods
are also "moving targets". I'm really not sure how to test those.

These methods get their answers from some source of data, right?

So the way you test them is with test data. Rather than plugging the code
into the real data, you make a controlled set of test data, plug them into
that, and test that they behave in the way you would expect given the test
data. You then assume that they'll behave the right way when plugged into
real data.

Naturally, this means structuring your code so that it can easily be
plugged into different sources of data. This is probably a good thing
anyway.

tom

 

[Lew]

@Test
public void getLocalesReturnsAllLocales() {
Collection<Locale> locales = RhinoUtils.getLocales();
Locale[] referenceLocales = Locale.getAvailableLocales();
assertEquals(referenceLocales.length, locales.length);

locales.size()

 

[Tom Anderson]

@Test
public void getLocalesReturnsAllLocales() {
Collection<Locale> locales = RhinoUtils.getLocales();
Locale[] referenceLocales = Locale.getAvailableLocales();
assertEquals(referenceLocales.length, locales.length);

locales.size()

Apologies. I knew i was running a risk by posting uncompiled code in this
particular thread, but i really thought i'd got away with it this time.

tom

 

[Arved Sandstrom]

Out of interest, do those play well with other things which modify
bytecode, like JDO/JPA enhancers?

tom

That's a very good question. I've used both Emma and Cobertura with
EclipseLink, so you'd have EclipseLink dynamic weaving (change tracking,
lazy fetching) after the code coverage instrumentation; only in the
entity classes though.

Only my gut feeling, but if I were using anything that did more
elaborate bytecode manipulation than that then I figure I'd want to make
sure that the code coverage instrumentation was last.

AHS

 

[Arved Sandstrom]

[ SNIP ]

I see that you've given some good concrete suggestions for tools to use
in upcoming paragraphs. Are any of these static code analysis tools? If
not, can you suggest some good ones? I'd love to have some feedback on my
code via code reviews too but that's a little difficult to do. It's not
realistic to full (non-trivial) classes in the newsgroup so I just pick
and choose especially concerning problems and post snippets reflecting
them. I'm sure that you folks are all too busy to do detailed
walkthroughs of dozens of classes with hundreds of lines of codes in many
of them ;-) I must see if there is a local Java user group in this area
and see if I can develop a few buddies for doing code walkthroughs....

[ SNIP ]

FindBugs, if you're not using it already, is a good static code
analyzer. In my experience, just one decent tool like that, plus the
requisite excellent books like Bloch's "Effective Java" and willingness
to spend time reading those books to understand what the analyzer tells
you, is all you really need.

It's also useful to complement FindBugs with Checkstyle, which latter is
a good part of your toolkit for other purposes as well.

AHS

 

[Arved Sandstrom]

This principle is also very important for bug fixing. Any bug represents
a defect in the test suite. Fix the test suite first, and verify that
the new tests fail, before any attempt to fix the underlying bug.

Patricia

This discussion brings to mind something I'm faced with at work, for one
client. Although several of the applications I work on have sizeable
numbers of unit tests, they have been allowed to obsolesce, and there
have always been big gaps in coverage. We have a push on to add tests
and fix the ones that are already there. We are also adding functional
tests.

Now, here's the rub - in many cases it's known that the underlying code
is defective - human testers have recorded defects for specific use
cases. So when fixing or adding tests as part of this work, we have to
write no small number of tests in such a way that they will fail,
because if they don't fail then the tests are wrong.

AHS

 

[Arved Sandstrom]

As I said to Arved elsewhere in the thread, I find it frightening that
anyone could apply for a job as anything more than a trainee programmer
(one who didn't know anything about programming at all and was expecting
to be trained by the employer) without having some familiarity with
testing and, hopefully being able to demonstrate the familiarity. If
that's the norm, then I feel better about my chances of getting a job
programming in Java already

[ SNIP ]

One good thing about TDD, which has been mentioned elsewhere in this
thread (and in other threads), is that you can slide tests in as part of
the implementation, with no non-technical managers any the wiser. The
major problem with writing tests after the fact, even in reasonably
enlightened shops, is that at least three-quarters of all software
projects don't hit deadlines, for reasons that have very little to do
with the coding. In these circumstances anything that was planned for
the last phases, and appears to be non-critical, gets dispensed with.
Whereas if you start with tests, and refine them as your code gets
written, you barely impact the project schedule in the real world, but
end up with much better code, a complete set of tests, and the ability
to better react to change.

I don't have much sympathy for programmers who have never written tests,
but I do understand programmers who have no *work experience* writing
tests. The latter is quite common, because a majority of software teams
are disinterested in writing tests, or are not permitted to "waste time"
writing tests, or everyone is agreeable but they commit the mistake I
describe above. The former situation tells me that the programmer in
question doesn't do much professional reading, and can't be bothered to
spend some of their own time trying out a test framework...which would
take, like, a single evening.

AHS

 

[Tom Anderson]

This discussion brings to mind something I'm faced with at work, for one
client. Although several of the applications I work on have sizeable numbers
of unit tests, they have been allowed to obsolesce, and there have always
been big gaps in coverage. We have a push on to add tests and fix the ones
that are already there. We are also adding functional tests.

Now, here's the rub - in many cases it's known that the underlying code is
defective - human testers have recorded defects for specific use cases. So
when fixing or adding tests as part of this work, we have to write no small
number of tests in such a way that they will fail, because if they don't fail
then the tests are wrong.

Well, at least that makes it easy to know you've got the test right!

Norbert Seriously, folks, this is an old point:

http://c2.com/cgi-bin/wiki?CaptureBugsWithTests

tom