Reflection in JUnit tests

[Christopher Benson-Manica]

I am planning to write a utility class that will use reflection to
automatically exercise the getters and setters for a given object - it
will call all the set*() functions for the object that take primitives
or Strings (passing default values), keep track of what was passed to
each method, and then call get*() and verify that the return value is
as expected. The class will also be able to take two objects of the
same class and invoke all the get*() functions on each, and verify
that the values returned are identical.

My feeling is that I'm surely not the first person to want to do this,
so my question is has this been done before, and if so, where can I
obtain the library? I'm more than up to the task of writing it
myself, but I always prefer to build on existing work where possible.
Thanks!

 

[Eric Sosman]

I am planning to write a utility class that will use reflection to
automatically exercise the getters and setters for a given object - it
will call all the set*() functions for the object that take primitives
or Strings (passing default values), keep track of what was passed to
each method, and then call get*() and verify that the return value is
as expected. The class will also be able to take two objects of the
same class and invoke all the get*() functions on each, and verify
that the values returned are identical.

My feeling is that I'm surely not the first person to want to do this,
so my question is has this been done before, and if so, where can I
obtain the library? I'm more than up to the task of writing it
myself, but I always prefer to build on existing work where possible.
Thanks!

I'm afraid I can offer no recommendation for a tool that
makes the task any easier. However, I see difficulties with
such "blind" testing: the getters and setters may well enforce
conditions an oblivious test harness wouldn't know about.

public class Act {
private int number;
public int getNumber() {
return number;
}
public int setNumber(int number) {
if ((number & 1) != 0)
throw new IllegalArgumentException(
"no oddities allowed!");
this.number = number;
}
}

There could also be interactions between getters and
setters: setMinimumX() might affect the value returned by
getMaximumX(), for example, to maintain a constant delta-X.

 

[Christopher Benson-Manica]

I'm afraid I can offer no recommendation for a tool that
makes the task any easier. However, I see difficulties with
such "blind" testing: the getters and setters may well enforce
conditions an oblivious test harness wouldn't know about.

Right, those caveats obviously exist, and I think a goal of creating
something completely universal would be unattainable. But there are
many cases where objects have simple getters and setters (the class
that motivated this has 24 of each) that could be very conveniently
handled by the "blind" testing I have in mind.

 

[Chris Smith]

Right, those caveats obviously exist, and I think a goal of creating
something completely universal would be unattainable. But there are
many cases where objects have simple getters and setters (the class
that motivated this has 24 of each) that could be very conveniently
handled by the "blind" testing I have in mind.

Presumably you generated these things by some IDE feature, and they are
about the least likely possible place to have bugs in your software,
unless the bugs are in enforcing constraints on the values, which you've
excluded from your work. What would you test; that the functions don't
arbitrarily fail? What values would you try to set? How would you know
they are valid? If your code will involve some kind of specification
language for issues like that, then it might be useful. Otherwise, it
appears you are trying to obtain test coverage for free; and you'll
achieve the test coverage, but you won't derive any actual benefit from
it. I'm afraid you may have been bitten by the unit testing bug to the
point that you don't remember why you're doing it any longer.

This is beside the question of why you have so many accessor and mutator
methods in the first place. There are valid reasons, though...

 

[Christopher Benson-Manica]

Presumably you generated these things by some IDE feature, and they are
about the least likely possible place to have bugs in your software,
unless the bugs are in enforcing constraints on the values, which you've
excluded from your work.

Yes, that's true, and you may be right about having been bitten by the
unit testing bug. OTOH, if it's easy to get this kind of test
coverage (and it is), is there any argument against just doing it?

 

[Eric Sosman]

Yes, that's true, and you may be right about having been bitten by the
unit testing bug. OTOH, if it's easy to get this kind of test
coverage (and it is), is there any argument against just doing it?

None I can think of -- computer time is cheap. If the
methods are hand-coded rather than automatically generated,
it might even turn up a bug or two (e.g., incomplete edits of
cut-'n-paste boilerplate). But as Chris points out, methods
generated from a specification aren't likely to have any such
problem. To put it whimsically: How do you write a JUnit test
to check that the `return' statement operates correctly?

Maybe you should be looking not for something that generates
tests from code, but for something that generates both the code
and the tests from specifications. Such a thing (if it exists;
I apologize again for being such a thrower of cold water) would
seem to have more "value add" than oblivious tests generated only
from the method signatures.

 

[AndrewMcDonagh]

I am planning to write a utility class that will use reflection to
automatically exercise the getters and setters for a given object - it
will call all the set*() functions for the object that take primitives
or Strings (passing default values), keep track of what was passed to
each method, and then call get*() and verify that the return value is
as expected. The class will also be able to take two objects of the
same class and invoke all the get*() functions on each, and verify
that the values returned are identical.

My feeling is that I'm surely not the first person to want to do this,
so my question is has this been done before, and if so, where can I
obtain the library? I'm more than up to the task of writing it
myself, but I always prefer to build on existing work where possible.
Thanks!

As someone who's been using JUnit (and other xUnit frameworks) for over
4 years now, I can offer some advice....YMMV...


Its very common to only test getters & setters that do something other
than the simple variety.

Even then, most of the time, these methods are covered by tests that are
covering other (more fundamental) methods of the class under test.

In over 4 years of JUnit usage, I've only seen one maybe two occurrences
where a test would have helped speed up spotting a problem with the
code. Even in these cases, the other tests detected the problem, they
just didn't have the granularity to allow for quick identification of
the root problem of the test failure. We had to debug the test to see
the issue was with the setter/getter.

Debugging is something I rarely need to do, as I write unit tests for
all code I write - this allows me to know it works or not without going
anywhere near the bugger or running the program.

Indeed, I use it so infrequently, I cant remember the keyboard short
cuts for stepping into/over/outof ! LOL

 

[Chris Smith]

Yes, that's true, and you may be right about having been bitten by the
unit testing bug. OTOH, if it's easy to get this kind of test
coverage (and it is), is there any argument against just doing it?

I think there's a pretty good argument against just doing it. That
argument is that you will be potentially falsifying and/or sabotaging
metrics like unit test code coverage, which may be run against your code
now or in the future. It will become misleading in the future when
someone tricks themselves into thinking that there's some benefit to
that code. If nothing else, the time you spend implementing this could
have been spent writing useful tests.

Furthermore, I don't know if you could make it work in a general way.
If you write this code at the expense of needing to allow null or zero
values to be passed into all your setter methods even where it doesn't
make sense to do so, then that's a definite harm.

 

[Christopher Benson-Manica]

I think there's a pretty good argument against just doing it. That
argument is that you will be potentially falsifying and/or sabotaging
metrics like unit test code coverage, which may be run against your code
now or in the future.

I admit that I'm still very new to the unit test paradigm, but it
seems that perhaps using unit test code coverage as an end-all, be-all
metric is a flawed plan for this very reason...

If nothing else, the time you spend implementing this could
have been spent writing useful tests.

Well, so far it has been, as I haven't attempted to implement it yet,
in favor of working on those useful tests

Furthermore, I don't know if you could make it work in a general way.
If you write this code at the expense of needing to allow null or zero
values to be passed into all your setter methods even where it doesn't
make sense to do so, then that's a definite harm.

If it doesn't make sense to pass 0 or null, IMHO a good setter will
throw IllegalArgumentException, although of course that doesn't change
your point - for the record, the implementation I may yet write would
not pass null or 0.

 

[Christopher Benson-Manica]

problem. To put it whimsically: How do you write a JUnit test
to check that the `return' statement operates correctly?

I suppose the point is taken

Maybe you should be looking not for something that generates
tests from code, but for something that generates both the code
and the tests from specifications. Such a thing (if it exists;
I apologize again for being such a thrower of cold water) would
seem to have more "value add" than oblivious tests generated only
from the method signatures.

I'm using an IDE which seems to have a strong reputation, so perhaps I
will look into trying to locate such a tool.

 

[Christopher Benson-Manica]

Its very common to only test getters & setters that do something other
than the simple variety.

Even then, most of the time, these methods are covered by tests that are
covering other (more fundamental) methods of the class under test.

Granted, although I have to admit that I am a horrible pedant living
in a pragmatist's body, so the temptation to cover *everything* is
non-trivial

Debugging is something I rarely need to do, as I write unit tests for
all code I write - this allows me to know it works or not without going
anywhere near the bugger or running the program.

Having spent the first 2.5 years of my career debugging (often in
horribly time-consuming ways), I'm looking forward to enjoying the
relative bliss afforded by unit tests.

 

[Chris Smith]

I admit that I'm still very new to the unit test paradigm, but it
seems that perhaps using unit test code coverage as an end-all, be-all
metric is a flawed plan for this very reason...

Any metric becomes useless the instant that shallow goals of improving
the metric are adopted. That's not unique to unit test coverage. It's
a foundational law of process metrics in general. It's easier to game
the system than to do good work.

My comment arises from the impression that you're trying to write this
code so as to gain additional unit test code coverage. If instead you
have specific tangible goals in mind for your project, then you are best
qualified to decide whether the technique fulfills those goals. At the
very least, the rest of us could give better advice if we knew what they
were. But if your goals are "gotta unit test everything I can", then my
comments apply.

If it doesn't make sense to pass 0 or null, IMHO a good setter will
throw IllegalArgumentException, although of course that doesn't change
your point - for the record, the implementation I may yet write would
not pass null or 0.

Absolutely. If you pass sensible data (or pass non-sensible data and
verify that the code fails as documented), then this could be useful.
In my previous response, I pointed out that if you have some kind of
specification for how the code should behave and you intend to generate
your unit tests from that specification, then this could be a *very*
good idea. It's only in the absence of that specification that I think
you're doing the wrong thing.

 

[Christopher Benson-Manica]

Absolutely. If you pass sensible data (or pass non-sensible data and
verify that the code fails as documented), then this could be useful.
In my previous response, I pointed out that if you have some kind of
specification for how the code should behave and you intend to generate
your unit tests from that specification, then this could be a *very*
good idea. It's only in the absence of that specification that I think
you're doing the wrong thing.

Since I don't have such a specification at the moment, I suppose I
will leave off this idea of mine for now; perhaps what is needed is to
formulate such a specification to make my bad idea good Anyway, I
appreciate your comments, and Mr. Sosman's, and will bear them in
mind, if not unto good fortune.