[ANN] celsoft.com/Battery 0.1.1

[Sean O'Dell]

Battery is a unit testing framework for Ruby. It captures all standard error
and output and reports the entire summary of all tests formatted as valid
YAML, for easier reading and parsing. Another key feature is that all tests
run in the order they are added to their batteries, rather than arbitrarily.

See the celsoft.com/Battery homepage for more information and documentation.

Homepage: http://battery.rubyforge.org/
Download: http://rubyforge.org/frs/?group_id=268&release_id=531

Sean O'Dell

 

[David A. Black]

Hi --

Battery is a unit testing framework for Ruby. It captures all standard error
and output and reports the entire summary of all tests formatted as valid
YAML, for easier reading and parsing. Another key feature is that all tests
run in the order they are added to their batteries, rather than arbitrarily.

You're implying, I believe, that Test::Unit runs the test methods in
arbitrary order. It doesn't; it runs them in alphabetical order.


David

 

[Jim Menard]

Hi --



You're implying, I believe, that Test::Unit runs the test methods in
arbitrary order. It doesn't; it runs them in alphabetical order.

Is that guaranteed by the API or should I not rely on it?

Jim

 

[Nathaniel Talbott]

Is that guaranteed by the API or should I not rely on it?

It is guaranteed. Even if other orderings are added at some point in
the future, running in alphabetical order will always be supported.


Nathaniel
Terralien, Inc.

<(><

 

[Michael Campbell]

You're implying, I believe, that Test::Unit runs the test methods in

Is that guaranteed by the API or should I not rely on it?

Beside your point a bit here, but you shouldn't rely on tests running
in ANY order.

 

[Sean O'Dell]

Hi --



You're implying, I believe, that Test::Unit runs the test methods in
arbitrary order. It doesn't; it runs them in alphabetical order.

When tests are run alphabetically, it makes it tough to run the same tests in
specific orders. I wanted to be able to take a single test, and run it
combined with other tests in varying orders. celsoft.com/Battery lets me
create batteries in which I add the same test multiple times but in the order
I choose.

Sean O'Dell

 

[Jim Weirich]

Nathaniel Talbott said:

It is guaranteed. Even if other orderings are added at some point in
the future, running in alphabetical order will always be supported.

Even if it is guaranteed, I recommend not relying on it in the tests
themselves. Each test should be very loosely coupled to its siblings and
the order should not matter.

 

[Sean O'Dell]

Nathaniel Talbott said:

Even if it is guaranteed, I recommend not relying on it in the tests
themselves. Each test should be very loosely coupled to its siblings and
the order should not matter.

The order shouldn't matter, but it's better to have control over the order
rather than have them run in an arbitrary order selected by the library.
With control, you could randomize the list of tests and run the randomized
battery n times. When the tests are alphabetical, they're always run in
alphabetical order.

Sean O'Dell

 

[Michael Campbell]

The order shouldn't matter, but it's better to have control over the order
rather than have them run in an arbitrary order selected by the library.
With control, you could randomize the list of tests and run the randomized
battery n times. When the tests are alphabetical, they're always run in
alphabetical order.

How is alphabetical order any better/worse than "the order they are
added to their batteries" in this regard? Or does your framework
allow order randomization?

 

[Sean O'Dell]

How is alphabetical order any better/worse than "the order they are
added to their batteries" in this regard? Or does your framework
allow order randomization?

Tests are stored as an array of blocks and are called in the order they appear
in the array. I just now added an accessor method to the @tests instance
variable so you can arrange the order anyway you want..

But even before that change, you could create a number of batteries and call
battery#add_test in any order you prefer to create your own run order for
each battery, then call battery#run for each, using a common report object,
and get one big final report for it all.

Sean O'Dell

 

[David A. Black]

Hi --

When tests are run alphabetically, it makes it tough to run the same tests in
specific orders. I wanted to be able to take a single test, and run it
combined with other tests in varying orders. celsoft.com/Battery lets me
create batteries in which I add the same test multiple times but in the order
I choose.

I wasn't talking about the merits of one design or the other, just
wanted to clarify the point that in Nathaniel's design the tests are
not run arbitrarily.

David

 

[Sean O'Dell]

I wasn't talking about the merits of one design or the other, just
wanted to clarify the point that in Nathaniel's design the tests are
not run arbitrarily.

Arbitrary meaning that, from a library USER'S viewpoint, the decision to order
tests alphabetically is without reason and seems hastily settled upon.
Perhaps the library developer had a good reason, but as a user of the library
with different needs, I don't see/concur with the reason.

Anyway, if you like it alphabetical, you got one library. If you like to
order tests yourself, now you have another.

Sean O'Dell

 

[Charles Comstock]

Nathaniel Talbott said:

Even if it is guaranteed, I recommend not relying on it in the tests
themselves. Each test should be very loosely coupled to its siblings and
the order should not matter.

Yes but I can think of a very simple reason someone might want to reorder there
tests a bit. Right now there is a bug in ruby 1.8.2 that causes one of the unit
tests to segfault the interpreter when running an extension test. As a result
the test suite never completes, so I only know what works before it. If this
happened in a unit test that was of my own product there might exist a case in
which I wanted to double check all my others tests before the complete failure
case. Particularly if I was writing more specific unit tests to try to figure
out what was causing the segfault.

I might also just want the reports on related components to be in the same area
when they fail if I change something.

Clearly you shouldn't set up tests to depend on ordering but for your own
understanding of the output I can think of lots of reasons you might want your
own ordering.

Charlie

 

[Gennady Bystritsky]

Yes but I can think of a very simple reason someone might want to
reorder there
tests a bit. Right now there is a bug in ruby 1.8.2 that causes one
of the unit
tests to segfault the interpreter when running an extension test. As
a result
the test suite never completes, so I only know what works before it.
If this
happened in a unit test that was of my own product there might exist a
case in
which I wanted to double check all my others tests before the complete
failure
case. Particularly if I was writing more specific unit tests to try
to figure
out what was causing the segfault.

In your own tests you can always put
flunk("To be fixed later")

or something like that in the beginning of the segfaulting test or
rename the test so that it does not run at all (I usually put
underscore in front of the test name). This does not justify
introducing the test order at all. There might be other reasons,
though. I simply haven't heard of any good one yet.

By the way, with Test::Unit you have ability to specify the exact test
method or a test class on the command line, even using regexps. There's
a bug in test specification though, however it is easily fixed. I did
not submit usual report, sorry. With this feature, if I have a huge
test suite, I can concentrate on a particular problem I am working on
right now, and when it finally passes run the whole suite, happily
leaning at the back of my chair.

I might also just want the reports on related components to be in the
same area
when they fail if I change something.

Clearly you shouldn't set up tests to depend on ordering but for your
own
understanding of the output I can think of lots of reasons you might
want your
own ordering.

Charlie

Sincerely,
Gennady Bystritsky

 

[David A. Black]

Arbitrary meaning that, from a library USER'S viewpoint, the decision to order
tests alphabetically is without reason and seems hastily settled upon.
Perhaps the library developer had a good reason, but as a user of the library
with different needs, I don't see/concur with the reason.

Have you checked the archives, or asked Nathaniel? That might help
you see the reasoning behind the work (including input from the
community, Nathaniel's attention to which is utterly exemplary), and
then decide in a more informed manner whether or not you concur with
it.

Also, now that you've been made aware of the fact that Test::Unit runs
the tests in alphabetical order, it would be much more collegial not
to cling to the use of the word "arbitrary" and its derivatives in an
effort to make retroactive sense of your original, incorrect claim.
I've watched Nathaniel work on Test::Unit over the years, and heard
him talk about it online and in person, and I can assure you that
there is nothing arbitrary about his design whatsoever. Anyone
writing a testing framework in Ruby, whether they make the same
decisions he has or not, should be grateful or at least gracious.


David

 

[Sean O'Dell]

Have you checked the archives, or asked Nathaniel? That might help
you see the reasoning behind the work (including input from the
community, Nathaniel's attention to which is utterly exemplary), and
then decide in a more informed manner whether or not you concur with
it.

Also, now that you've been made aware of the fact that Test::Unit runs
the tests in alphabetical order, it would be much more collegial not
to cling to the use of the word "arbitrary" and its derivatives in an
effort to make retroactive sense of your original, incorrect claim.
I've watched Nathaniel work on Test::Unit over the years, and heard
him talk about it online and in person, and I can assure you that
there is nothing arbitrary about his design whatsoever. Anyone
writing a testing framework in Ruby, whether they make the same
decisions he has or not, should be grateful or at least gracious.

ar·bi·trar·y
adj.
Determined by chance, whim, or impulse, and not by necessity, reason, or
principle.

There is no fundamental reason tests should be ordered alphabetically. I've
demonstrated that it's only too easy to order tests any way I want, so I know
alphabetic is not out of necessity. So then, what reason or principle
determines that tests should be run alphabetically? I see none. Is there a
reason and/or principal that says they must be run with that order? Please
tell me if there is, and I'll retract.

Sean O'Dell

 

[David A. Black]

arbitrary:

Determined by chance, whim, or impulse, and not by necessity, reason, or
principle.

There is no fundamental reason tests should be ordered alphabetically.

Maybe, maybe not -- but in either case, I was addressing your original
claim, which was not that choosing alphabetical order over other designs
was arbitrary, but rather that in Nathaniel's design, the tests were
run in arbitrary (i.e., determined by chance, whim, etc.) order. They
aren't, so I wanted to let you and other readers know.

This had (and has) nothing to do with the processing of choosing
alphabetical order (which you introduced into the discussion later),
but rather with the fact that alphabetical order is not, itself,
arbitrary -- and, concommitantly, with the fact that Nathaniel didn't
deserve to have his design misrepresented as involving something it
doesn't.


David

 

[Sean O'Dell]

Maybe, maybe not -- but in either case, I was addressing your original
claim, which was not that choosing alphabetical order over other designs
was arbitrary, but rather that in Nathaniel's design, the tests were
run in arbitrary (i.e., determined by chance, whim, etc.) order. They
aren't, so I wanted to let you and other readers know.

This had (and has) nothing to do with the processing of choosing
alphabetical order (which you introduced into the discussion later),
but rather with the fact that alphabetical order is not, itself,
arbitrary -- and, concommitantly, with the fact that Nathaniel didn't
deserve to have his design misrepresented as involving something it
doesn't.

When you're given no choice, as far as you're concerned as a library user,
they're run in an arbitrary order. The fact that it turns out they are run
alphabetically doesn't change the fact that the system of ordering is
arbitrary. There is no rhyme, reason or principal determining the order; it
was decided by the author at his prerogative. If the tests were run
according to the length of the method names, that would still be an arbitrary
order. As far as I'm concerned, the order that tests are run is a black box.

Sean O'Dell

 

[James Britt]

There is no rhyme, reason or principal determining the order; it
was decided by the author at his prerogative.

I think David's point was that the order was decided after much
community discussion and end-user participation, not on some developer's
whim. Perhaps you disagree, or are unfamiliar, with the reasons for
selecting this order, but that's not to say none exist.

If, after you've looked into the history of the code, you still find the
order arbitrary, then there's probably not much more anyone can say,
except perhaps that most users don't find the order arbitrary at all.

But, of course, maybe they are all wrong and just haven't been
enlightened.

James

 

[Jean-Hugues ROBERT]

When you're given no choice, as far as you're concerned as a library user,
they're run in an arbitrary order. The fact that it turns out they are run
alphabetically doesn't change the fact that the system of ordering is
arbitrary. There is no rhyme, reason or principal determining the order; it
was decided by the author at his prerogative. If the tests were run
according to the length of the method names, that would still be an arbitrary
order. As far as I'm concerned, the order that tests are run is a black box.

Sean O'Dell

Hi Sean,

I suspect that there is some misunderstanding here. That you would
call the order "arbitrary" conveys some negative connotation that
you certainly want to avoid not to appear rude or condescending.
In France a typical "arbitrary" decision was King's one to put you
on jail (lettre de cachet). As a result "arbitrary" tends to
mean "lack of respect".

I also suspect that the decision to "order" tests is not at all
arbitrary compared to the decision not to sort tests. I tend to
think that it is a good thing to order things and alphabetic order
is a reasonable choice that is easy to understand by everybody.

Once things are ordered, you can take advantage of that and for
example name your test AAA until they first work. That way you
get early results. Once your test passes, you then rename it in a
more understandable way. That's just a trick, but it helps, and
would not be possible if there was no documented order.

For sure, it is always a good thing to give rationals and if
the "ordering" was not documented properly then that could be
improved. Something that seems arbitrary at first look may
turn out to be solid once understood. This requires both some
help from the author and some good will of the reader.

Yours,

JeanHuguesRobert