Test::Unit: assert_follows_spec() (or something like that)

[David Garamond]

For deterministic functions/methods, the same set of inputs should
always produces the same output. What about adding something like:

assert_follows_spec(method_or_class_name, spec)

Where spec is an array of input and output pairs (and optionally an
exception, if the input should not be accepted):

[{input => [1, 1, 1], output => 1},
{input => [8, 1, 2], output => 4},
{input => [8, 3, 4], output => 6},
{input => [4, 3, 2], output => 6},
{input => [1, 2, nil], exception => ArgumentError},
{input => [1, 2, 0], exception => ZeroDivisionError}]

This can make unit tests shorter and more readable. We can transform the
spec into an HTML table, include it into the API documentation, etc.

The assert_follows_spec() could even perform several iterations of tests
if wanted, to test deterministic-ness.

 

[Simon Strandgaard]

For deterministic functions/methods, the same set of inputs should
always produces the same output. What about adding something like:

assert_follows_spec(method_or_class_name, spec)

Where spec is an array of input and output pairs (and optionally an
exception, if the input should not be accepted):

[{input => [1, 1, 1], output => 1},
{input => [8, 1, 2], output => 4},
{input => [8, 3, 4], output => 6},
{input => [4, 3, 2], output => 6},
{input => [1, 2, nil], exception => ArgumentError},
{input => [1, 2, 0], exception => ZeroDivisionError}]

This can make unit tests shorter and more readable. We can transform the
spec into an HTML table, include it into the API documentation, etc.

The assert_follows_spec() could even perform several iterations of tests
if wanted, to test deterministic-ness.

Good idea.. for ages I have been using

io = [
['inputx', 'outputx'],
['bla', 'outputbla'],
['ruby', 'outputruby'],
]
input, expected = io.transpose
actual = input.map{|i| stuff(i) }
assert_equal(expected, actual)

If there are too many input/output entries, then its difficult to
quickly identify where the problem are located.

A more well-crafted assertion method, which are able to output
more precisely where the problems are located, would be greak.

 

[Jean-Hugues ROBERT]

For deterministic functions/methods, the same set of inputs should always produces the same output. What about adding something like:

assert_follows_spec(method_or_class_name, spec)

Where spec is an array of input and output pairs (and optionally an exception, if the input should not be accepted):

[{input => [1, 1, 1], output => 1},
{input => [8, 1, 2], output => 4},
{input => [8, 3, 4], output => 6},
{input => [4, 3, 2], output => 6},
{input => [1, 2, nil], exception => ArgumentError},
{input => [1, 2, 0], exception => ZeroDivisionError}]

This can make unit tests shorter and more readable. We can transform the spec into an HTML table, include it into the API documentation, etc.

The assert_follows_spec() could even perform several iterations of tests if wanted, to test deterministic-ness.

Hi Dave,

I just started to use Test::Unit (only to discover that it provides a framework for what I have done in a ad hoc way for years). Nice.

I assume that using assert_block() it is rather easy to implement assert_follows_spec().

However, the name "follows_spec" may be slightly misleading, because the true "spec" is more like the full suite of tests than the subset that is provided to assert_follows_spec(). Maybe "assert_calls" is a more appropriate name,
because what is done is that: calls with parameters and expected results.

The idea of running the calls multiple times is for sure useful sometimes.

About the generation of HTML documentation, I feel, again, that the full spec is not easy to grab. I would
rather like to see some RDoc tool that would consider test cases as "examples". There would then be a need
for some cross-ref where you can jump to test cases that involve some method you're looking at.

This way one would leverage the value of test cases: they would become more useful as documentation items.

I do agree that there is room for improvement regarding the size of tests. assert_calls() is one improvement.

Another one I can think of is a way to switch to the next test without creating a method for it. Today I tend to tests multiple things in a single test method, more like a sequence of method calls than individual methods. Coining a name for the test method is not easy (because it tests so many things). I end up calling them test_1, test_2 and so on, that is not very useful.

I would like something like "testing( msg)" where msg would complement the context that would be displayed if some assert bombs (in addition to the name of the test method as is done today). In the end the name of the test method should be more like the name of the "use case" I am testing. testing(msg) would describe the intermediary steps of
the "use case".

I'll dig further in Test::Unit source code if/when time permit.

Yours,

JeanHuguesRobert

 

[David Garamond]

For deterministic functions/methods, the same set of inputs should
always produces the same output. What about adding something like:

assert_follows_spec(method_or_class_name, spec)

Where spec is an array of input and output pairs (and optionally an
exception, if the input should not be accepted):

[{input => [1, 1, 1], output => 1},
{input => [8, 1, 2], output => 4},
{input => [8, 3, 4], output => 6},
{input => [4, 3, 2], output => 6},
{input => [1, 2, nil], exception => ArgumentError},
{input => [1, 2, 0], exception => ZeroDivisionError}]

Of course the above should be written as :input, utput, etc. Too much
writing YAML, I guess .

Some more ideas:

* max_delta for floats:

{:input => [1, 3.0], utput => 0.333333333333333,
:max_delta => 0.000000000000001}

* support for non-deterministic func/methods, like 'output_in' (multiple
possible outputs), 'output_class' and 'output_range' (for any acceptable
output or a certain clas and inside a certain range). e.g. for testing
Kernel#rand:

[{:input => [1], utput => 0},
{:input => [3], utput_in => [0, 1, 2]},
{:input => [], utput_class => Float,
utput_range => 0...1.0}]

Basically, I'm seeing assert_follows_spec (or assert_calls, or whatever)
as simply an aggregate of lots of simpler asserts (assert_equals,
assert_raises, etc) as one big assert. The difference is, it's shorter
(it saves some keystrokes), nicer/tidier, and the "spec" itself is a
data structure and can be changed/manipulated/displayed/etc more easily.

Of course, this "spec" in many cases cannot cover all aspects of the
testing, only simpler ones. So perhaps a better name for "spec" is desired.

 

[Florian Gross]

Hi
Moin.


About the generation of HTML documentation, I feel, again, that the
full spec is not easy to grab. I would rather like to see some RDoc
tool that would consider test cases as "examples".

I'm currently embedding test cases into the documentation. They look a
lot like example code. I think this is commonly called "Specification by
example". I've written a tool for automatically extracting and running
them. Here is an example:

# Matches the space between words.
# This is a zero-width anchor.
#
# re = Regexp::English.word_boundary
# text = "Hello World!"
# text.gsub(re, "*") # => "*Hello* *World*!"
# # Matches the word "and", but not "band"
# re2 = Regexp::English.new do
# literal("and").surrounded_by(word_boundary)
# end
# "band".match(re2) # => nil
# "and".match(re2)[0] # => "and"
def word_boundary; Node::Anchor.new(/\b/); end

If this test case where to fail you would get an useful error like this:

1) Failure:
test_word_boundary()
[(eval):3:in `eval'
(eval):7:in `test_word_boundary'
(eval):3:in `test_word_boundary']:
c:/ruby/lib/ruby/1.8/test/unit/assertions.rb:32:in `assert_block'<"\"*Hello* *World*!\""> expected but was <"\"Hello World!\"">.

Yours,
JeanHuguesRobert

Regards,
Florian Gross

 

[Simon Strandgaard]

For deterministic functions/methods, the same set of inputs should
always produces the same output. What about adding something like:

assert_follows_spec(method_or_class_name, spec)

Where spec is an array of input and output pairs (and optionally an
exception, if the input should not be accepted):

[{input => [1, 1, 1], output => 1},
{input => [8, 1, 2], output => 4},
{input => [8, 3, 4], output => 6},
{input => [4, 3, 2], output => 6},
{input => [1, 2, nil], exception => ArgumentError},
{input => [1, 2, 0], exception => ZeroDivisionError}]

Of course the above should be written as :input, utput, etc. Too much
writing YAML, I guess .

Some more ideas:

* max_delta for floats:

{:input => [1, 3.0], utput => 0.333333333333333,
:max_delta => 0.000000000000001}

* support for non-deterministic func/methods, like 'output_in' (multiple
possible outputs), 'output_class' and 'output_range' (for any acceptable
output or a certain clas and inside a certain range). e.g. for testing
Kernel#rand:

[{:input => [1], utput => 0},
{:input => [3], utput_in => [0, 1, 2]},
{:input => [], utput_class => Float,
utput_range => 0...1.0}]

What about :in ut :fail :error

Basically, I'm seeing assert_follows_spec (or assert_calls, or whatever)
as simply an aggregate of lots of simpler asserts (assert_equals,
assert_raises, etc) as one big assert. The difference is, it's shorter
(it saves some keystrokes), nicer/tidier, and the "spec" itself is a
data structure and can be changed/manipulated/displayed/etc more easily.

Of course, this "spec" in many cases cannot cover all aspects of the
testing, only simpler ones. So perhaps a better name for "spec" is desired.

What about 'assert_array' as name.. it sort of indicates that there
are multiple element to compare against.

def func(val)
val + val
end
expected = [
{ :in=>1, ut=>2 },
{ :in=>'a', ut=>'aa' },
]
assert_array(expected, :func)

Or perhaps

assert_inout(expected, :func)

 

[Jean-Hugues ROBERT]

JeanHuguesRobert: About the generation of HTML documentation, I feel, again, that the
full spec is not easy to grab. I would rather like to see some RDoc
tool that would consider test cases as "examples".

I'm currently embedding test cases into the documentation. They look a
lot like example code. I think this is commonly called "Specification by
example". I've written a tool for automatically extracting and running
them. Here is an example:

# Matches the space between words.
# This is a zero-width anchor.
#
# re = Regexp::English.word_boundary
# text = "Hello World!"
# text.gsub(re, "*") # => "*Hello* *World*!"
def word_boundary(....
If this test case where to fail you would get an useful error like this:

1) Failure:
test_word_boundary()
[(eval):3:in `eval'
(eval):7:in `test_word_boundary'
(eval):3:in `test_word_boundary']:
c:/ruby/lib/ruby/1.8/test/unit/assertions.rb:32:in `assert_block'<"\"*Hello* *World*!\""> expected but was <"\"Hello World!\"">.

Regards,
Florian Gross

This is definitely nice to me. Additionaly I don't like so much the idea of large distance between the code and the test code. But in your scheme a least part of the test code is close, right before the tested code, that's good I think.

"xxx # => something" is apparently a well accepted convention to describe the expected result of some code. I guess that some "standard" is needed to fully describe the expected result beyond simple cases.

I suspect that the error msg could be improved slightly, you most probably want to know the exact file/lineno where the assert failed. This may require some changes in test/unit that you probably will propose once your code is complete.

Keep me posted.

Yours,

Jean-Hugues Robert

 

[David Garamond]

[{:input => [1], utput => 0},
{:input => [3], utput_in => [0, 1, 2]},
{:input => [], utput_class => Float,
utput_range => 0...1.0}]

What about :in ut :fail :error

:in and ut are fine with me. As for the other perhaps we should mirror
more closely with the other assert_*() method names, e.g. :raises
(assert_raises), etc.

assert_array(expected, :func)
assert_inout(expected, :func)

Both are better than assert_follows_spec I think

 

[David Garamond]

This is definitely nice to me. Additionaly I don't like so much the idea of large distance between the code and the test code. But in your scheme a least part of the test code is close, right before the tested code, that's good I think.

My current gripe with unit test code is that they're the ugliest code I
write . There are lots of duplication (from copy-and-paste),
no/little modularization, full of quick 'n' dirty hacks, and I write
them often without thinking about maintenance at all.

I think this is also true for most unit tests out there. If you want to
look at the worst spaghetti code, look at the unit test

 

[Jim Weirich]

Both are better than assert_follows_spec I think

How about: assert_examples

 

[gabriele renzi]

il Sat, 29 May 2004 13:34:45 +0200, Florian Gross <[email protected]> ha
scritto::

I'm currently embedding test cases into the documentation. They look a
lot like example code. I think this is commonly called "Specification by
example". I've written a tool for automatically extracting and running
them. Here is an example:

<snip>

just a question: what if you need many tests per method/class?

 

[Florian Gross]

just a question: what if you need many tests per method/class?

Many tests or many assertions?

 

[gabriele renzi]

Many tests or many assertions?

Mh.. I'd say tests.
I believe that, once you find a bug in , say, a method, you should add
a test for it after you fix it. Probably the test is just one
assertion, anyway. What happens to the doctests ? do you let them grow
endlessy?

 

[Guillaume Marcais]

assert_matrix?

Guillaume.

Le 29 mai 04, à 21:33, Its Me a écrit :

 

[Florian Gross]

Mh.. I'd say tests.
I believe that, once you find a bug in , say, a method, you should add
a test for it after you fix it. Probably the test is just one
assertion, anyway. What happens to the doctests ? do you let them grow
endlessy?

No, I think you should only put tests in there that also have a strong
purpose as examples. I think it's actually okay to have those tests that
are cloaked as example code in the documentation and stuff that does
deeper testing in regular test suits.

But it might also make sense to put all of them into the documentation
so that the user knows for sure what belongs to the interface of the
method (the things that are tested for) and what's implementation.
(Unspecified behavior, the things that aren't tested for.)

 

[Eric Hodel]

--5vNYLRcllDrimb99
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

For deterministic functions/methods, the same set of inputs should=20
always produces the same output. What about adding something like:
=20
assert_follows_spec(method_or_class_name, spec)
=20
Where spec is an array of input and output pairs (and optionally an=20
exception, if the input should not be accepted):
=20
[{input =3D> [1, 1, 1], output =3D> 1},
{input =3D> [8, 1, 2], output =3D> 4},
{input =3D> [8, 3, 4], output =3D> 6},
{input =3D> [4, 3, 2], output =3D> 6},
{input =3D> [1, 2, nil], exception =3D> ArgumentError},
{input =3D> [1, 2, 0], exception =3D> ZeroDivisionError}]

This can make unit tests shorter and more readable. We can transform the= =20
spec into an HTML table, include it into the API documentation, etc.
=20
The assert_follows_spec() could even perform several iterations of tests= =20
if wanted, to test deterministic-ness.

If you've got a spec like this, why not write a method to autogenerate
test methods?

--=20
Eric Hodel - (e-mail address removed) - http://segment7.net
All messages signed with fingerprint:
FEC2 57F1 D465 EB15 5D6E 7C11 332A 551C 796C 9F04


--5vNYLRcllDrimb99
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (FreeBSD)

iD8DBQFAvMLpMypVHHlsnwQRAiMMAKCa7UqFf5HbftP7fcSI21iBzbHF8wCgwo5I
fN3jkjU06itjkycx1IQysCs=
=lO+n
-----END PGP SIGNATURE-----

--5vNYLRcllDrimb99--

 

[David Garamond]

If you've got a spec like this, why not write a method to autogenerate
test methods?

Why is that better than something like assert_follows_spec()? It's not
more readable IMO, it's not shorter, it adds an additional step in the
Makefile/build process.

 

[Eric Hodel]

--qySB1iFW++5nzUxH
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

=20
Why is that better than something like assert_follows_spec()? It's not=20
more readable IMO, it's not shorter, it adds an additional step in the=20
Makefile/build process.

No additional steps. Ruby does it for you:

require 'test/unit'

def my_meth(args)
args.inject 0 do |acc, item|
acc + item
end
end

class MyTest < Test::Unit::TestCase

SPEC_ONE =3D [[1, 1, 1], 3]
SPEC_TWO =3D [[8, 1, 2], 11]
SPEC_THREE =3D [[8, 3, 4], 15]
SPEC_FOUR =3D [[4, 3, 2], 9]

constants.each do |name|
next unless name =3D~ /^SPEC_(.*)/

eval "def test_#{$1.downcase}
input =3D self.class.const_get(\"#{name}\")[0]
output =3D self.class.const_get(\"#{name}\")[1]
assert_equal output, my_meth(input)
end"
end
end


--=20
Eric Hodel - (e-mail address removed) - http://segment7.net
All messages signed with fingerprint:
FEC2 57F1 D465 EB15 5D6E 7C11 332A 551C 796C 9F04


--qySB1iFW++5nzUxH
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (FreeBSD)

iD8DBQFAv1ZDMypVHHlsnwQRAlOlAJ4qUrwaBPIH9ieeMn3x4GI5HcbS0gCg7/r8
HR82jDVMT6FrS+siaVG2oJI=
=zRMF
-----END PGP SIGNATURE-----

--qySB1iFW++5nzUxH--

 

[Florian Gross]

it adds an additional step in the Makefile/build process.

No additional steps. Ruby does it for you:

require 'test/unit'

def my_meth(args)
args.inject 0 do |acc, item|
acc + item
end
end

class MyTest < Test::Unit::TestCase

SPEC_ONE = [[1, 1, 1], 3]
SPEC_TWO = [[8, 1, 2], 11]
SPEC_THREE = [[8, 3, 4], 15]
SPEC_FOUR = [[4, 3, 2], 9]

constants.each do |name|
next unless name =~ /^SPEC_(.*)/

eval "def test_#{$1.downcase}
input = self.class.const_get(\"#{name}\")[0]
output = self.class.const_get(\"#{name}\")[1]
assert_equal output, my_meth(input)
end"
end
end

Note that you don't need eval() to do this:

def my_meth(*args)
args.inject(0) do |state, item|
state + item
end
end

class MyTest < Test::Unit::TestCase
# This is an Array of Arrays of Arrays -- we should really use
# something with more implicit semantics instead.
Specs = [
[[8, 1, 1], 3],
[[8, 1, 2], 11],
[[8, 3, 4], 15],
[[4, 3, 2], 9]
]

Specs.each_with_index do |(input, output), index|
define_method"test_spec_#{index}") do
assert_equal(output, my_meth(*input))
end
end
end

Regards,
Florian Gross