[RDOC] Using unit-tests as examples for a documentation

[benny]

dear list,

suddenly this idea jumped on my brain:

when using unit tests we got this pretty methods like
assert_equal()
and
assert_match()

in which we are telling what we expect to get back from a method.
For this methods we invent more or less realistical examples for
how we could use the tested method.
We only need to select realistic examples to let them be meaningful and if
they are, they could in fact serve as examples for a documentation as well.

In a similar way we use the "don'ts" - the prohibited usage of our method
as flesh for the
assert_raise()

tests (although I am not sure if negative examples are that useful for a
documentation).

As we all know: in the documentation nothing beats a good example!

Now the idea: Why not use the examples in the unit-tests (at least the
"positive ones") as link to a popup window "examples" in the
HTML-documentation created by rdoc?

For this to work properly we might need a convention for the locations of
the unit-tests (e.g. as given in Progr. RubyV.2, page 159*) and an optional
switch for rdoc to tell it to search for unit-tests and make the examples
for the corresponding methods.

what do you think about it? might that be possible or would there be so much
divergence in the modelling of unit tests that rdoc won't have a chance to
tie that all together?

I see a big advantage coming out of this approach.
Consider you were using unit-tests anyway on a regular basis. then you would
run them at least anytime you make changes to your old code. so you will
have to fix the unit-tests as well if you do some changes that affect the
usage of your method. once changed the automatically build examples would
be up to date if you are running rdoc the next time. (and even an outdated,
"forgotten" documentation might stay useful when there are working examples
coming with it).

Surely that would affect the way we organize unit-test name parameter
therein etc. but that should not be that bad.


regards, benny

* the example given is:

roman
|-----lib/
| |---- roman.rb
| |----- other files....
|----test/
| |-----test_roman.rb
| |----- other tests......

 

[Dave Thomas]

Now the idea: Why not use the examples in the unit-tests (at least the
"positive ones") as link to a popup window "examples" in the
HTML-documentation created by rdoc?

This has been raised before. The problem is that it's hard to work out
which tests test which methods: method naming conventions would help,
but even so you quite often have three test methods testing one
production method.

An alternative might be to embed the UTs in the comment for the method,
and then have TestUnit extract and run them automatically. I personally
quite like that approach, although it would need some special markup
that RDoc doesn't yet support.


Cheers

Dave

 

[benny]

This has been raised before.

sorry, I didn't notice that. Don't want to bore the list.

The problem is that it's hard to work out
which tests test which methods: method naming conventions would help,
but even so you quite often have three test methods testing one
production method.

but that could be made by a really simple convention:
the name of each test method for method "foo" should be

test_foo_individual_name

Anyway this would be an *option*, i.e. test-methods not following that
convention would silently slip away (which might also be a useful feature).

An alternative might be to embed the UTs in the comment for the method,
and then have TestUnit extract and run them automatically. I personally
quite like that approach, although it would need some special markup
that RDoc doesn't yet support.

I don't think it is good to embed the testing methods into the
documentation. Think of packaging your sources: it might be useful to
separate the tests from the code for smaller packages. the user when
generating the documentation would then not have the examples in the docs
but the developer might offer them on the online docs.

also think of a replacement for a set of given classes methods. you might
want to use the same tests on different development branches to ensure the
compatibility. for using rdoc in this case you start it in the
corresponding branch and make a simple symlink named "test" to the
unit-test directories.

regards,
benny

 

[Florian Gross]

This has been raised before. The problem is that it's hard to work out
which tests test which methods: method naming conventions would help,
but even so you quite often have three test methods testing one
production method.

An alternative might be to embed the UTs in the comment for the method,
and then have TestUnit extract and run them automatically. I personally
quite like that approach, although it would need some special markup
that RDoc doesn't yet support.

I'm currently using indentation as a sample code marker in
test-extract.rb and would welcome official tags for this. (Maybe
prefixing each line of code with optional whitespace and "|"?)
Integrating support for this into RDoc might also have other benefits
for output formatting and so on.

Here's a sample of what I am currently doing:

# Causes a pattern which would otherwise match
# greedily to match non-greedily.
#
# This can only be applied to #count and #optional
# based constructs.
#
# re_greedy = Regexp::English.new do
# literal("<") + something + literal(">")
# end
# re_non_greedy = Regexp::English.new do
# literal("<") + something.minimal + literal(">")
# end
# str = "<html><body>"
# re_greedy.match(str)[0] # => "<html><body>"
# re_non_greedy.match(str)[0] # => "<html>"
#
# This method will do nothing when applied to a Node
# which is already minimal:
#
# re_minimal = Regexp::English.something.minimal
# re_minimal.minimal == re_minimal # => true
#
# However, it will raise a NameError Exception when
# you try to call it on a Regexp Node that can't be
# made non-greedy:
#
# re = Regexp::English.literal("foo")
# re.minimal # raises NameError
def minimal
<implementation snipped>
end

The next version of test/unit will also feature a new syntax for this.
That one will use sample { assert_equal(code, result) } style test
cases directly before the method IIRC.

 

[Dave Thomas]

I'm currently using indentation as a sample code marker in
test-extract.rb and would welcome official tags for this. (Maybe
prefixing each line of code with optional whitespace and "|"?)
Integrating support for this into RDoc might also have other benefits
for output formatting and so on.

I'm not sure what you're asking for here--could you explain at bit
more? (RDoc will already set this indented code as code, won't it?)


Cheers

Dave

 

[Florian Gross]

I'm not sure what you're asking for here--could you explain at bit more?
(RDoc will already set this indented code as code, won't it?)

I'm thinking along the lines of a "sample code" icon or something
similar that can be clicked to display or hide the embedded sample code
and possibly also setup code that needs to be run for actually making
the sample code work for the HTML output.

Maybe something like in this simplified example:

class KeyedList
...

# Removes all items from a keyed list.
#
# % aList = KeyedList["foo", "bar", "qux"]
# | aList.clear
# | aList.size # => 0
def clear
size.times { pop }
end
end

Where "%" would by default be hidden and could be uncovered by clicking
the icon. I'm not sure about the symbols so feel free to change them to
something that makes more sense for you.

I think the biggest point would however be formatting samples
consistently so that they can be distinguished easily from other things
that make sense in a whitespace preserved area.

 

[benny]

Maybe something like in this simplified example:

class KeyedList
...

# Removes all items from a keyed list.
#
# % aList = KeyedList["foo", "bar", "qux"]
# | aList.clear
# | aList.size # => 0
def clear
size.times { pop }
end
end

could you please explain how that syntax would fit into Test::Unit?

benny

 

[Florian Gross]

# Removes all items from a keyed list.
#
# % aList = KeyedList["foo", "bar", "qux"]
# | aList.clear
# | aList.size # => 0

could you please explain how that syntax would fit into Test::Unit?

I have written a tool that scans such samples out of source code and
turns them into test cases.

 

[Dave Thomas]

Now the idea: Why not use the examples in the unit-tests (at least the
"positive ones") as link to a popup window "examples" in the
HTML-documentation created by rdoc?

I've been thinking a bit about this. I'm nervous about imposing my own
rules on how tests are organized and how methods are named: I'd like
RDoc to be as transparent as possible.

So here's another take on it. You can already link from the method to
its test using standard RDoc hyperlinks. How about adding a slight bit
of new linking smarts: if you hyperlink to a class/method where the
class name starts Test and the method name starts test_, RDoc will
hyperlink to it in a different window. That way you can read the
mainline code, and scroll to the appropriate test in the test window by
clicking. You comment block might contain:

# burble burble burble. Burble burble burble. Burble burble burble
# burble burble. See the unit test in TestThingie#test_create

def create
...
end

It'll use a named window, so all test code will share just one
additional window.


Cheers

Dave

 

[Florian Gross]

I've been thinking a bit about this. I'm nervous about imposing my own
rules on how tests are organized and how methods are named: I'd like
RDoc to be as transparent as possible.

Well, it would just be a style suggestion. There's no reason to enforce
tests being inlined into the documentation of methods, but there are use
cases where it makes good sense to do so. (Specification by example
comes to mind.) I'm not sure I understand the point about method names.

So here's another take on it. You can already link from the method to
its test using standard RDoc hyperlinks. How about adding a slight bit
of new linking smarts: if you hyperlink to a class/method where the
class name starts Test and the method name starts test_, RDoc will
hyperlink to it in a different window. That way you can read the
mainline code, and scroll to the appropriate test in the test window by
clicking. You comment block might contain:

# burble burble burble. Burble burble burble. Burble burble burble
# burble burble. See the unit test in TestThingie#test_create

I think this is too indirect. I'd really like to see the sample code
inlined in the documentation, but formatted differently so it stands
out. Being able to toggle the visibility of sample setup code would also
be a nice thing, but it isn't as important to me. Of course this is just
my personal opinion and other's are likely to have other thoughts about
this.

 

[Dave Thomas]

I think this is too indirect. I'd really like to see the sample code
inlined in the documentation, but formatted differently so it stands
out. Being able to toggle the visibility of sample setup code would
also be a nice thing, but it isn't as important to me. Of course this
is just my personal opinion and other's are likely to have other
thoughts about this.

Are you talking about sample code or unit tests here. For sample code,
I don't see why you'd want to hide it--wouldn't it be part of the
normal comment of the method?


Cheers

Dave

 

[Florian Gross]

Are you talking about sample code or unit tests here. For sample code, I
don't see why you'd want to hide it--wouldn't it be part of the normal
comment of the method?

I'm talking about cases where both are the same thing. (The overlap is
really quite big. It works out well and lets you automatically keep the
documentation and the code in synchronization. It offers quite some
benefits to treat test and sample code as the same thing though it does
not make sense for all test code to be used for samples.) However test
code can sometimes need complex set-up code for which it could make
sense to be hidden by default. I think a Ruby tutorial / book used the
technique of providing samples and hiding the parts that were explained
only in later chapters.

 

[benny]

I've been thinking a bit about this. I'm nervous about imposing my own
rules on how tests are organized and how methods are named: I'd like
RDoc to be as transparent as possible.

So here's another take on it. You can already link from the method to
its test using standard RDoc hyperlinks. How about adding a slight bit
of new linking smarts: if you hyperlink to a class/method where the
class name starts Test and the method name starts test_, RDoc will
hyperlink to it in a different window. That way you can read the
mainline code, and scroll to the appropriate test in the test window by
clicking. You comment block might contain:

# burble burble burble. Burble burble burble. Burble burble burble
# burble burble. See the unit test in TestThingie#test_create

def create
...
end

It'll use a named window, so all test code will share just one
additional window.


Cheers

Dave

(hoping your still watching this thread)

I find it a good idea. Maybe instead of worring about the best solution we
should allow both: unit-test in the documentation and documentation in the
unit-tests. your proposal has the advantage to be able to put some extra
documentation about the given examples in this popup-window (and separate
this from the main documentation, too).

following your proposal the question arises how to mark the parts of the
testing code that should be shown in the window / how to hide the
preparation.(when I got you right, you thought, that we build the
documentation also in the test-directory and link to it , but that would
hide the concrete code, wouldn't it ? and writing the same code 2 times -
for the comment block and the code itself is annoying)


cheers, benny