D
Daniel Berger
All,
This is a long post, and I hate to sound like I'm lecturing, but...
Matz's announcement of 1.8.3 beta 1 reminds me of something. We need a
test suite. A large, comprehensive test suite. Written using
test-unit. And we need it NOW.
I know that Chad Fowler started something along those lines (the
"Rubicon" project on RubyForge). But, that was mainly a translation of
RubyUnit to TestUnit. I don't think those tests were sufficient. They
were also written in the 1.6.x days, making them potentially obsolete.
Another thing that concerns me is that some packages in the standard
library don't appear to have any test suite. Or, at least. they aren't
in the 'test' directory.
So, we need to fix this. Here's a few suggestions I have:
* Each class in the core library should have it's own directory under
'test' in the Ruby distro. Then each method should have its own test
case. That's right - each *METHOD* (for the core Ruby classes).
Thus:
ruby-1.8.3/
test/
core/
Array/
tc_clear.rb
tc_collect.rb
tc_collect_bang.rb
tc_compact.rb
...
Bignum/
tc_div.rb
...
stdlib/
Logger/
...
As things stand now we've got a single file for each core class, but if
we write comprehensive test cases for each method, putting every test in
a single file would become unwieldy. Hence the suggestion to have each
method in its own file.
Regarding the packages in the standard library. I know some of them
have the tests inlined in the source code (Pathname, for example). I
believe such test suites should be separated from the source itself, and
put in the 'stdlib' subdirectory I propose. Having it in a single,
standard location makes it easier for developers to find and run,
instead of having to dig through subdirectories manually. It might also
prove to be immensely useful for anyone developing packages around test
failure reporting tools.
We also need a well organized, professional looking, online database
where folks can report success and failures. Such a database would
contain information like platform, compiler, Ruby version, etc., and
list which tests succeeded and which failed. This could all be
automated during a "make test" (presumably optional). Such a database
could also be used for individual packages.
Here's an example from CPAN to show you what I mean (though I don't
particularly like the layout):
http://testers.cpan.org/show/Set-Array.html#Set-Array-0.12
As an aside, I would like to see Rake included in 1.8.3 and beyond
because I think it could help with automating some of these things.
Having said all this, this project is just too big for one person to do
in their free time. Short of someone offering me a full time salary to
pursue my goal, this effort will need many volunteers.
So, here's what I think we need to do:
* Register a project on RubyForge (I'll handle that, depending on the
response I get).
* Canvas for volunteers - one person per one or two core classes/modules
(ideally). See below.
* Volunteers would be responsible for writing test cases for their
assigned classes/modules.
* The authors of the various stdlib packages would be responsible for
maintaining their own test suites, or for finding someone to handle that
task for them.
* One or two people would write the online database that would handle
test reports. (Perhaps a Rails or Nitro app).
Hopefully, this rant hasn't come off as condescending or rude. I'm just
a little frustrated with the current state of the test suite, especially
when I see just how well done the Perl test suite is. It was also
motivated by the desire to submit core changes without having to provide
a custom test case every time. I could submit a diff, perhaps a bug
fix, and say with confidence, "Yup, still passes the test suite", to say
nothing of actually finding potential bugs in the already existant code,
or potentially useless code.
Anyway, if you would like to volunteer, please respond to this post or
send me a personal email at djberg96 at yahoo dot com. Indicate which
classes or modules you would like to be responsible for, though please
limit yourself to two, max, at least initially.
If anyone feels that I am out of line, or that this is a waste of time,
I'm listening.
Regards
Dan
PS - I'll take the Array class.
This is a long post, and I hate to sound like I'm lecturing, but...
Matz's announcement of 1.8.3 beta 1 reminds me of something. We need a
test suite. A large, comprehensive test suite. Written using
test-unit. And we need it NOW.
I know that Chad Fowler started something along those lines (the
"Rubicon" project on RubyForge). But, that was mainly a translation of
RubyUnit to TestUnit. I don't think those tests were sufficient. They
were also written in the 1.6.x days, making them potentially obsolete.
Another thing that concerns me is that some packages in the standard
library don't appear to have any test suite. Or, at least. they aren't
in the 'test' directory.
So, we need to fix this. Here's a few suggestions I have:
* Each class in the core library should have it's own directory under
'test' in the Ruby distro. Then each method should have its own test
case. That's right - each *METHOD* (for the core Ruby classes).
Thus:
ruby-1.8.3/
test/
core/
Array/
tc_clear.rb
tc_collect.rb
tc_collect_bang.rb
tc_compact.rb
...
Bignum/
tc_div.rb
...
stdlib/
Logger/
...
As things stand now we've got a single file for each core class, but if
we write comprehensive test cases for each method, putting every test in
a single file would become unwieldy. Hence the suggestion to have each
method in its own file.
Regarding the packages in the standard library. I know some of them
have the tests inlined in the source code (Pathname, for example). I
believe such test suites should be separated from the source itself, and
put in the 'stdlib' subdirectory I propose. Having it in a single,
standard location makes it easier for developers to find and run,
instead of having to dig through subdirectories manually. It might also
prove to be immensely useful for anyone developing packages around test
failure reporting tools.
We also need a well organized, professional looking, online database
where folks can report success and failures. Such a database would
contain information like platform, compiler, Ruby version, etc., and
list which tests succeeded and which failed. This could all be
automated during a "make test" (presumably optional). Such a database
could also be used for individual packages.
Here's an example from CPAN to show you what I mean (though I don't
particularly like the layout):
http://testers.cpan.org/show/Set-Array.html#Set-Array-0.12
As an aside, I would like to see Rake included in 1.8.3 and beyond
because I think it could help with automating some of these things.
Having said all this, this project is just too big for one person to do
in their free time. Short of someone offering me a full time salary to
pursue my goal, this effort will need many volunteers.
So, here's what I think we need to do:
* Register a project on RubyForge (I'll handle that, depending on the
response I get).
* Canvas for volunteers - one person per one or two core classes/modules
(ideally). See below.
* Volunteers would be responsible for writing test cases for their
assigned classes/modules.
* The authors of the various stdlib packages would be responsible for
maintaining their own test suites, or for finding someone to handle that
task for them.
* One or two people would write the online database that would handle
test reports. (Perhaps a Rails or Nitro app).
Hopefully, this rant hasn't come off as condescending or rude. I'm just
a little frustrated with the current state of the test suite, especially
when I see just how well done the Perl test suite is. It was also
motivated by the desire to submit core changes without having to provide
a custom test case every time. I could submit a diff, perhaps a bug
fix, and say with confidence, "Yup, still passes the test suite", to say
nothing of actually finding potential bugs in the already existant code,
or potentially useless code.
Anyway, if you would like to volunteer, please respond to this post or
send me a personal email at djberg96 at yahoo dot com. Indicate which
classes or modules you would like to be responsible for, though please
limit yourself to two, max, at least initially.
If anyone feels that I am out of line, or that this is a waste of time,
I'm listening.
Regards
Dan
PS - I'll take the Array class.