[PROPOSAL] We need a comprehensive test suite

Discussion in 'Ruby' started by Daniel Berger, May 12, 2005.

  1. All,

    This is a long post, and I hate to sound like I'm lecturing, but...

    Matz's announcement of 1.8.3 beta 1 reminds me of something. We need a
    test suite. A large, comprehensive test suite. Written using
    test-unit. And we need it NOW.

    I know that Chad Fowler started something along those lines (the
    "Rubicon" project on RubyForge). But, that was mainly a translation of
    RubyUnit to TestUnit. I don't think those tests were sufficient. They
    were also written in the 1.6.x days, making them potentially obsolete.

    Another thing that concerns me is that some packages in the standard
    library don't appear to have any test suite. Or, at least. they aren't
    in the 'test' directory.

    So, we need to fix this. Here's a few suggestions I have:

    * Each class in the core library should have it's own directory under
    'test' in the Ruby distro. Then each method should have its own test
    case. That's right - each *METHOD* (for the core Ruby classes).

    Thus:

    ruby-1.8.3/
    test/
    core/
    Array/
    tc_clear.rb
    tc_collect.rb
    tc_collect_bang.rb
    tc_compact.rb
    ...
    Bignum/
    tc_div.rb
    ...
    stdlib/
    Logger/
    ...

    As things stand now we've got a single file for each core class, but if
    we write comprehensive test cases for each method, putting every test in
    a single file would become unwieldy. Hence the suggestion to have each
    method in its own file.

    Regarding the packages in the standard library. I know some of them
    have the tests inlined in the source code (Pathname, for example). I
    believe such test suites should be separated from the source itself, and
    put in the 'stdlib' subdirectory I propose. Having it in a single,
    standard location makes it easier for developers to find and run,
    instead of having to dig through subdirectories manually. It might also
    prove to be immensely useful for anyone developing packages around test
    failure reporting tools.

    We also need a well organized, professional looking, online database
    where folks can report success and failures. Such a database would
    contain information like platform, compiler, Ruby version, etc., and
    list which tests succeeded and which failed. This could all be
    automated during a "make test" (presumably optional). Such a database
    could also be used for individual packages.

    Here's an example from CPAN to show you what I mean (though I don't
    particularly like the layout):

    http://testers.cpan.org/show/Set-Array.html#Set-Array-0.12

    As an aside, I would like to see Rake included in 1.8.3 and beyond
    because I think it could help with automating some of these things.

    Having said all this, this project is just too big for one person to do
    in their free time. Short of someone offering me a full time salary to
    pursue my goal, this effort will need many volunteers.

    So, here's what I think we need to do:

    * Register a project on RubyForge (I'll handle that, depending on the
    response I get).
    * Canvas for volunteers - one person per one or two core classes/modules
    (ideally). See below.
    * Volunteers would be responsible for writing test cases for their
    assigned classes/modules.
    * The authors of the various stdlib packages would be responsible for
    maintaining their own test suites, or for finding someone to handle that
    task for them.
    * One or two people would write the online database that would handle
    test reports. (Perhaps a Rails or Nitro app).

    Hopefully, this rant hasn't come off as condescending or rude. I'm just
    a little frustrated with the current state of the test suite, especially
    when I see just how well done the Perl test suite is. It was also
    motivated by the desire to submit core changes without having to provide
    a custom test case every time. I could submit a diff, perhaps a bug
    fix, and say with confidence, "Yup, still passes the test suite", to say
    nothing of actually finding potential bugs in the already existant code,
    or potentially useless code.

    Anyway, if you would like to volunteer, please respond to this post or
    send me a personal email at djberg96 at yahoo dot com. Indicate which
    classes or modules you would like to be responsible for, though please
    limit yourself to two, max, at least initially.

    If anyone feels that I am out of line, or that this is a waste of time,
    I'm listening.

    Regards

    Dan

    PS - I'll take the Array class. :)
    Daniel Berger, May 12, 2005
    #1
    1. Advertising

  2. Daniel Berger

    Guest

    Daniel Berger wrote:
    >
    > Matz's announcement of 1.8.3 beta 1 reminds me of something. We need a
    > test suite. A large, comprehensive test suite. Written using
    > test-unit. And we need it NOW.


    I whole-heartedly agree.

    I was sad to see Rubicon's luster fade, and to see "make test"
    compete it's job so quick and quietly...

    Regards,
    --
    Bil Kleb
    http://fun3d.larc.nasa.gov
    , May 12, 2005
    #2
    1. Advertising

  3. ha scritto:
    > Daniel Berger wrote:
    >
    >>
    >> Matz's announcement of 1.8.3 beta 1 reminds me of something. We need
    >> a test suite. A large, comprehensive test suite. Written using
    >> test-unit. And we need it NOW.

    >
    >
    > I whole-heartedly agree.
    >
    > I was sad to see Rubicon's luster fade, and to see "make test"
    > compete it's job so quick and quietly...


    well there is make test-all.. but I think Daniel is saying that even it
    is not enough
    gabriele renzi, May 12, 2005
    #3
  4. Daniel Berger

    John Wilger Guest

    On 5/12/05, Daniel Berger <> wrote:
    > * One or two people would write the online database that would handle
    > test reports. (Perhaps a Rails or Nitro app).


    I'm not sure how much help I could give to writing all the actual
    tests, but I'd certainly be willing to help with the database app.

    --=20
    Regards,
    John Wilger

    -----------
    Alice came to a fork in the road. "Which road do I take?" she asked.
    "Where do you want to go?" responded the Cheshire cat.
    "I don't know," Alice answered.
    "Then," said the cat, "it doesn't matter."
    - Lewis Carrol, Alice in Wonderland
    John Wilger, May 12, 2005
    #4
  5. Daniel Berger

    Joe Van Dyk Guest

    On 5/12/05, <> wrote:
    > Daniel Berger wrote:
    > >
    > > Matz's announcement of 1.8.3 beta 1 reminds me of something. We need a
    > > test suite. A large, comprehensive test suite. Written using
    > > test-unit. And we need it NOW.

    >=20
    > I whole-heartedly agree.
    >=20
    > I was sad to see Rubicon's luster fade, and to see "make test"
    > compete it's job so quick and quietly...


    Me too. I'm a terrible programmer, but would love to help.
    Joe Van Dyk, May 12, 2005
    #5
  6. * Daniel Berger () wrote:
    > All,
    >
    > This is a long post, and I hate to sound like I'm lecturing, but...
    >
    > Matz's announcement of 1.8.3 beta 1 reminds me of something. We need a
    > test suite. A large, comprehensive test suite. Written using
    > test-unit. And we need it NOW.


    I can go along with this. But before anyone commits too much time on this
    it would be nice to know what the core developers think about this.

    If I were to join this effort I'd like to know it was going to be included
    in the ruby source tree.

    Great effort.

    Cheers,

    Emiel
    --
    Emiel van de Laar
    Emiel van de Laar, May 12, 2005
    #6
  7. On 12/05/05, Emiel van de Laar <> wrote:
    > * Daniel Berger () wrote:
    > > All,
    > >
    > > This is a long post, and I hate to sound like I'm lecturing, but...
    > >
    > > Matz's announcement of 1.8.3 beta 1 reminds me of something. We need a
    > > test suite. A large, comprehensive test suite. Written using
    > > test-unit. And we need it NOW.

    >=20
    > I can go along with this. But before anyone commits too much time on this
    > it would be nice to know what the core developers think about this.
    >=20
    > If I were to join this effort I'd like to know it was going to be include=

    d
    > in the ruby source tree.
    >=20
    > Great effort.
    >=20
    > Cheers,
    >=20
    > Emiel
    > --
    > Emiel van de Laar
    >=20
    >=20


    A worthy goal I think. I'd love to learn something more about testing
    in such a hands on exercise. I'd even skip a ruby quiz or two for this
    ;)

    But, as Emiel said, only if its worthwile.

    best regards,

    Brian Schr=F6der

    --=20
    http://ruby.brian-schroeder.de/

    Stringed instrument chords: http://chordlist.brian-schroeder.de/
    Brian Schröder, May 12, 2005
    #7
  8. Daniel Berger

    Eric Hodel Guest

    On 12 May 2005, at 08:59, Daniel Berger wrote:

    > All,
    >
    > This is a long post, and I hate to sound like I'm lecturing, but...
    >
    > Matz's announcement of 1.8.3 beta 1 reminds me of something. We
    > need a test suite. A large, comprehensive test suite. Written
    > using test-unit. And we need it NOW.
    >
    > I know that Chad Fowler started something along those lines (the
    > "Rubicon" project on RubyForge). But, that was mainly a
    > translation of RubyUnit to TestUnit. I don't think those tests
    > were sufficient. They were also written in the 1.6.x days, making
    > them potentially obsolete.


    They're not too bad, actually.

    Plus the framework handles both 1.6 and 1.8 and changes between minor/
    major revs quite handily.

    They *do* need some loving to get up to 1.8, though.

    Ryan and I are using Rubicon's Array tests for implementing MetaRuby,
    and have noted missing tests in the suite, these include the block
    form of Array#fill and no tests were present for Array#zip. I don't
    know if Ryan has checked in those patches yet. Its not as if the
    entire core library was changed when we went from 1.6 to 1.8.

    While not perfect, reinventing Rubicon because Array was missing a
    handful of tests would be quite silly.

    > Another thing that concerns me is that some packages in the
    > standard library don't appear to have any test suite. Or, at
    > least. they aren't in the 'test' directory.
    >
    > So, we need to fix this. Here's a few suggestions I have:
    >
    > * Each class in the core library should have it's own directory
    > under 'test' in the Ruby distro. Then each method should have its
    > own test case. That's right - each *METHOD* (for the core Ruby
    > classes).


    [snip tree]

    > As things stand now we've got a single file for each core class,
    > but if we write comprehensive test cases for each method, putting
    > every test in a single file would become unwieldy. Hence the
    > suggestion to have each method in its own file.


    I don't think so:

    $ ls *Array* *String* *Hash* | xargs wc -l | sort -r
    3267 total
    1363 StringBase.rb
    1224 ArrayBase.rb
    634 HashBase.rb
    16 TestString.rb
    16 TestArray.rb
    14 TestHash.rb

    A 1500 line ruby file is far from unmanageable. This also reduces
    the benefit for people making their own Ruby implementation. Instead
    of looking through one file when implementing Array, they now have to
    look through over 100.

    > As an aside, I would like to see Rake included in 1.8.3 and beyond
    > because I think it could help with automating some of these things.


    Rubicon uses make and some custom C utilities. Using rake would
    reduce the utility for people using the test suite for alternative
    Ruby implementations. (JRuby uses this test suite, and so does
    MetaRuby.)

    > Having said all this, this project is just too big for one person
    > to do in their free time. Short of someone offering me a full time
    > salary to pursue my goal, this effort will need many volunteers.
    >
    > So, here's what I think we need to do:
    >
    > * Register a project on RubyForge (I'll handle that, depending on
    > the response I get).


    I think you'd be better off working on Rubicon. It has 75% of the
    work done for you already, has nice reports, and has many of the
    platform/version differences spelled out.

    You'll simply be reinventing the wheel by writing your own brand-new
    test suite for Ruby.

    > * Canvas for volunteers - one person per one or two core classes/
    > modules (ideally). See below.
    > * Volunteers would be responsible for writing test cases for their
    > assigned classes/modules.


    Please, don't reinvent the wheel.

    > * The authors of the various stdlib packages would be responsible
    > for maintaining their own test suites, or for finding someone to
    > handle that task for them.


    I think this should be broken apart from the core libraries. It is
    of lesser, but equally important, utility to Ruby implementors.

    > * One or two people would write the online database that would
    > handle test reports. (Perhaps a Rails or Nitro app).
    >
    > Hopefully, this rant hasn't come off as condescending or rude. I'm
    > just a little frustrated with the current state of the test suite,
    > especially when I see just how well done the Perl test suite is.
    > It was also motivated by the desire to submit core changes without
    > having to provide a custom test case every time. I could submit a
    > diff, perhaps a bug fix, and say with confidence, "Yup, still
    > passes the test suite", to say nothing of actually finding
    > potential bugs in the already existant code, or potentially useless
    > code.
    >
    > Anyway, if you would like to volunteer, please respond to this post
    > or send me a personal email at djberg96 at yahoo dot com. Indicate
    > which classes or modules you would like to be responsible for,
    > though please limit yourself to two, max, at least initially.
    >
    > If anyone feels that I am out of line, or that this is a waste of
    > time, I'm listening.


    I think you should take a serious look at Rubicon before rolling your
    own. Rubicon already has at least 75%, possibly 90% of the work you
    need already done. Save yourself lots of time and give it a little
    love, and you'll have something much, much better with much, much
    less effort.

    --
    Eric Hodel - - http://segment7.net
    FEC2 57F1 D465 EB15 5D6E 7C11 332A 551C 796C 9F04
    Eric Hodel, May 12, 2005
    #8
  9. Daniel Berger

    Chad Fowler Guest

    On 5/12/05, Eric Hodel <> wrote:
    >=20
    > On 12 May 2005, at 08:59, Daniel Berger wrote:
    >=20
    > I think you should take a serious look at Rubicon before rolling your
    > own. Rubicon already has at least 75%, possibly 90% of the work you
    > need already done. Save yourself lots of time and give it a little
    > love, and you'll have something much, much better with much, much
    > less effort.
    >=20
    > --


    Agreed. It would really be a waste not to use what's there. It's
    probably in much better shape than you expect. And, that's thanks to
    others who are _actively_ contributing to Rubicon.

    By the way, I moved the project to RubyForge but I didn't create
    Rubicon. It was created originally by Dave and Andy. The ideal would
    be to get Rubicon included in Ruby core (Matz agreed to it--my fault
    for not pushing it in at the time....anyone with commit access have
    time to do that?) and have it run when you do "make test".



    --=20

    Chad Fowler
    http://chadfowler.com
    http://rubycentral.org=20
    http://rubygarden.org=20
    http://rubygems.rubyforge.org (over 300,000 gems served!)
    Chad Fowler, May 12, 2005
    #9
  10. Eric Hodel wrote:
    >
    > On 12 May 2005, at 08:59, Daniel Berger wrote:
    >


    <snip>

    >
    > They're not too bad, actually.
    >
    > Plus the framework handles both 1.6 and 1.8 and changes between minor/
    > major revs quite handily.
    >
    > They *do* need some loving to get up to 1.8, though.


    Ok, I hadn't checked the CVS repository. I only noticed that there
    hasn't been a new file uploaded to the project in over a year. I
    thought it was dead.

    >
    > Ryan and I are using Rubicon's Array tests for implementing MetaRuby,
    > and have noted missing tests in the suite, these include the block form
    > of Array#fill and no tests were present for Array#zip. I don't know if
    > Ryan has checked in those patches yet. Its not as if the entire core
    > library was changed when we went from 1.6 to 1.8.
    >
    > While not perfect, reinventing Rubicon because Array was missing a
    > handful of tests would be quite silly.


    I don't totally understand the way tests are written/organized in
    Rubicon. I'm also confused by the 'rubyunit' requirement. Isn't
    everything rewritten using testunit?

    <snip>

    >> As things stand now we've got a single file for each core class, but
    >> if we write comprehensive test cases for each method, putting every
    >> test in a single file would become unwieldy. Hence the suggestion to
    >> have each method in its own file.

    >
    >
    > I don't think so:
    >
    > $ ls *Array* *String* *Hash* | xargs wc -l | sort -r
    > 3267 total
    > 1363 StringBase.rb
    > 1224 ArrayBase.rb
    > 634 HashBase.rb
    > 16 TestString.rb
    > 16 TestArray.rb
    > 14 TestHash.rb
    >
    > A 1500 line ruby file is far from unmanageable. This also reduces the
    > benefit for people making their own Ruby implementation. Instead of
    > looking through one file when implementing Array, they now have to look
    > through over 100.


    We're talking about an extremely small number of people this would
    annoy. Having each method in its own file is much easier on my brain
    and reduces the possibility of side-effects, i.e. one test inadvertantly
    affecting another test.

    >> As an aside, I would like to see Rake included in 1.8.3 and beyond
    >> because I think it could help with automating some of these things.

    >
    >
    > Rubicon uses make and some custom C utilities. Using rake would reduce
    > the utility for people using the test suite for alternative Ruby
    > implementations. (JRuby uses this test suite, and so does MetaRuby.)


    Perhaps, but using Rake would (in theory) significantly reduce cross
    platform issues. The make and custom C utilties could still be left in
    for precisely the reasons you mention, though.

    <snip>

    > I think you'd be better off working on Rubicon. It has 75% of the work
    > done for you already, has nice reports, and has many of the
    > platform/version differences spelled out.
    >
    > You'll simply be reinventing the wheel by writing your own brand-new
    > test suite for Ruby.


    <snip>

    > I think you should take a serious look at Rubicon before rolling your
    > own. Rubicon already has at least 75%, possibly 90% of the work you
    > need already done. Save yourself lots of time and give it a little
    > love, and you'll have something much, much better with much, much less
    > effort.
    >


    Alright, I will look at Rubicon again. :)

    Regards,

    Dan
    Daniel Berger, May 12, 2005
    #10
  11. Hi,

    In message "Re: [PROPOSAL] We need a comprehensive test suite"
    on Fri, 13 May 2005 04:24:49 +0900, Chad Fowler <> writes:

    |By the way, I moved the project to RubyForge but I didn't create
    |Rubicon. It was created originally by Dave and Andy. The ideal would
    |be to get Rubicon included in Ruby core (Matz agreed to it--my fault
    |for not pushing it in at the time....anyone with commit access have
    |time to do that?) and have it run when you do "make test".

    Why don't you just copy Rubicon tests in the test/ suites? Is it too
    hard to do? If Chad (or anyone else) needs CVS commit access, feel
    free to ask to <>.

    matz.
    Yukihiro Matsumoto, May 12, 2005
    #11
  12. Daniel Berger

    Chad Fowler Guest

    On 5/12/05, Yukihiro Matsumoto <> wrote:
    > Hi,
    >=20
    > In message "Re: [PROPOSAL] We need a comprehensive test suite"
    > on Fri, 13 May 2005 04:24:49 +0900, Chad Fowler <=
    > writes:
    >=20
    > |By the way, I moved the project to RubyForge but I didn't create
    > |Rubicon. It was created originally by Dave and Andy. The ideal would
    > |be to get Rubicon included in Ruby core (Matz agreed to it--my fault
    > |for not pushing it in at the time....anyone with commit access have
    > |time to do that?) and have it run when you do "make test".
    >=20
    > Why don't you just copy Rubicon tests in the test/ suites? Is it too
    > hard to do? If Chad (or anyone else) needs CVS commit access, feel
    > free to ask to <>.
    >=20


    Because I've been incredibly lazy about getting my CVS access back.=20
    Anyone? Anyone?

    I also haven't had time/focus to get it checked in and make sure it
    really works right. I don't want to create a mess.

    --=20

    Chad Fowler
    http://chadfowler.com
    http://rubycentral.org=20
    http://rubygarden.org=20
    http://rubygems.rubyforge.org (over 300,000 gems served!)
    Chad Fowler, May 12, 2005
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Maziar Aflatoun

    Looking for free online test suite

    Maziar Aflatoun, Jul 19, 2005, in forum: ASP .Net
    Replies:
    0
    Views:
    365
    Maziar Aflatoun
    Jul 19, 2005
  2. SR
    Replies:
    0
    Views:
    331
  3. Robert Brewer
    Replies:
    13
    Views:
    437
  4. Replies:
    6
    Views:
    350
    CBFalconer
    Feb 28, 2007
  5. D. Krmpotic
    Replies:
    4
    Views:
    223
    NAKAMURA, Hiroshi
    Aug 3, 2007
Loading...

Share This Page