Test::Unit -- multiple errors in test method ???

Discussion in 'Ruby' started by Johan Holmberg, Sep 15, 2003.

  1. Hi !

    I have been writing some unit tests with Test::Unit.

    I've noted that when an assertion fails in a test method, the
    remaining assertions in the same test method aren't even excuted.
    Here is an example:

    require 'test/unit'

    class TC_example < Test::Unit::TestCase

    def test_a
    assert_equal 2, 1 + 1
    assert_equal 5, 2 + 2 # gives error
    assert_equal 7, 2 + 5 # never executed !!!
    assert_equal 8, 3 + 4 # never executed !!!
    end

    end

    When running this I get:

    C:\> ruby TC_example.rb
    Loaded suite TC_example
    Started
    F
    Finished in 0.01 seconds.

    1) Failure!!!
    test_a(TC_example) [TC_example1.rb:8]:
    <5> expected but was
    <4>

    1 tests, 2 assertions, 1 failures, 0 errors


    The two last assertions aren't even executed.
    Is this a but or an intentional "feature" ?

    I've looked at the way JUNIT in Java does the same thing,
    and there I got *all* errors reported. That seems much more useful
    to mee.

    With the current Test::Unit behaviour in Ruby, I can't know if
    an error "hides" a number of other errors until I've fixed the first
    error (it feels like having a C-compiler that only report the first
    compilation error ...).

    I have tested this with Ruby 1.8, and also with a more recent
    CVS-snapshot.

    Am I missing something obvious,
    or should this be considered a bug in Test::Unit ?

    /Johan Holmberg
    Johan Holmberg, Sep 15, 2003
    #1
    1. Advertising

  2. Johan Holmberg

    David Corbin Guest

    On Monday 15 September 2003 08:08, Johan Holmberg wrote:
    > Hi !
    >
    > I have been writing some unit tests with Test::Unit.
    > The two last assertions aren't even executed.
    > Is this a but or an intentional "feature" ?


    "feature"

    >
    > I've looked at the way JUNIT in Java does the same thing,
    > and there I got *all* errors reported. That seems much more useful
    > to mee.


    You'd better look more closely. JUnit stops execution of the test at the
    first failed assertion. Of course, it doesn't report how many assertions
    it's making, so maybe it's less obvious to you.

    >
    > With the current Test::Unit behaviour in Ruby, I can't know if
    > an error "hides" a number of other errors until I've fixed the first
    > error (it feels like having a C-compiler that only report the first
    > compilation error ...).
    >


    If you've got a number of assertions in one test case, while not always, often
    they are "ascendent". That is, the second asssertion can only really be
    checked. if the first one is true. For example, so people might right

    assert_not_nil(a)
    assert_equals("foo",a.name)


    >
    > /Johan Holmberg


    --
    David Corbin <>
    David Corbin, Sep 15, 2003
    #2
    1. Advertising

  3. On Mon, 15 Sep 2003, David Corbin wrote:
    > >
    > > I've looked at the way JUNIT in Java does the same thing,
    > > and there I got *all* errors reported. That seems much more useful
    > > to mee.

    >
    > You'd better look more closely. JUnit stops execution of the test at the
    > first failed assertion. Of course, it doesn't report how many assertions
    > it's making, so maybe it's less obvious to you.
    >


    Yes, I missed that.

    >
    > If you've got a number of assertions in one test case, while not
    > always, often they are "ascendent". That is, the second
    > asssertion can only really be checked. if the first one is true.
    >


    Yes, in case this was a feature I suspected the thinking was like
    that. But how about these cases ?

    - the Rubicon tests for Ruby. The test methods test a number of
    things in the same method. As soon as *any* test fails the whole
    test suite soon becomes increasingly useless (maybe one could say
    that there should not be any errors, but that has always been
    the case when I have tried to run Rubicon ...)

    - I have tried to write some table driven tests. How should I do
    this ? Currently I tried something in the following style:

    ...
    @@data = [
    [2, 1, 1],
    [5, 2, 2], # should fail
    [7, 2, 5],
    [8, 3, 4], # should fail
    ]

    def test_b
    for facit, aa, bb in @@data
    assert_equal facit, aa + bb
    end
    end
    ...

    Here I have several tests that are not "increasing".
    (just before I post this I saw David Alan Blacks answer mentioning
    Test::Unit::AssertionFailedError. Should I catch that ? )


    Thanks for the replies,
    /Johan Holmberg
    Johan Holmberg, Sep 15, 2003
    #3
  4. On Mon, 15 Sep 2003 22:08:22 +0900, Johan Holmberg wrote:

    > I have been writing some unit tests with Test::Unit.
    >
    > I've noted that when an assertion fails in a test method, the
    > remaining assertions in the same test method aren't even excuted.
    > Here is an example:



    It is considered good practize only to *one* assertion per testcase, this
    way you won't have any assertions which never gets executed.

    Even though I doesn't always follow that convention :)

    --
    Simon Strandgaard
    Simon Strandgaard, Sep 15, 2003
    #4
  5. Johan Holmberg

    ts Guest

    >>>>> "J" == Johan Holmberg <> writes:

    J> - I have tried to write some table driven tests. How should I do
    J> this ? Currently I tried something in the following style:

    Something like this ?

    svg% cat b.rb
    #!/usr/bin/ruby
    require 'test/unit'

    class TC_example < Test::Unit::TestCase

    @@data = [
    [2, 1, 1, true],
    [5, 2, 2, false], # should fail
    [7, 2, 5, true],
    [8, 3, nil, TypeError], # should fail
    ]

    def test_a
    for facit, aa, bb, res in @@data
    case res
    when true
    assert_equal(facit, aa + bb)
    when false
    assert_not_same(facit, aa + bb)
    else
    assert_raises(res) { aa + bb }
    end
    end
    end
    end

    svg%

    svg% b.rb
    Loaded suite ./b
    Started
    ts, Sep 15, 2003
    #5
  6. "Simon Strandgaard" <> schrieb im Newsbeitrag
    news:p...
    > On Mon, 15 Sep 2003 22:08:22 +0900, Johan Holmberg wrote:
    >
    > > I have been writing some unit tests with Test::Unit.
    > >
    > > I've noted that when an assertion fails in a test method, the
    > > remaining assertions in the same test method aren't even excuted.
    > > Here is an example:

    >
    >
    > It is considered good practize only to *one* assertion per testcase,

    this
    > way you won't have any assertions which never gets executed.


    Did you mean to say "*one* assertion per test" (i.e. test method)? One
    assertion per test case (i.e. class) would be a bit lavish...

    But even if so: JUnit encourages the convention of one test per method,
    which typically leads to multiple assertions per test.

    > Even though I doesn't always follow that convention :)


    Same here.

    Regards

    robert
    Robert Klemme, Sep 15, 2003
    #6
  7. Johan Holmberg

    ts Guest

    >>>>> "J" == Johan Holmberg <> writes:

    J> No, not really.
    J> When I wrote "should fail" I just meant that I had written a line in
    J> the table that *as it stands* give an error.

    Then this ?

    svg% cat b.rb
    #!/usr/bin/ruby
    require 'test/unit'

    class TC_example < Test::Unit::TestCase

    @@data = [
    [2, 1, 1],
    [5, 2, 2], # should fail
    [7, 2, 5],
    [8, 3, 7], # should fail
    ]

    def test_a
    error = []
    for facit, aa, bb in @@data
    begin
    assert_equal(facit, aa + bb)
    rescue Exception
    error << $!
    end
    end
    error.each {|e| add_failure(e.message, e.backtrace) }
    end
    end

    svg%

    svg% b.rb
    Loaded suite ./b
    Started
    FF
    Finished in 0.002194 seconds.

    1) Failure!!!
    test_a(TC_example) [./b.rb:17]:
    <5> expected but was
    <4>

    2) Failure!!!
    test_a(TC_example) [./b.rb:17]:
    <8> expected but was
    <10>

    1 tests, 4 assertions, 2 failures, 0 errors
    svg%


    Guy Decoux
    ts, Sep 15, 2003
    #7
  8. On Mon, 15 Sep 2003, ts wrote:
    >
    > >>>>> "J" == Johan Holmberg <> writes:

    >
    > J> No, not really.
    > J> When I wrote "should fail" I just meant that I had written a line in
    > J> the table that *as it stands* give an error.
    >
    > Then this ?
    >


    Yes !
    Thanks, for the example code.

    It seems to solve my problems in the case where I want to use
    table-driven tests, and where each test (i.e. line in the table) is
    independent of the others.

    /Johan Holmberg



    > svg% cat b.rb
    > #!/usr/bin/ruby
    > require 'test/unit'
    >
    > class TC_example < Test::Unit::TestCase
    >
    > @@data = [
    > [2, 1, 1],
    > [5, 2, 2], # should fail
    > [7, 2, 5],
    > [8, 3, 7], # should fail
    > ]
    >
    > def test_a
    > error = []
    > for facit, aa, bb in @@data
    > begin
    > assert_equal(facit, aa + bb)
    > rescue Exception
    > error << $!
    > end
    > end
    > error.each {|e| add_failure(e.message, e.backtrace) }
    > end
    > end
    >
    > svg%
    >
    > svg% b.rb
    > Loaded suite ./b
    > Started
    > FF
    > Finished in 0.002194 seconds.
    >
    > 1) Failure!!!
    > test_a(TC_example) [./b.rb:17]:
    > <5> expected but was
    > <4>
    >
    > 2) Failure!!!
    > test_a(TC_example) [./b.rb:17]:
    > <8> expected but was
    > <10>
    >
    > 1 tests, 4 assertions, 2 failures, 0 errors
    > svg%
    >
    >
    > Guy Decoux
    >
    >
    Johan Holmberg, Sep 15, 2003
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Edvard Majakari
    Replies:
    4
    Views:
    681
    Edvard Majakari
    Feb 25, 2005
  2. VvanN
    Replies:
    5
    Views:
    484
    Phlip
    Apr 28, 2006
  3. kathy
    Replies:
    0
    Views:
    1,304
    kathy
    Aug 2, 2011
  4. Robert Klemme

    Method test::unit::TestSuite#<<(test)

    Robert Klemme, Aug 6, 2003, in forum: Ruby
    Replies:
    0
    Views:
    99
    Robert Klemme
    Aug 6, 2003
  5. timr
    Replies:
    2
    Views:
    159
Loading...

Share This Page