Is it possible to dynamically extend Test::Unit test cases?

Discussion in 'Ruby' started by David Mitchell, Jul 15, 2008.

  1. Hello list,

    I've got some "normal" test cases that look like this


    class Testcases < Test::Unit::TestCase

    def setup
    ...
    end
    def teardown
    ...
    end
    def test_1
    ...
    end
    def test_2
    ...
    end
    end


    The test cases run as expected, but now I need to add some new test
    cases at run time. What I think I need to do is something like this


    require 'testcases'
    t = Testcases.new
    t.send:)define_method, "test_defined_at_run_time", "")


    but I'm getting "wrong number of arguments (0 for 1)" when I try to
    create the instance of Testcases.

    Is there some way around this?

    Thanks in advance

    David Mitchell
     
    David Mitchell, Jul 15, 2008
    #1
    1. Advertising

  2. David Mitchell

    phlip Guest

    David Mitchell wrote:

    > The test cases run as expected, but now I need to add some new test
    > cases at run time.


    Why? Just curious...

    > What I think I need to do is something like this


    Try this:

    class MySuite < Test::Unit::TestCase

    [:concept_1, :concept_2, :concept_3].each do |thang|
    define_method "test_#{thang}" do
    assert_concept thang
    end
    end

    def assert_concept(thang)
    # now convert the thang symbol to a real thing and test it
    end

    end

    > t.send:)define_method, "test_defined_at_run_time", "")


    You are trying too hard. define_method is easier than that to call.

    And note I put most of the processing _outside_ the define_method. Its only job
    is to construct a test case name and pass in a thang. This helps make
    assert_concept() more comprehensible.

    --
    Phlip
     
    phlip, Jul 15, 2008
    #2
    1. Advertising

  3. Thanks Phlip,

    The reason I need to do this is that we've got a small Watir-based DSL
    written to allow us to drive an app through code that looks sort of
    like:
    login("fred", "password")
    click_tab("Reports")
    click_drilldown("Asia")
    open_report("Some report title")
    ...

    It essentially lets us construct test cases in something like plain English.

    We built a few test cases using the DSL with a "normal"
    Test::Unit::TestCase approach, then showed them to our testers.
    Everyone was pretty excited about it; we can generate our own test
    data using the DSL, the testers can comprehend the DSL without having
    to dig into the nuts and bolts of the application itself, we can
    finally build a full regression test suite for an application that's
    basically a pig to drive using normal automation testing tools like
    QTP, the scripts we write using the DSL are easy to maintain over
    time, and everyone's happy.

    Once our testers got a look at that, they pointed out what should've
    been obvious all along: we can now get actual business users to write
    a lot of the test cases using the DSL, rather than using specialist
    testers. Rather than writing huge business requirements documents
    that have a habit of getting misinterpreted, we can get the business
    users to create what are essentially test cases using the DSL, and
    that gives the developers a reasonably unambiguous description of how
    things are supposed to work - we'll save a whole lot of time and money
    we're currently wasting translating between business-speak and
    developer-speak.

    The only problem was the "scaffolding" code; apparently business users
    are incapable of writing/extending code that looks like

    class Testcases < Test::Unit::TestCase
    def test_1
    <<DSL stuff here>>
    end
    def test_2
    <<more DSL stuff here>>
    end
    ...
    end

    but they are capable of creating a bunch of test cases in individual
    files that contain nothing but the DSL commands. They'll use e.g.
    Notepad to create test cases in individual files that look like:
    login('fred', 'password')
    click_tab('Reports')
    ...

    Fine with me - I just work here...

    So now I've got a situation where we're going to have business users
    generating loads of test cases using our DSL (without any of that
    nasty complicated Test::Unit::TestCase stuff), saving them in flat
    files, and we need to run be able to run some unspecified number of
    test cases that will change over time. What I need to be able to do
    is something like:

    Dir.glob("app_test_cases/**/*.app).each do |test_script_file|
    <<grab the content of each file, build a new Test::Unit::TestCase
    wrapper around it and eval it>>
    end

    That's no problem - I've got most of this working already; all I
    needed was the way to dynamically add new test cases, and you've now
    given me a way to do that. I'll have a play with it tomorrow.

    Thanks again

    David Mitchell

    2008/7/15 phlip <>:
    > David Mitchell wrote:
    >
    >> The test cases run as expected, but now I need to add some new test
    >> cases at run time.

    >
    > Why? Just curious...
    >
    >> What I think I need to do is something like this

    >
    > Try this:
    >
    > class MySuite < Test::Unit::TestCase
    >
    > [:concept_1, :concept_2, :concept_3].each do |thang|
    > define_method "test_#{thang}" do
    > assert_concept thang
    > end
    > end
    >
    > def assert_concept(thang)
    > # now convert the thang symbol to a real thing and test it
    > end
    >
    > end
    >
    >> t.send:)define_method, "test_defined_at_run_time", "")

    >
    > You are trying too hard. define_method is easier than that to call.
    >
    > And note I put most of the processing _outside_ the define_method. Its only
    > job is to construct a test case name and pass in a thang. This helps make
    > assert_concept() more comprehensible.
    >
    > --
    > Phlip
    >
    >
     
    David Mitchell, Jul 15, 2008
    #3
  4. David Mitchell

    Phlip Guest

    David Mitchell wrote:

    > The reason I need to do this is that we've got a small Watir-based DSL
    > written to allow us to drive an app through code that looks sort of
    > like:
    > login("fred", "password")
    > click_tab("Reports")
    > click_drilldown("Asia")
    > open_report("Some report title")
    > ..
    >
    > It essentially lets us construct test cases in something like plain English.


    Do your programmers run that after every edit?

    > We built a few test cases using the DSL with a "normal"
    > Test::Unit::TestCase approach, then showed them to our testers.
    > Everyone was pretty excited about it; we can generate our own test
    > data using the DSL, the testers can comprehend the DSL without having
    > to dig into the nuts and bolts of the application itself, we can
    > finally build a full regression test suite for an application that's
    > basically a pig to drive using normal automation testing tools like
    > QTP, the scripts we write using the DSL are easy to maintain over
    > time, and everyone's happy.


    Asking the question another way - do your developers write any tests?

    > Once our testers got a look at that, they pointed out what should've
    > been obvious all along: we can now get actual business users to write
    > a lot of the test cases using the DSL, rather than using specialist
    > testers. Rather than writing huge business requirements documents
    > that have a habit of getting misinterpreted, we can get the business
    > users to create what are essentially test cases using the DSL, and
    > that gives the developers a reasonably unambiguous description of how
    > things are supposed to work - we'll save a whole lot of time and money
    > we're currently wasting translating between business-speak and
    > developer-speak.


    Awesome! Now, can your business side actually run the tests - such as thru a web
    site with a "what if" interface?

    > The only problem was the "scaffolding" code; apparently business users
    > are incapable of writing/extending code that looks like
    >
    > class Testcases < Test::Unit::TestCase
    > def test_1
    > <<DSL stuff here>>
    > end
    > def test_2
    > <<more DSL stuff here>>
    > end
    > ...
    > end
    >
    > but they are capable of creating a bunch of test cases in individual
    > files that contain nothing but the DSL commands. They'll use e.g.
    > Notepad to create test cases in individual files that look like:
    > login('fred', 'password')
    > click_tab('Reports')


    That sounds like Fitnesse's territory. It does the Notepad thing, but with a
    real GUI around it. Your customer team writes the test criteria in a DSL, and
    FIT acts as a test runner.

    > Fine with me - I just work here...
    >
    > So now I've got a situation where we're going to have business users
    > generating loads of test cases using our DSL (without any of that
    > nasty complicated Test::Unit::TestCase stuff), saving them in flat
    > files, and we need to run be able to run some unspecified number of
    > test cases that will change over time. What I need to be able to do
    > is something like:
    >
    > Dir.glob("app_test_cases/**/*.app).each do |test_script_file|
    > <<grab the content of each file, build a new Test::Unit::TestCase
    > wrapper around it and eval it>>
    > end


    Now the slight problem is you are using TestCase as your runner, when it's full
    of features you don't need, and thin on features you actually do need. More below.

    > That's no problem - I've got most of this working already; all I
    > needed was the way to dynamically add new test cases, and you've now
    > given me a way to do that. I'll have a play with it tomorrow.


    I use define_method, in Rails, like this:

    [all this controller's actions].each do
    define_method test_one_action_#{action} do
    # test one common thing
    end
    end

    The first thing you need to look for is if your failures are humane, or if they
    are a huge mass of developer-friendly diagnostics and stack traces. The great
    thing about a DSL (per RSpec) is (reputedly!) that faults can lead with clear
    English too: "The frob should have returned 42 but it returned 43".

    --
    Phlip
     
    Phlip, Jul 15, 2008
    #4
  5. [Note: parts of this message were removed to make it a legal post.]

    David,

    test/spec or shoulda or rspec can be used to provide a more "dsl-ish" front
    end for your tests.

    Bret

    On Tue, Jul 15, 2008 at 7:39 AM, David Mitchell <> wrote:

    > Thanks Phlip,
    >
    > The reason I need to do this is that we've got a small Watir-based DSL
    > written to allow us to drive an app through code that looks sort of
    > like:
    > login("fred", "password")
    > click_tab("Reports")
    > click_drilldown("Asia")
    > open_report("Some report title")
    > ...
    >
    > It essentially lets us construct test cases in something like plain
    > English.
    >
    > We built a few test cases using the DSL with a "normal"
    > Test::Unit::TestCase approach, then showed them to our testers.
    > Everyone was pretty excited about it; we can generate our own test
    > data using the DSL, the testers can comprehend the DSL without having
    > to dig into the nuts and bolts of the application itself, we can
    > finally build a full regression test suite for an application that's
    > basically a pig to drive using normal automation testing tools like
    > QTP, the scripts we write using the DSL are easy to maintain over
    > time, and everyone's happy.
    >
    > Once our testers got a look at that, they pointed out what should've
    > been obvious all along: we can now get actual business users to write
    > a lot of the test cases using the DSL, rather than using specialist
    > testers. Rather than writing huge business requirements documents
    > that have a habit of getting misinterpreted, we can get the business
    > users to create what are essentially test cases using the DSL, and
    > that gives the developers a reasonably unambiguous description of how
    > things are supposed to work - we'll save a whole lot of time and money
    > we're currently wasting translating between business-speak and
    > developer-speak.
    >
    > The only problem was the "scaffolding" code; apparently business users
    > are incapable of writing/extending code that looks like
    >
    > class Testcases < Test::Unit::TestCase
    > def test_1
    > <<DSL stuff here>>
    > end
    > def test_2
    > <<more DSL stuff here>>
    > end
    > ...
    > end
    >
    > but they are capable of creating a bunch of test cases in individual
    > files that contain nothing but the DSL commands. They'll use e.g.
    > Notepad to create test cases in individual files that look like:
    > login('fred', 'password')
    > click_tab('Reports')
    > ...
    >
    > Fine with me - I just work here...
    >
    > So now I've got a situation where we're going to have business users
    > generating loads of test cases using our DSL (without any of that
    > nasty complicated Test::Unit::TestCase stuff), saving them in flat
    > files, and we need to run be able to run some unspecified number of
    > test cases that will change over time. What I need to be able to do
    > is something like:
    >
    > Dir.glob("app_test_cases/**/*.app).each do |test_script_file|
    > <<grab the content of each file, build a new Test::Unit::TestCase
    > wrapper around it and eval it>>
    > end
    >
    > That's no problem - I've got most of this working already; all I
    > needed was the way to dynamically add new test cases, and you've now
    > given me a way to do that. I'll have a play with it tomorrow.
    >
    > Thanks again
    >
    > David Mitchell
    >
    > 2008/7/15 phlip <>:
    > > David Mitchell wrote:
    > >
    > >> The test cases run as expected, but now I need to add some new test
    > >> cases at run time.

    > >
    > > Why? Just curious...
    > >
    > >> What I think I need to do is something like this

    > >
    > > Try this:
    > >
    > > class MySuite < Test::Unit::TestCase
    > >
    > > [:concept_1, :concept_2, :concept_3].each do |thang|
    > > define_method "test_#{thang}" do
    > > assert_concept thang
    > > end
    > > end
    > >
    > > def assert_concept(thang)
    > > # now convert the thang symbol to a real thing and test it
    > > end
    > >
    > > end
    > >
    > >> t.send:)define_method, "test_defined_at_run_time", "")

    > >
    > > You are trying too hard. define_method is easier than that to call.
    > >
    > > And note I put most of the processing _outside_ the define_method. Its

    > only
    > > job is to construct a test case name and pass in a thang. This helps make
    > > assert_concept() more comprehensible.
    > >
    > > --
    > > Phlip
    > >
    > >

    >
    >



    --
    Bret Pettichord
    CTO, WatirCraft LLC, http://www.watircraft.com
    Lead Developer, Watir, http://wtr.rubyforge.org
    Blog (Essays), http://www.io.com/~wazmo/blog
    MiniBlog (Links), http://feeds.feedburner.com/bretshotlist
     
    Bret Pettichord, Jul 15, 2008
    #5
  6. David Mitchell

    aidy Guest

    On Jul 15, 7:33 am, Bret Pettichord <> wrote:
    > [Note:  parts of this message were removed to make it a legal post.]
    >
    > David,
    >
    > test/spec or shoulda or rspec can be used to provide a more "dsl-ish" front
    > end for your tests.
    >
    > Bret


    Bret's right.

    I would also argue that the xUnit narrative for workflow tests is
    inadequate.

    You could object model your AUT

    g_mail.goto_url
    g_mail.welcome_page.sign_up_for_google_mail_link.click
    gmail.create_account_page.first_name_field.set('aidy')
    gmail.create_account_page.last_name_field.set('lewis')

    example: http://wiki.openqa.org/display/WTR/Example Frameworks

    You can then wrap this up in Rspec story runner tests

    acceptance test:

    Given a user at gmail
    When clicks to create a account
    And enters first and second name
    Then #....

    steps

    Given "a user at gmail" do
    g_mail.goto_url
    end

    When "clicks to create a account" do
    g_mail.welcome_page.sign_up_for_google_mail_link.click
    end

    When "enters first and second name" do
    gmail.create_account.page.first_name_field.set('aidy')
    gmail.create_account.page.last_name_field.set('lewis')
    end

    ....

    I will example this and post on Watir site

    Aidy
     
    aidy, Jul 15, 2008
    #6
  7. Thanks Bret and Aidy

    I'll check out the options you've suggested today. I've only ever
    needed test/unit in the past, so have never bothered to look into
    alternate test frameworks.

    Regards

    David Mitchell

    2008/7/16 aidy <>:
    > On Jul 15, 7:33 am, Bret Pettichord <> wrote:
    >> [Note: parts of this message were removed to make it a legal post.]
    >>
    >> David,
    >>
    >> test/spec or shoulda or rspec can be used to provide a more "dsl-ish" front
    >> end for your tests.
    >>
    >> Bret

    >
    > Bret's right.
    >
    > I would also argue that the xUnit narrative for workflow tests is
    > inadequate.
    >
    > You could object model your AUT
    >
    > g_mail.goto_url
    > g_mail.welcome_page.sign_up_for_google_mail_link.click
    > gmail.create_account_page.first_name_field.set('aidy')
    > gmail.create_account_page.last_name_field.set('lewis')
    >
    > example: http://wiki.openqa.org/display/WTR/Example Frameworks
    >
    > You can then wrap this up in Rspec story runner tests
    >
    > acceptance test:
    >
    > Given a user at gmail
    > When clicks to create a account
    > And enters first and second name
    > Then #....
    >
    > steps
    >
    > Given "a user at gmail" do
    > g_mail.goto_url
    > end
    >
    > When "clicks to create a account" do
    > g_mail.welcome_page.sign_up_for_google_mail_link.click
    > end
    >
    > When "enters first and second name" do
    > gmail.create_account.page.first_name_field.set('aidy')
    > gmail.create_account.page.last_name_field.set('lewis')
    > end
    >
    > ....
    >
    > I will example this and post on Watir site
    >
    > Aidy
    >
    >
     
    David Mitchell, Jul 15, 2008
    #7
  8. David Mitchell

    Tachikoma Guest

    On Jul 15, 3:19 pm, David Mitchell <> wrote:
    > Hello list,
    >
    > I've got some "normal" test cases that look like this
    >
    > class Testcases < Test::Unit::TestCase
    >
    >   def setup
    >     ...
    >   end
    >   def teardown
    >     ...
    >   end
    >   def test_1
    >     ...
    >   end
    >   def test_2
    >     ...
    >   end
    > end
    >
    > The test cases run as expected, but now I need to add some new test
    > cases at run time.  What I think I need to do is something like this
    >
    > require 'testcases'
    > t = Testcases.new
    > t.send:)define_method, "test_defined_at_run_time", "")
    >
    > but I'm getting "wrong number of arguments (0 for 1)" when I try to
    > create the instance of Testcases.
    >
    > Is there some way around this?
    >
    > Thanks in advance
    >
    > David Mitchell


    I took the following test:

    require 'test/unit/ui/console/testrunner'
    require "test/unit"
    class TestCases < Test::Unit::TestCase
    def test_case_1
    p "case 1"
    end
    end
    eval "class TestCase2 < TestCases;def test_case_2;p 'case 2';end;end;"
    Test::Unit::UI::Console::TestRunner.run(TestCases)
    Test::Unit::UI::Console::TestRunner.run(TestCase2)

    It gives:
    Loaded suite TestCases
    Started
    "case 1"
    .
    Finished in 0.000433 seconds.

    1 tests, 0 assertions, 0 failures, 0 errors
    Loaded suite
    Started
    "case 1"
    ."case 2"
    .
    Finished in 0.001912 seconds.

    2 tests, 0 assertions, 0 failures, 0 errors

    It works well.and following is the same

    require 'test/unit/ui/console/testrunner'
    require "test/unit"
    class TestCases < Test::Unit::TestCase
    def test_case_1
    p "case 1"
    end
    end
    test_case_2 = Class.new(TestCases) do
    def test_case_2
    p "case 2"
    end
    end
    Test::Unit::UI::Console::TestRunner.run(TestCases)
    Test::Unit::UI::Console::TestRunner.run(test_case_2)

    if you take a look into the testcase.rb
    def initialize(test_method_name)
    so,I wrote the following code:

    require 'test/unit/ui/console/testrunner'
    require "test/unit"
    class TestCases < Test::Unit::TestCase
    def test_case_1
    p "case 1"
    end
    def test_case_2
    p "case 2"
    end
    end
    test_case_2 = TestCases.new:)test_case_2)
    Test::Unit::UI::Console::TestRunner.run(TestCases)
    Test::Unit::UI::Console::TestRunner.run(test_case_2)

    the result:
    Loaded suite TestCases
    Started
    "case 1"
    ."case 2"
    .
    Finished in 0.00051 seconds.

    2 tests, 0 assertions, 0 failures, 0 errors
    Loaded suite test_case_2(TestCases)
    Started
    "case 2"
    .
    Finished in 0.000314 seconds.

    1 tests, 0 assertions, 0 failures, 0 errors

    so the initialize function takes a methods name,and only the method is
    running if you run the instance
    but if you run the Class,the test* methods will all run through the
    TestCase.suite [testcase.rb]

    I hope this will help
     
    Tachikoma, Jul 16, 2008
    #8
  9. David Mitchell

    Nick Sieger Guest

    On Tue, Jul 15, 2008 at 7:39 AM, David Mitchell <> wrote:
    >
    > The only problem was the "scaffolding" code; apparently business users
    > are incapable of writing/extending code that looks like
    >
    > class Testcases < Test::Unit::TestCase
    > def test_1
    > <<DSL stuff here>>
    > end
    > def test_2
    > <<more DSL stuff here>>
    > end
    > ...
    > end
    >
    > but they are capable of creating a bunch of test cases in individual
    > files that contain nothing but the DSL commands. They'll use e.g.
    > Notepad to create test cases in individual files that look like:
    > login('fred', 'password')
    > click_tab('Reports')
    > ...
    >
    > Fine with me - I just work here...
    >
    > So now I've got a situation where we're going to have business users
    > generating loads of test cases using our DSL (without any of that
    > nasty complicated Test::Unit::TestCase stuff), saving them in flat
    > files, and we need to run be able to run some unspecified number of
    > test cases that will change over time. What I need to be able to do
    > is something like:
    >
    > Dir.glob("app_test_cases/**/*.app).each do |test_script_file|
    > <<grab the content of each file, build a new Test::Unit::TestCase
    > wrapper around it and eval it>>
    > end
    >
    > That's no problem - I've got most of this working already; all I
    > needed was the way to dynamically add new test cases, and you've now
    > given me a way to do that. I'll have a play with it tomorrow.


    If you haven't already, you might look at systir [1] -- does almost
    exactly what you're describing. It's a little old, and perhaps not
    maintained much, but it could give you an idea of how to do the
    test/unit testcase definition more cleanly.

    /Nick

    [1]: http://www.atomicobject.com/pages/System Testing in Ruby
     
    Nick Sieger, Jul 16, 2008
    #9
  10. David Mitchell

    aidy Guest

    Hi Nick

    > If you haven't already, you might look at systir [1] -- does almost
    > exactly what you're describing. It's a little old, and perhaps not
    > maintained much, but it could give you an idea of how to do the
    > test/unit testcase definition more cleanly.
    >
    > /Nick
    >
    > [1]:http://www.atomicobject.com/pages/System Testing in Ruby- Hide quoted text -
    >
    > - Show quoted text -


    The issues I have with the more procedural frameworks is that you are
    increasing your code base and thus maintenance.

    For example

    set_terms_of_use_checkbox
    clear_terms_of_use_checkbox

    Will wrap the same object in two methods

    Whereas this:

    gmail.sign_up_page.terms_of_use_checkbox.set
    gmail.sign_up_page.terms_of_use_checkbox.clear

    Is one object with two different methods operated upon it.

    Aidy
     
    aidy, Jul 16, 2008
    #10
  11. David Mitchell

    aidy Guest

    On 15 Jul, 22:33, David Mitchell <> wrote:
    > Thanks Bret andAidy
    >
    > I'll check out the options you've suggested today.  I've only ever
    > needed test/unit in the past, so have never bothered to look into
    > alternate test frameworks.
    >
    > Regards
    >
    > David Mitchell
    >
    > 2008/7/16aidy<>:
    >
    >
    >
    > > On Jul 15, 7:33 am, Bret Pettichord <> wrote:
    > >> [Note:  parts of this message were removed to make it a legal post.]

    >
    > >> David,

    >
    > >> test/spec or shoulda or rspec can be used to provide a more "dsl-ish" front
    > >> end for your tests.

    >
    > >> Bret

    >
    > > Bret's right.

    >
    > > I would also argue that the xUnit narrative for workflow tests is
    > > inadequate.

    >
    > > You could object model your AUT

    >
    > > g_mail.goto_url
    > > g_mail.welcome_page.sign_up_for_google_mail_link.click
    > > gmail.create_account_page.first_name_field.set('aidy')
    > > gmail.create_account_page.last_name_field.set('lewis')

    >
    > > example:http://wiki.openqa.org/display/WTR/Example Frameworks

    >
    > > You can then wrap this up in Rspec story runner tests

    >
    > > acceptance test:

    >
    > > Given a user at gmail
    > > When clicks to create a account
    > > And enters first and second name
    > > Then #....

    >
    > > steps

    >
    > > Given "a user at gmail" do
    > >  g_mail.goto_url
    > > end

    >
    > > When "clicks to create a account" do
    > >  g_mail.welcome_page.sign_up_for_google_mail_link.click
    > > end

    >
    > > When "enters first and second name" do
    > >  gmail.create_account.page.first_name_field.set('aidy')
    > >  gmail.create_account.page.last_name_field.set('lewis')
    > > end

    >
    > > ....

    >
    > > I will example this and post on Watir site

    >
    > >Aidy- Hide quoted text -

    >
    > - Show quoted text -


    I have added an example of the Rspec story runner that is used with
    the AUT object model and Watir

    http://wiki.openqa.org/display/WTR/Example Frameworks

    Aidy
     
    aidy, Jul 16, 2008
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Charlie
    Replies:
    1
    Views:
    392
    Lauwie
    Aug 22, 2003
  2. sripathi
    Replies:
    0
    Views:
    895
    sripathi
    Dec 31, 2009
  3. Replies:
    9
    Views:
    141
    Eric Hodel
    Nov 15, 2005
  4. aidy

    loop unit test cases?

    aidy, Jul 14, 2007, in forum: Ruby
    Replies:
    2
    Views:
    123
  5. jitendra gupta
    Replies:
    0
    Views:
    107
    jitendra gupta
    Mar 8, 2013
Loading...

Share This Page