random testing with Test::Unit

Discussion in 'Ruby' started by Eric Mahurin, Sep 26, 2005.

  1. Eric Mahurin

    Eric Mahurin Guest

    Does anybody else do random testing on their ruby code besides
    me? I picked it up because of my background (IC design) where
    random testing is one of the verification techniques of
    hardware.

    So far, I've used 3 random testing strategies for my ruby code.
    Here are the different ways that the expected response is
    determined using those strategies:

    - compare multiple implementations of the same functionality
    (i.e. StringIO vs. IO, ordered hash vs. Hash).

    - AOP. Make a wrapper class or module that does pre and post
    checks of the methods. Use that wrapper class/module when
    testing. When the pre-check fails, the current test would be
    aborted/skipped (random generation of inputs yielded something
    invalid). When the post-check fails, the test would fail.

    - determine the expected response (from some random input)
    within the test suite.


    For all of these strategies, I've had to hack up Test::Unit in
    the same way to add various features. Here are the things I've
    added:

    + instead of running the test_* methods in a fixed order once,
    run them in a random order for N passes.

    + abort testing when the first failure is reached.

    + easy way to skip the rest of the current test (when the
    random input is invalid). Don't include this in the number of
    tests or mark it as skipped.

    + way to pass number of passes of the test_* methods on the
    command line.

    + display random seed and way of passing it in on the command
    line.

    + test suite should have access to the --verbose level or
    another switch to control the debugging verbosity.

    + way to pass command-line options down to the test suite to
    control various things - what methods to test, what
    classes to test, various other flags.


    I would like to see most of these (optional) features added to
    test/unit. Anybody else have opinions on this subject?



    =09
    =09
    ______________________________________________________=20
    Yahoo! for Good=20
    Donate to the Hurricane Katrina relief effort.=20
    http://store.yahoo.com/redcross-donate3/=20
     
    Eric Mahurin, Sep 26, 2005
    #1
    1. Advertising

  2. Eric Mahurin

    Kev Jackson Guest


    >- AOP. Make a wrapper class or module that does pre and post
    >checks of the methods. Use that wrapper class/module when
    >testing. When the pre-check fails, the current test would be
    >aborted/skipped (random generation of inputs yielded something
    >invalid). When the post-check fails, the test would fail.
    >
    >

    Hi I'd love to see how you implemented this as I wanted to write my own
    AOP-like code for ruby but I got stuck with my lack of ruby knowledge!

    Kev
     
    Kev Jackson, Sep 27, 2005
    #2
    1. Advertising

  3. ------=_Part_4103_6481245.1127838264241
    Content-Type: text/plain; charset=ISO-8859-1
    Content-Transfer-Encoding: quoted-printable
    Content-Disposition: inline

    On 9/26/05, Eric Mahurin <> wrote:
    >
    > Does anybody else do random testing on their ruby code besides
    > me? I picked it up because of my background (IC design) where
    > random testing is one of the verification techniques of
    > hardware.



    I cannot remember having used random testing for Ruby code, though I've use=
    d
    it for code in other languages. Having a nice framework for it would be,
    well, nice.

    For all of these strategies, I've had to hack up Test::Unit in
    > the same way to add various features. Here are the things I've
    > added:
    >
    > + instead of running the test_* methods in a fixed order once,
    > run them in a random order for N passes.



    Running N times sounds like a subclass of TestCase would be appropriate.
    Otherwise, how do you distinguish between what tests should be run once and
    which should be run several time (for statistical coverage)?

    Random order sounds reasonable for everything, anyway.


    + abort testing when the first failure is reached.


    Also subclass material.

    + easy way to skip the rest of the current test (when the
    > random input is invalid). Don't include this in the number of
    > tests or mark it as skipped.



    Also subclass material, I think.

    + way to pass number of passes of the test_* methods on the
    > command line.
    >
    > + display random seed and way of passing it in on the command
    > line.
    >
    > + test suite should have access to the --verbose level or
    > another switch to control the debugging verbosity.



    My gut feeling is that this is inappropriate. I see a major part of the
    Test::Unit style testing as the fact that the programmer only has to check
    for pass/fail. Passing through verbiosity information seems to turn this on
    its head, encouraging setting up the output for inspection. (On the other
    hand, I have sometimes needed information for debugging - calling this
    "debug level" would do much to ease my concerns...)

    + way to pass command-line options down to the test suite to
    > control various things - what methods to test, what
    > classes to test, various other flags.



    I also get an initial bad feeling about this, as it makes it totally
    necessary to use the console-based testrunner.

    Eivind.
    --
    Hazzle free packages for Ruby?
    RPA is available from http://www.rubyarchive.org/

    ------=_Part_4103_6481245.1127838264241--
     
    Eivind Eklund, Sep 27, 2005
    #3
  4. Eric Mahurin

    Eric Mahurin Guest

    I agree that a derived class of Test::Unit::TestCase (maybe
    Test::Unit::RandomTestCase) would be appropriate. I tried this
    first. Unfortunately, I found several places where doing this
    became problematic - test runners/collectors, rubygems?? (don't
    remember now). There is the assumption that all derived
    classes of Test::Unit::TestCase are the actual test case
    classes for testing (not the case for this
    Test::Unit::RandomTestCase class). At the time, it seemed
    easier to hack into Test::Unit::TestCase and
    Test::Unit::TestSuite. Now I'm hacking into TestSuite#tests to
    redefine tests.each. It still doesn't seem like the right way.
    Some derived classes that handle random testing
    (RandomTestCase and RandomTestSuite) seems like the right way.

    --- Eivind Eklund <> wrote:

    > On 9/26/05, Eric Mahurin <> wrote:
    > >
    > > Does anybody else do random testing on their ruby code

    > besides
    > > me? I picked it up because of my background (IC design)

    > where
    > > random testing is one of the verification techniques of
    > > hardware.

    >=20
    >=20
    > I cannot remember having used random testing for Ruby code,
    > though I've used
    > it for code in other languages. Having a nice framework for
    > it would be,
    > well, nice.
    >=20
    > For all of these strategies, I've had to hack up Test::Unit
    > in
    > > the same way to add various features. Here are the things

    > I've
    > > added:
    > >
    > > + instead of running the test_* methods in a fixed order

    > once,
    > > run them in a random order for N passes.

    >=20
    >=20
    > Running N times sounds like a subclass of TestCase would be
    > appropriate.
    > Otherwise, how do you distinguish between what tests should
    > be run once and
    > which should be run several time (for statistical coverage)?
    >=20
    > Random order sounds reasonable for everything, anyway.
    >=20
    >=20
    > + abort testing when the first failure is reached.
    >=20
    >=20
    > Also subclass material.
    >=20
    > + easy way to skip the rest of the current test (when the
    > > random input is invalid). Don't include this in the number

    > of
    > > tests or mark it as skipped.

    >=20
    >=20
    > Also subclass material, I think.
    >=20
    > + way to pass number of passes of the test_* methods on the
    > > command line.
    > >
    > > + display random seed and way of passing it in on the

    > command
    > > line.
    > >
    > > + test suite should have access to the --verbose level or
    > > another switch to control the debugging verbosity.

    >=20
    >=20
    > My gut feeling is that this is inappropriate. I see a major
    > part of the
    > Test::Unit style testing as the fact that the programmer only
    > has to check
    > for pass/fail. Passing through verbiosity information seems
    > to turn this on
    > its head, encouraging setting up the output for inspection.
    > (On the other
    > hand, I have sometimes needed information for debugging -
    > calling this
    > "debug level" would do much to ease my concerns...)
    >=20
    > + way to pass command-line options down to the test suite to
    > > control various things - what methods to test, what
    > > classes to test, various other flags.

    >=20
    >=20
    > I also get an initial bad feeling about this, as it makes it
    > totally
    > necessary to use the console-based testrunner.


    As you guessed, I use random testing for both testing and
    debugging. With hand-coded tests, there is not as big a need
    for spitting out debug info, because the failure points you to
    the line # and you can gather what the inputs were to easily
    recreate the problem. This is not so with random testing -
    especially when there is some state carried over from one
    random test to the next (not done in hand-coded tests). When
    you have failures with random testing, you need a way to track
    down exactly what happened. You need extra debug info.

    Also since random testing can run many more tests and still be
    useful (thousands - even millions), having the ability to
    narrow the testing down to a specific area to test/debug would
    be useful.

    I've never used anything but a console test runner. Where does
    stdout/stderr go for the other runners. If you put it in a log
    file, it seems like it could still be useful for other runners.


    > Eivind.
    > --
    > Hazzle free packages for Ruby?
    > RPA is available from http://www.rubyarchive.org/
    >=20




    =09
    =09
    ______________________________________________________=20
    Yahoo! for Good=20
    Donate to the Hurricane Katrina relief effort.=20
    http://store.yahoo.com/redcross-donate3/=20
     
    Eric Mahurin, Sep 27, 2005
    #4
  5. Eric Mahurin

    Eric Mahurin Guest

    I gave another shot at making a derived TestCase class. I only
    needed one hack to effectively make it ignore this class when
    running tests: undef_method:)default_test). Here is what I
    came up with that has many of the features I wanted:

    require 'test/unit'

    module Test =20
    module Unit
    class RandomTestCase < TestCase
    def self.suite
    $debug_level =3D (ENV['DEBUG_LEVEL']||0).to_i if
    $debug_level.nil?
    if $random_seed.nil?
    $random_seed =3D ENV['RAND_SEED'].to_i.nonzero? ||
    (srand;srand)
    srand($random_seed)
    puts("random_seed: #{$random_seed}") if
    $debug_level>=3D1
    end
    random_iterations =3D (ENV['RAND_ITER']||8).to_f
    methods =3D Regexp.new(ENV['TEST_METHODS']||".*")
    suite =3D super
    tests =3D suite.tests
    tests.reject! { |t|
    !methods.match(t.name.gsub(/\Atest_/,''))
    }
    (class << tests;self;end).class_eval {
    def each
    catch:)stop_suite) {
    (@iterations*size).to_i.times {
    catch:)invalid_test) {
    yield(slice(rand(size)))
    }
    }
    }
    end
    }
    tests.instance_eval { @iterations =3D random_iterations }
    suite
    end
    undef_method:)default_test)
    def teardown
    if not passed?
    puts("\nrandom_seed: #{$random_seed}")
    throw:)stop_suite)
    end
    end
    end
    end
    end


    Here is that example I gave earlier using the above for random
    testing Array#push and Array#pop using AOP:


    class ArrayAOP < Array
    include Test::Unit::Assertions
    def push(*args)
    print("#{self.inspect}.push(*#{args.inspect}) -> ") if=20
    $debug_level>=3D1
    n =3D size
    na =3D args.size
    ret =3D super
    assert_equal(n+na,size)
    assert_equal(slice(-na,na),args)
    assert_same(self,ret)
    p(ret) if $debug_level>=3D1
    ret
    end
    def pop
    print("#{self.inspect}.pop -> ") if $debug_level>=3D1
    n =3D size
    ret0 =3D n.nonzero? ? slice(-1) : nil
    ret =3D super
    assert_equal(ret0,ret)
    assert_equal(n.nonzero? ? n-1 : 0,size)
    p(ret) if $debug_level>=3D1
    ret
    end
    end

    class ArrayTest < Test::Unit::RandomTestCase
    def self.suite
    @@object =3D ArrayAOP[]
    super
    end
    def test_push
    args =3D []
    rand(4).times {
    args << [0,"",[],nil,false][rand(5)]
    }
    @@object.push(*args)
    end
    def test_pop
    rand(4).times {
    @@object.pop
    }
    end
    end


    As you can see from above, with this RandomTestCase class, you
    can do random testing very easily. But, it still would be nice
    to get options from the command line rather than the
    environment as I'm doing in RandomTestCase.

    --- Eivind Eklund <> wrote:

    > On 9/26/05, Eric Mahurin <> wrote:
    > >
    > > Does anybody else do random testing on their ruby code

    > besides
    > > me? I picked it up because of my background (IC design)

    > where
    > > random testing is one of the verification techniques of
    > > hardware.

    >=20
    >=20
    > I cannot remember having used random testing for Ruby code,
    > though I've used
    > it for code in other languages. Having a nice framework for
    > it would be,
    > well, nice.
    >=20
    > For all of these strategies, I've had to hack up Test::Unit
    > in
    > > the same way to add various features. Here are the things

    > I've
    > > added:
    > >
    > > + instead of running the test_* methods in a fixed order

    > once,
    > > run them in a random order for N passes.

    >=20
    >=20
    > Running N times sounds like a subclass of TestCase would be
    > appropriate.
    > Otherwise, how do you distinguish between what tests should
    > be run once and
    > which should be run several time (for statistical coverage)?
    >=20
    > Random order sounds reasonable for everything, anyway.
    >=20
    >=20
    > + abort testing when the first failure is reached.
    >=20
    >=20
    > Also subclass material.
    >=20
    > + easy way to skip the rest of the current test (when the
    > > random input is invalid). Don't include this in the number

    > of
    > > tests or mark it as skipped.

    >=20
    >=20
    > Also subclass material, I think.
    >=20
    > + way to pass number of passes of the test_* methods on the
    > > command line.
    > >
    > > + display random seed and way of passing it in on the

    > command
    > > line.
    > >
    > > + test suite should have access to the --verbose level or
    > > another switch to control the debugging verbosity.

    >=20
    >=20
    > My gut feeling is that this is inappropriate. I see a major
    > part of the
    > Test::Unit style testing as the fact that the programmer only
    > has to check
    > for pass/fail. Passing through verbiosity information seems
    > to turn this on
    > its head, encouraging setting up the output for inspection.
    > (On the other
    > hand, I have sometimes needed information for debugging -
    > calling this
    > "debug level" would do much to ease my concerns...)
    >=20
    > + way to pass command-line options down to the test suite to
    > > control various things - what methods to test, what
    > > classes to test, various other flags.

    >=20
    >=20
    > I also get an initial bad feeling about this, as it makes it
    > totally
    > necessary to use the console-based testrunner.
    >=20
    > Eivind.
    > --
    > Hazzle free packages for Ruby?
    > RPA is available from http://www.rubyarchive.org/
    >=20




    =09
    __________________________________=20
    Yahoo! Mail - PC Magazine Editors' Choice 2005=20
    http://mail.yahoo.com
     
    Eric Mahurin, Sep 27, 2005
    #5
  6. Eric Mahurin

    Jeff Wood Guest

    ------=_Part_3883_22504389.1127854946434
    Content-Type: text/plain; charset=ISO-8859-1
    Content-Transfer-Encoding: quoted-printable
    Content-Disposition: inline

    Why do you need DEBUG LEVEL or whatever when the Ruby
    environment/interpreter already has $VERBOSE ... ???

    I mean, it's already there ... just turn up the verbosity ...

    ruby -v mytestcases.rb

    ... and your done ... ( am i missing something? )

    j.

    On 9/27/05, Eric Mahurin <> wrote:
    >
    > I gave another shot at making a derived TestCase class. I only
    > needed one hack to effectively make it ignore this class when
    > running tests: undef_method:)default_test). Here is what I
    > came up with that has many of the features I wanted:
    >
    > require 'test/unit'
    >
    > module Test
    > module Unit
    > class RandomTestCase < TestCase
    > def self.suite
    > $debug_level =3D (ENV['DEBUG_LEVEL']||0).to_i if
    > $debug_level.nil?
    > if $random_seed.nil?
    > $random_seed =3D ENV['RAND_SEED'].to_i.nonzero? ||
    > (srand;srand)
    > srand($random_seed)
    > puts("random_seed: #{$random_seed}") if
    > $debug_level>=3D1
    > end
    > random_iterations =3D (ENV['RAND_ITER']||8).to_f
    > methods =3D Regexp.new(ENV['TEST_METHODS']||".*")
    > suite =3D super
    > tests =3D suite.tests
    > tests.reject! { |t|
    > !methods.match(t.name.gsub(/\Atest_/,''))
    > }
    > (class << tests;self;end).class_eval {
    > def each
    > catch:)stop_suite) {
    > (@iterations*size).to_i.times {
    > catch:)invalid_test) {
    > yield(slice(rand(size)))
    > }
    > }
    > }
    > end
    > }
    > tests.instance_eval { @iterations =3D random_iterations }
    > suite
    > end
    > undef_method:)default_test)
    > def teardown
    > if not passed?
    > puts("\nrandom_seed: #{$random_seed}")
    > throw:)stop_suite)
    > end
    > end
    > end
    > end
    > end
    >
    >
    > Here is that example I gave earlier using the above for random
    > testing Array#push and Array#pop using AOP:
    >
    >
    > class ArrayAOP < Array
    > include Test::Unit::Assertions
    > def push(*args)
    > print("#{self.inspect}.push(*#{args.inspect}) -> ") if
    > $debug_level>=3D1
    > n =3D size
    > na =3D args.size
    > ret =3D super
    > assert_equal(n+na,size)
    > assert_equal(slice(-na,na),args)
    > assert_same(self,ret)
    > p(ret) if $debug_level>=3D1
    > ret
    > end
    > def pop
    > print("#{self.inspect}.pop -> ") if $debug_level>=3D1
    > n =3D size
    > ret0 =3D n.nonzero? ? slice(-1) : nil
    > ret =3D super
    > assert_equal(ret0,ret)
    > assert_equal(n.nonzero? ? n-1 : 0,size)
    > p(ret) if $debug_level>=3D1
    > ret
    > end
    > end
    >
    > class ArrayTest < Test::Unit::RandomTestCase
    > def self.suite
    > @@object =3D ArrayAOP[]
    > super
    > end
    > def test_push
    > args =3D []
    > rand(4).times {
    > args << [0,"",[],nil,false][rand(5)]
    > }
    > @@object.push(*args)
    > end
    > def test_pop
    > rand(4).times {
    > @@object.pop
    > }
    > end
    > end
    >
    >
    > As you can see from above, with this RandomTestCase class, you
    > can do random testing very easily. But, it still would be nice
    > to get options from the command line rather than the
    > environment as I'm doing in RandomTestCase.
    >
    > --- Eivind Eklund <> wrote:
    >
    > > On 9/26/05, Eric Mahurin <> wrote:
    > > >
    > > > Does anybody else do random testing on their ruby code

    > > besides
    > > > me? I picked it up because of my background (IC design)

    > > where
    > > > random testing is one of the verification techniques of
    > > > hardware.

    > >
    > >
    > > I cannot remember having used random testing for Ruby code,
    > > though I've used
    > > it for code in other languages. Having a nice framework for
    > > it would be,
    > > well, nice.
    > >
    > > For all of these strategies, I've had to hack up Test::Unit
    > > in
    > > > the same way to add various features. Here are the things

    > > I've
    > > > added:
    > > >
    > > > + instead of running the test_* methods in a fixed order

    > > once,
    > > > run them in a random order for N passes.

    > >
    > >
    > > Running N times sounds like a subclass of TestCase would be
    > > appropriate.
    > > Otherwise, how do you distinguish between what tests should
    > > be run once and
    > > which should be run several time (for statistical coverage)?
    > >
    > > Random order sounds reasonable for everything, anyway.
    > >
    > >
    > > + abort testing when the first failure is reached.
    > >
    > >
    > > Also subclass material.
    > >
    > > + easy way to skip the rest of the current test (when the
    > > > random input is invalid). Don't include this in the number

    > > of
    > > > tests or mark it as skipped.

    > >
    > >
    > > Also subclass material, I think.
    > >
    > > + way to pass number of passes of the test_* methods on the
    > > > command line.
    > > >
    > > > + display random seed and way of passing it in on the

    > > command
    > > > line.
    > > >
    > > > + test suite should have access to the --verbose level or
    > > > another switch to control the debugging verbosity.

    > >
    > >
    > > My gut feeling is that this is inappropriate. I see a major
    > > part of the
    > > Test::Unit style testing as the fact that the programmer only
    > > has to check
    > > for pass/fail. Passing through verbiosity information seems
    > > to turn this on
    > > its head, encouraging setting up the output for inspection.
    > > (On the other
    > > hand, I have sometimes needed information for debugging -
    > > calling this
    > > "debug level" would do much to ease my concerns...)
    > >
    > > + way to pass command-line options down to the test suite to
    > > > control various things - what methods to test, what
    > > > classes to test, various other flags.

    > >
    > >
    > > I also get an initial bad feeling about this, as it makes it
    > > totally
    > > necessary to use the console-based testrunner.
    > >
    > > Eivind.
    > > --
    > > Hazzle free packages for Ruby?
    > > RPA is available from http://www.rubyarchive.org/
    > >

    >
    >
    >
    >
    > __________________________________
    > Yahoo! Mail - PC Magazine Editors' Choice 2005
    > http://mail.yahoo.com
    >
    >



    --
    "http://ruby-lang.org -- do you ruby?"

    Jeff Wood

    ------=_Part_3883_22504389.1127854946434--
     
    Jeff Wood, Sep 27, 2005
    #6
  7. Eric Mahurin

    Eric Mahurin Guest

    Yes, there are several ruby switches to control
    warnings/debugging/verbosity:

    -d set debugging flags (set $DEBUG to true)
    -v print version number, then turn on verbose
    mode
    -w turn warnings on for your script
    -W[level] set warning level; 0=3Dsilence, 1=3Dmedium,
    2=3Dverbose (default)

    In addition to setting some global variables ($VERBOSE, $DEBUG,
    $-v, $-w, $-d), these turn on warnings that some ruby code may
    not like (produce excess warnings). -d probably also turns on
    some internal ruby debugging, but I don't know where.

    In addition to this, the autorunner in test/unit has
    --verbose[=3Dlevel]. I would like to reuse this, but it is not
    very easy to access (can get it through
    ObjectSpace.each_object).

    The TEST_METHODS below isn't needed as test/unit/autorunner
    already has a --name option which works fine for this.

    --- Jeff Wood <> wrote:

    > Why do you need DEBUG LEVEL or whatever when the Ruby
    > environment/interpreter already has $VERBOSE ... ???
    >=20
    > I mean, it's already there ... just turn up the verbosity ...
    >=20
    > ruby -v mytestcases.rb
    >=20
    > ... and your done ... ( am i missing something? )
    >=20
    > j.
    >=20
    > On 9/27/05, Eric Mahurin <> wrote:
    > >
    > > I gave another shot at making a derived TestCase class. I

    > only
    > > needed one hack to effectively make it ignore this class

    > when
    > > running tests: undef_method:)default_test). Here is what I
    > > came up with that has many of the features I wanted:
    > >
    > > require 'test/unit'
    > >
    > > module Test
    > > module Unit
    > > class RandomTestCase < TestCase
    > > def self.suite
    > > $debug_level =3D (ENV['DEBUG_LEVEL']||0).to_i if
    > > $debug_level.nil?
    > > if $random_seed.nil?
    > > $random_seed =3D ENV['RAND_SEED'].to_i.nonzero? ||
    > > (srand;srand)
    > > srand($random_seed)
    > > puts("random_seed: #{$random_seed}") if
    > > $debug_level>=3D1
    > > end
    > > random_iterations =3D (ENV['RAND_ITER']||8).to_f
    > > methods =3D Regexp.new(ENV['TEST_METHODS']||".*")
    > > suite =3D super
    > > tests =3D suite.tests
    > > tests.reject! { |t|
    > > !methods.match(t.name.gsub(/\Atest_/,''))
    > > }
    > > (class << tests;self;end).class_eval {
    > > def each
    > > catch:)stop_suite) {
    > > (@iterations*size).to_i.times {
    > > catch:)invalid_test) {
    > > yield(slice(rand(size)))
    > > }
    > > }
    > > }
    > > end
    > > }
    > > tests.instance_eval { @iterations =3D random_iterations }
    > > suite
    > > end
    > > undef_method:)default_test)
    > > def teardown
    > > if not passed?
    > > puts("\nrandom_seed: #{$random_seed}")
    > > throw:)stop_suite)
    > > end
    > > end
    > > end
    > > end
    > > end
    > >
    > >
    > > Here is that example I gave earlier using the above for

    > random
    > > testing Array#push and Array#pop using AOP:
    > >
    > >
    > > class ArrayAOP < Array
    > > include Test::Unit::Assertions
    > > def push(*args)
    > > print("#{self.inspect}.push(*#{args.inspect}) -> ") if
    > > $debug_level>=3D1
    > > n =3D size
    > > na =3D args.size
    > > ret =3D super
    > > assert_equal(n+na,size)
    > > assert_equal(slice(-na,na),args)
    > > assert_same(self,ret)
    > > p(ret) if $debug_level>=3D1
    > > ret
    > > end
    > > def pop
    > > print("#{self.inspect}.pop -> ") if $debug_level>=3D1
    > > n =3D size
    > > ret0 =3D n.nonzero? ? slice(-1) : nil
    > > ret =3D super
    > > assert_equal(ret0,ret)
    > > assert_equal(n.nonzero? ? n-1 : 0,size)
    > > p(ret) if $debug_level>=3D1
    > > ret
    > > end
    > > end
    > >
    > > class ArrayTest < Test::Unit::RandomTestCase
    > > def self.suite
    > > @@object =3D ArrayAOP[]
    > > super
    > > end
    > > def test_push
    > > args =3D []
    > > rand(4).times {
    > > args << [0,"",[],nil,false][rand(5)]
    > > }
    > > @@object.push(*args)
    > > end
    > > def test_pop
    > > rand(4).times {
    > > @@object.pop
    > > }
    > > end
    > > end
    > >
    > >
    > > As you can see from above, with this RandomTestCase class,

    > you
    > > can do random testing very easily. But, it still would be

    > nice
    > > to get options from the command line rather than the
    > > environment as I'm doing in RandomTestCase.
    > >
    > > --- Eivind Eklund <> wrote:
    > >
    > > > On 9/26/05, Eric Mahurin <> wrote:
    > > > >
    > > > > Does anybody else do random testing on their ruby code
    > > > besides
    > > > > me? I picked it up because of my background (IC design)
    > > > where
    > > > > random testing is one of the verification techniques of
    > > > > hardware.
    > > >
    > > >
    > > > I cannot remember having used random testing for Ruby

    > code,
    > > > though I've used
    > > > it for code in other languages. Having a nice framework

    > for
    > > > it would be,
    > > > well, nice.
    > > >
    > > > For all of these strategies, I've had to hack up

    > Test::Unit
    > > > in
    > > > > the same way to add various features. Here are the

    > things
    > > > I've
    > > > > added:
    > > > >
    > > > > + instead of running the test_* methods in a fixed

    > order
    > > > once,
    > > > > run them in a random order for N passes.
    > > >
    > > >
    > > > Running N times sounds like a subclass of TestCase would

    > be
    > > > appropriate.
    > > > Otherwise, how do you distinguish between what tests

    > should
    > > > be run once and
    > > > which should be run several time (for statistical

    > coverage)?
    > > >
    > > > Random order sounds reasonable for everything, anyway.
    > > >
    > > >
    > > > + abort testing when the first failure is reached.
    > > >
    > > >
    > > > Also subclass material.
    > > >
    > > > + easy way to skip the rest of the current test (when the
    > > > > random input is invalid). Don't include this in the

    > number
    > > > of
    > > > > tests or mark it as skipped.
    > > >
    > > >
    > > > Also subclass material, I think.
    > > >
    > > > + way to pass number of passes of the test_* methods on

    > the
    > > > > command line.
    > > > >
    > > > > + display random seed and way of passing it in on the
    > > > command
    > > > > line.
    > > > >
    > > > > + test suite should have access to the --verbose level

    > or
    > > > > another switch to control the debugging verbosity.
    > > >
    > > >
    > > > My gut feeling is that this is inappropriate. I see a

    > major
    > > > part of the
    > > > Test::Unit style testing as the fact that the programmer

    > only
    > > > has to check
    > > > for pass/fail. Passing through verbiosity information

    > seems
    > > > to turn this on
    > > > its head, encouraging setting up the output for

    > inspection.
    > > > (On the other
    > > > hand, I have sometimes needed information for debugging -
    > > > calling this
    > > > "debug level" would do much to ease my concerns...)
    > > >
    > > > + way to pass command-line options down to the test suite

    > to
    > > > > control various things - what methods to test, what
    > > > > classes to test, various other flags.
    > > >
    > > >
    > > > I also get an initial bad feeling about this, as it makes

    > it
    > > > totally
    > > > necessary to use the console-based testrunner.
    > > >
    > > > Eivind.
    > > > --
    > > > Hazzle free packages for Ruby?
    > > > RPA is available from http://www.rubyarchive.org/
    > > >

    > >
    > >
    > >
    > >
    > > __________________________________
    > > Yahoo! Mail - PC Magazine Editors' Choice 2005
    > > http://mail.yahoo.com
    > >
    > >

    >=20
    >=20
    > --
    > "http://ruby-lang.org -- do you ruby?"
    >=20
    > Jeff Wood
    >=20



    __________________________________________________
    Do You Yahoo!?
    Tired of spam? Yahoo! Mail has the best spam protection around=20
    http://mail.yahoo.com=20
     
    Eric Mahurin, Sep 27, 2005
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Edvard Majakari
    Replies:
    4
    Views:
    687
    Edvard Majakari
    Feb 25, 2005
  2. VvanN
    Replies:
    5
    Views:
    490
    Phlip
    Apr 28, 2006
  3. Ulrich Eckhardt

    unit-profiling, similar to unit-testing

    Ulrich Eckhardt, Nov 16, 2011, in forum: Python
    Replies:
    6
    Views:
    336
    Roy Smith
    Nov 18, 2011
  4. Bill Mosteller
    Replies:
    0
    Views:
    230
    Bill Mosteller
    Oct 22, 2009
  5. timr
    Replies:
    2
    Views:
    167
Loading...

Share This Page