random testing with Test::Unit

E

Eric Mahurin

Does anybody else do random testing on their ruby code besides
me? I picked it up because of my background (IC design) where
random testing is one of the verification techniques of
hardware.

So far, I've used 3 random testing strategies for my ruby code.
Here are the different ways that the expected response is
determined using those strategies:

- compare multiple implementations of the same functionality
(i.e. StringIO vs. IO, ordered hash vs. Hash).

- AOP. Make a wrapper class or module that does pre and post
checks of the methods. Use that wrapper class/module when
testing. When the pre-check fails, the current test would be
aborted/skipped (random generation of inputs yielded something
invalid). When the post-check fails, the test would fail.

- determine the expected response (from some random input)
within the test suite.


For all of these strategies, I've had to hack up Test::Unit in
the same way to add various features. Here are the things I've
added:

+ instead of running the test_* methods in a fixed order once,
run them in a random order for N passes.

+ abort testing when the first failure is reached.

+ easy way to skip the rest of the current test (when the
random input is invalid). Don't include this in the number of
tests or mark it as skipped.

+ way to pass number of passes of the test_* methods on the
command line.

+ display random seed and way of passing it in on the command
line.

+ test suite should have access to the --verbose level or
another switch to control the debugging verbosity.

+ way to pass command-line options down to the test suite to
control various things - what methods to test, what
classes to test, various other flags.


I would like to see most of these (optional) features added to
test/unit. Anybody else have opinions on this subject?



=09
=09
______________________________________________________=20
Yahoo! for Good=20
Donate to the Hurricane Katrina relief effort.=20
http://store.yahoo.com/redcross-donate3/=20
 
K

Kev Jackson

- AOP. Make a wrapper class or module that does pre and post
checks of the methods. Use that wrapper class/module when
testing. When the pre-check fails, the current test would be
aborted/skipped (random generation of inputs yielded something
invalid). When the post-check fails, the test would fail.
Hi I'd love to see how you implemented this as I wanted to write my own
AOP-like code for ruby but I got stuck with my lack of ruby knowledge!

Kev
 
E

Eivind Eklund

------=_Part_4103_6481245.1127838264241
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Does anybody else do random testing on their ruby code besides
me? I picked it up because of my background (IC design) where
random testing is one of the verification techniques of
hardware.


I cannot remember having used random testing for Ruby code, though I've use=
d
it for code in other languages. Having a nice framework for it would be,
well, nice.

For all of these strategies, I've had to hack up Test::Unit in
the same way to add various features. Here are the things I've
added:

+ instead of running the test_* methods in a fixed order once,
run them in a random order for N passes.


Running N times sounds like a subclass of TestCase would be appropriate.
Otherwise, how do you distinguish between what tests should be run once and
which should be run several time (for statistical coverage)?

Random order sounds reasonable for everything, anyway.


+ abort testing when the first failure is reached.


Also subclass material.

+ easy way to skip the rest of the current test (when the
random input is invalid). Don't include this in the number of
tests or mark it as skipped.


Also subclass material, I think.

+ way to pass number of passes of the test_* methods on the
command line.

+ display random seed and way of passing it in on the command
line.

+ test suite should have access to the --verbose level or
another switch to control the debugging verbosity.


My gut feeling is that this is inappropriate. I see a major part of the
Test::Unit style testing as the fact that the programmer only has to check
for pass/fail. Passing through verbiosity information seems to turn this on
its head, encouraging setting up the output for inspection. (On the other
hand, I have sometimes needed information for debugging - calling this
"debug level" would do much to ease my concerns...)

+ way to pass command-line options down to the test suite to
control various things - what methods to test, what
classes to test, various other flags.


I also get an initial bad feeling about this, as it makes it totally
necessary to use the console-based testrunner.

Eivind.
 
E

Eric Mahurin

I agree that a derived class of Test::Unit::TestCase (maybe
Test::Unit::RandomTestCase) would be appropriate. I tried this
first. Unfortunately, I found several places where doing this
became problematic - test runners/collectors, rubygems?? (don't
remember now). There is the assumption that all derived
classes of Test::Unit::TestCase are the actual test case
classes for testing (not the case for this
Test::Unit::RandomTestCase class). At the time, it seemed
easier to hack into Test::Unit::TestCase and
Test::Unit::TestSuite. Now I'm hacking into TestSuite#tests to
redefine tests.each. It still doesn't seem like the right way.
Some derived classes that handle random testing
(RandomTestCase and RandomTestSuite) seems like the right way.

--- Eivind Eklund said:
=20
=20
I cannot remember having used random testing for Ruby code,
though I've used
it for code in other languages. Having a nice framework for
it would be,
well, nice.
=20
For all of these strategies, I've had to hack up Test::Unit
in
=20
=20
Running N times sounds like a subclass of TestCase would be
appropriate.
Otherwise, how do you distinguish between what tests should
be run once and
which should be run several time (for statistical coverage)?
=20
Random order sounds reasonable for everything, anyway.
=20
=20
+ abort testing when the first failure is reached.
=20
=20
Also subclass material.
=20
+ easy way to skip the rest of the current test (when the
=20
=20
Also subclass material, I think.
=20
+ way to pass number of passes of the test_* methods on the
=20
=20
My gut feeling is that this is inappropriate. I see a major
part of the
Test::Unit style testing as the fact that the programmer only
has to check
for pass/fail. Passing through verbiosity information seems
to turn this on
its head, encouraging setting up the output for inspection.
(On the other
hand, I have sometimes needed information for debugging -
calling this
"debug level" would do much to ease my concerns...)
=20
+ way to pass command-line options down to the test suite to
=20
=20
I also get an initial bad feeling about this, as it makes it
totally
necessary to use the console-based testrunner.

As you guessed, I use random testing for both testing and
debugging. With hand-coded tests, there is not as big a need
for spitting out debug info, because the failure points you to
the line # and you can gather what the inputs were to easily
recreate the problem. This is not so with random testing -
especially when there is some state carried over from one
random test to the next (not done in hand-coded tests). When
you have failures with random testing, you need a way to track
down exactly what happened. You need extra debug info.

Also since random testing can run many more tests and still be
useful (thousands - even millions), having the ability to
narrow the testing down to a specific area to test/debug would
be useful.

I've never used anything but a console test runner. Where does
stdout/stderr go for the other runners. If you put it in a log
file, it seems like it could still be useful for other runners.




=09
=09
______________________________________________________=20
Yahoo! for Good=20
Donate to the Hurricane Katrina relief effort.=20
http://store.yahoo.com/redcross-donate3/=20
 
E

Eric Mahurin

I gave another shot at making a derived TestCase class. I only
needed one hack to effectively make it ignore this class when
running tests: undef_method:)default_test). Here is what I
came up with that has many of the features I wanted:

require 'test/unit'

module Test =20
module Unit
class RandomTestCase < TestCase
def self.suite
$debug_level =3D (ENV['DEBUG_LEVEL']||0).to_i if
$debug_level.nil?
if $random_seed.nil?
$random_seed =3D ENV['RAND_SEED'].to_i.nonzero? ||
(srand;srand)
srand($random_seed)
puts("random_seed: #{$random_seed}") if
$debug_level>=3D1
end
random_iterations =3D (ENV['RAND_ITER']||8).to_f
methods =3D Regexp.new(ENV['TEST_METHODS']||".*")
suite =3D super
tests =3D suite.tests
tests.reject! { |t|
!methods.match(t.name.gsub(/\Atest_/,''))
}
(class << tests;self;end).class_eval {
def each
catch:)stop_suite) {
(@iterations*size).to_i.times {
catch:)invalid_test) {
yield(slice(rand(size)))
}
}
}
end
}
tests.instance_eval { @iterations =3D random_iterations }
suite
end
undef_method:)default_test)
def teardown
if not passed?
puts("\nrandom_seed: #{$random_seed}")
throw:)stop_suite)
end
end
end
end
end


Here is that example I gave earlier using the above for random
testing Array#push and Array#pop using AOP:


class ArrayAOP < Array
include Test::Unit::Assertions
def push(*args)
print("#{self.inspect}.push(*#{args.inspect}) -> ") if=20
$debug_level>=3D1
n =3D size
na =3D args.size
ret =3D super
assert_equal(n+na,size)
assert_equal(slice(-na,na),args)
assert_same(self,ret)
p(ret) if $debug_level>=3D1
ret
end
def pop
print("#{self.inspect}.pop -> ") if $debug_level>=3D1
n =3D size
ret0 =3D n.nonzero? ? slice(-1) : nil
ret =3D super
assert_equal(ret0,ret)
assert_equal(n.nonzero? ? n-1 : 0,size)
p(ret) if $debug_level>=3D1
ret
end
end

class ArrayTest < Test::Unit::RandomTestCase
def self.suite
@@object =3D ArrayAOP[]
super
end
def test_push
args =3D []
rand(4).times {
args << [0,"",[],nil,false][rand(5)]
}
@@object.push(*args)
end
def test_pop
rand(4).times {
@@object.pop
}
end
end


As you can see from above, with this RandomTestCase class, you
can do random testing very easily. But, it still would be nice
to get options from the command line rather than the
environment as I'm doing in RandomTestCase.

--- Eivind Eklund said:
=20
=20
I cannot remember having used random testing for Ruby code,
though I've used
it for code in other languages. Having a nice framework for
it would be,
well, nice.
=20
For all of these strategies, I've had to hack up Test::Unit
in
=20
=20
Running N times sounds like a subclass of TestCase would be
appropriate.
Otherwise, how do you distinguish between what tests should
be run once and
which should be run several time (for statistical coverage)?
=20
Random order sounds reasonable for everything, anyway.
=20
=20
+ abort testing when the first failure is reached.
=20
=20
Also subclass material.
=20
+ easy way to skip the rest of the current test (when the
=20
=20
Also subclass material, I think.
=20
+ way to pass number of passes of the test_* methods on the
=20
=20
My gut feeling is that this is inappropriate. I see a major
part of the
Test::Unit style testing as the fact that the programmer only
has to check
for pass/fail. Passing through verbiosity information seems
to turn this on
its head, encouraging setting up the output for inspection.
(On the other
hand, I have sometimes needed information for debugging -
calling this
"debug level" would do much to ease my concerns...)
=20
+ way to pass command-line options down to the test suite to
=20
=20
I also get an initial bad feeling about this, as it makes it
totally
necessary to use the console-based testrunner.
=20
Eivind.



=09
__________________________________=20
Yahoo! Mail - PC Magazine Editors' Choice 2005=20
http://mail.yahoo.com
 
J

Jeff Wood

------=_Part_3883_22504389.1127854946434
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Why do you need DEBUG LEVEL or whatever when the Ruby
environment/interpreter already has $VERBOSE ... ???

I mean, it's already there ... just turn up the verbosity ...

ruby -v mytestcases.rb

... and your done ... ( am i missing something? )

j.

I gave another shot at making a derived TestCase class. I only
needed one hack to effectively make it ignore this class when
running tests: undef_method:)default_test). Here is what I
came up with that has many of the features I wanted:

require 'test/unit'

module Test
module Unit
class RandomTestCase < TestCase
def self.suite
$debug_level =3D (ENV['DEBUG_LEVEL']||0).to_i if
$debug_level.nil?
if $random_seed.nil?
$random_seed =3D ENV['RAND_SEED'].to_i.nonzero? ||
(srand;srand)
srand($random_seed)
puts("random_seed: #{$random_seed}") if
$debug_level>=3D1
end
random_iterations =3D (ENV['RAND_ITER']||8).to_f
methods =3D Regexp.new(ENV['TEST_METHODS']||".*")
suite =3D super
tests =3D suite.tests
tests.reject! { |t|
!methods.match(t.name.gsub(/\Atest_/,''))
}
(class << tests;self;end).class_eval {
def each
catch:)stop_suite) {
(@iterations*size).to_i.times {
catch:)invalid_test) {
yield(slice(rand(size)))
}
}
}
end
}
tests.instance_eval { @iterations =3D random_iterations }
suite
end
undef_method:)default_test)
def teardown
if not passed?
puts("\nrandom_seed: #{$random_seed}")
throw:)stop_suite)
end
end
end
end
end


Here is that example I gave earlier using the above for random
testing Array#push and Array#pop using AOP:


class ArrayAOP < Array
include Test::Unit::Assertions
def push(*args)
print("#{self.inspect}.push(*#{args.inspect}) -> ") if
$debug_level>=3D1
n =3D size
na =3D args.size
ret =3D super
assert_equal(n+na,size)
assert_equal(slice(-na,na),args)
assert_same(self,ret)
p(ret) if $debug_level>=3D1
ret
end
def pop
print("#{self.inspect}.pop -> ") if $debug_level>=3D1
n =3D size
ret0 =3D n.nonzero? ? slice(-1) : nil
ret =3D super
assert_equal(ret0,ret)
assert_equal(n.nonzero? ? n-1 : 0,size)
p(ret) if $debug_level>=3D1
ret
end
end

class ArrayTest < Test::Unit::RandomTestCase
def self.suite
@@object =3D ArrayAOP[]
super
end
def test_push
args =3D []
rand(4).times {
args << [0,"",[],nil,false][rand(5)]
}
@@object.push(*args)
end
def test_pop
rand(4).times {
@@object.pop
}
end
end


As you can see from above, with this RandomTestCase class, you
can do random testing very easily. But, it still would be nice
to get options from the command line rather than the
environment as I'm doing in RandomTestCase.

--- Eivind Eklund said:
I cannot remember having used random testing for Ruby code,
though I've used
it for code in other languages. Having a nice framework for
it would be,
well, nice.

For all of these strategies, I've had to hack up Test::Unit
in


Running N times sounds like a subclass of TestCase would be
appropriate.
Otherwise, how do you distinguish between what tests should
be run once and
which should be run several time (for statistical coverage)?

Random order sounds reasonable for everything, anyway.


+ abort testing when the first failure is reached.


Also subclass material.

+ easy way to skip the rest of the current test (when the


Also subclass material, I think.

+ way to pass number of passes of the test_* methods on the


My gut feeling is that this is inappropriate. I see a major
part of the
Test::Unit style testing as the fact that the programmer only
has to check
for pass/fail. Passing through verbiosity information seems
to turn this on
its head, encouraging setting up the output for inspection.
(On the other
hand, I have sometimes needed information for debugging -
calling this
"debug level" would do much to ease my concerns...)

+ way to pass command-line options down to the test suite to


I also get an initial bad feeling about this, as it makes it
totally
necessary to use the console-based testrunner.

Eivind.




__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com


--
"http://ruby-lang.org -- do you ruby?"

Jeff Wood

------=_Part_3883_22504389.1127854946434--
 
E

Eric Mahurin

Yes, there are several ruby switches to control
warnings/debugging/verbosity:

-d set debugging flags (set $DEBUG to true)
-v print version number, then turn on verbose
mode
-w turn warnings on for your script
-W[level] set warning level; 0=3Dsilence, 1=3Dmedium,
2=3Dverbose (default)

In addition to setting some global variables ($VERBOSE, $DEBUG,
$-v, $-w, $-d), these turn on warnings that some ruby code may
not like (produce excess warnings). -d probably also turns on
some internal ruby debugging, but I don't know where.

In addition to this, the autorunner in test/unit has
--verbose[=3Dlevel]. I would like to reuse this, but it is not
very easy to access (can get it through
ObjectSpace.each_object).

The TEST_METHODS below isn't needed as test/unit/autorunner
already has a --name option which works fine for this.

--- Jeff Wood said:
Why do you need DEBUG LEVEL or whatever when the Ruby
environment/interpreter already has $VERBOSE ... ???
=20
I mean, it's already there ... just turn up the verbosity ...
=20
ruby -v mytestcases.rb
=20
... and your done ... ( am i missing something? )
=20
j.
=20
I gave another shot at making a derived TestCase class. I only
needed one hack to effectively make it ignore this class when
running tests: undef_method:)default_test). Here is what I
came up with that has many of the features I wanted:

require 'test/unit'

module Test
module Unit
class RandomTestCase < TestCase
def self.suite
$debug_level =3D (ENV['DEBUG_LEVEL']||0).to_i if
$debug_level.nil?
if $random_seed.nil?
$random_seed =3D ENV['RAND_SEED'].to_i.nonzero? ||
(srand;srand)
srand($random_seed)
puts("random_seed: #{$random_seed}") if
$debug_level>=3D1
end
random_iterations =3D (ENV['RAND_ITER']||8).to_f
methods =3D Regexp.new(ENV['TEST_METHODS']||".*")
suite =3D super
tests =3D suite.tests
tests.reject! { |t|
!methods.match(t.name.gsub(/\Atest_/,''))
}
(class << tests;self;end).class_eval {
def each
catch:)stop_suite) {
(@iterations*size).to_i.times {
catch:)invalid_test) {
yield(slice(rand(size)))
}
}
}
end
}
tests.instance_eval { @iterations =3D random_iterations }
suite
end
undef_method:)default_test)
def teardown
if not passed?
puts("\nrandom_seed: #{$random_seed}")
throw:)stop_suite)
end
end
end
end
end


Here is that example I gave earlier using the above for random
testing Array#push and Array#pop using AOP:


class ArrayAOP < Array
include Test::Unit::Assertions
def push(*args)
print("#{self.inspect}.push(*#{args.inspect}) -> ") if
$debug_level>=3D1
n =3D size
na =3D args.size
ret =3D super
assert_equal(n+na,size)
assert_equal(slice(-na,na),args)
assert_same(self,ret)
p(ret) if $debug_level>=3D1
ret
end
def pop
print("#{self.inspect}.pop -> ") if $debug_level>=3D1
n =3D size
ret0 =3D n.nonzero? ? slice(-1) : nil
ret =3D super
assert_equal(ret0,ret)
assert_equal(n.nonzero? ? n-1 : 0,size)
p(ret) if $debug_level>=3D1
ret
end
end

class ArrayTest < Test::Unit::RandomTestCase
def self.suite
@@object =3D ArrayAOP[]
super
end
def test_push
args =3D []
rand(4).times {
args << [0,"",[],nil,false][rand(5)]
}
@@object.push(*args)
end
def test_pop
rand(4).times {
@@object.pop
}
end
end


As you can see from above, with this RandomTestCase class, you
can do random testing very easily. But, it still would be nice
to get options from the command line rather than the
environment as I'm doing in RandomTestCase.

--- Eivind Eklund said:
Does anybody else do random testing on their ruby code
besides
me? I picked it up because of my background (IC design)
where
random testing is one of the verification techniques of
hardware.


I cannot remember having used random testing for Ruby code,
though I've used
it for code in other languages. Having a nice framework for
it would be,
well, nice.

For all of these strategies, I've had to hack up Test::Unit
in
the same way to add various features. Here are the things
I've
added:

+ instead of running the test_* methods in a fixed order
once,
run them in a random order for N passes.


Running N times sounds like a subclass of TestCase would be
appropriate.
Otherwise, how do you distinguish between what tests should
be run once and
which should be run several time (for statistical coverage)?

Random order sounds reasonable for everything, anyway.


+ abort testing when the first failure is reached.


Also subclass material.

+ easy way to skip the rest of the current test (when the
random input is invalid). Don't include this in the number
of
tests or mark it as skipped.


Also subclass material, I think.

+ way to pass number of passes of the test_* methods on the
command line.

+ display random seed and way of passing it in on the
command
line.

+ test suite should have access to the --verbose level or
another switch to control the debugging verbosity.


My gut feeling is that this is inappropriate. I see a major
part of the
Test::Unit style testing as the fact that the programmer only
has to check
for pass/fail. Passing through verbiosity information seems
to turn this on
its head, encouraging setting up the output for inspection.
(On the other
hand, I have sometimes needed information for debugging -
calling this
"debug level" would do much to ease my concerns...)

+ way to pass command-line options down to the test suite to
control various things - what methods to test, what
classes to test, various other flags.


I also get an initial bad feeling about this, as it makes it
totally
necessary to use the console-based testrunner.

Eivind.




__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com
=20
=20
--
"http://ruby-lang.org -- do you ruby?"
=20
Jeff Wood
=20


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around=20
http://mail.yahoo.com=20
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,904
Latest member
HealthyVisionsCBDPrice

Latest Threads

Top