Test::Unit run order

[Dylan]

Hey there! I had a question about the order tests are run in
Test::Unit. I have a class with about 10 tests and I'd like to be able
to better control the order they run in. Right now they just run
alphabetically, and I could just add prefixes to my test names, but
I'm likely not going to be the one updating this code in the future
and I'd like to keep it as straightforward and simple as possible
(i.e. not end up with test_admintests_part1_login as a test name). I'm
keeping them all in one class because I'm using CI_reporter to
generate the XML results and I'd like it to be in one file. Is there
any way to change the order individual tests run, or is there a way to
have CI_reporter create only 1 XML file for multiple test suites. I've
looked around quite a bit for answers to either of these questions but
have so far found nothing (although it's a distinct possibility that I
just totally missed it an need a smack on the head). Thank you very
much!

-Dylan

 

[James Coglan]

[Note: parts of this message were removed to make it a legal post.]

Hey there! I had a question about the order tests are run in
Test::Unit. I have a class with about 10 tests and I'd like to be able
to better control the order they run in. Right now they just run
alphabetically, and I could just add prefixes to my test names, but
I'm likely not going to be the one updating this code in the future
and I'd like to keep it as straightforward and simple as possible
(i.e. not end up with test_admintests_part1_login as a test name). I'm
keeping them all in one class because I'm using CI_reporter to
generate the XML results and I'd like it to be in one file. Is there
any way to change the order individual tests run, or is there a way to
have CI_reporter create only 1 XML file for multiple test suites. I've
looked around quite a bit for answers to either of these questions but
have so far found nothing (although it's a distinct possibility that I
just totally missed it an need a smack on the head). Thank you very
much!

I can't speak about your XML stuff, but I'm pretty sure the
Test::Unit::TestCase.suite method sorts the tests by name so you can't
change the order. Your options are:

1) Construct the test suite yourself rather than relying on the above method
to do it for you. You'll have to dig around in Test::Unit's source to figure
out how to run tests without using AutoRunner.

2) Take a look at some of the code in test/unit/ui and look at the logging
events that tests fire. These should allow you to collect event data and
reorder it however you like before creating a report.

Both will require some inspecting of Test::Unit's source, but I'm sure
others have better suggestions. Test::Unit's README is pretty good if you
need to know some details on its internals, e.g. how TestCases are converted
to suites and run.

 

[Daniel Berger]

-----Original Message-----

From: Dylan [mailto:[email protected]]
Sent: Wednesday, July 15, 2009 11:15 AM
To: ruby-talk ML
Subject: Test::Unit run order

Hey there! I had a question about the order tests are run in
Test::Unit. I have a class with about 10 tests and I'd like to be able
to better control the order they run in. Right now they just run
alphabetically, and I could just add prefixes to my test names, but
I'm likely not going to be the one updating this code in the future
and I'd like to keep it as straightforward and simple as possible
(i.e. not end up with test_admintests_part1_login as a test name). I'm
keeping them all in one class because I'm using CI_reporter to
generate the XML results and I'd like it to be in one file. Is there
any way to change the order individual tests run, or is there a way to
have CI_reporter create only 1 XML file for multiple test suites. I've
looked around quite a bit for answers to either of these questions but
have so far found nothing (although it's a distinct possibility that I
just totally missed it an need a smack on the head). Thank you very
much!

I submitted a feature request for Test::Unix 2.x for something along these
lines and it was accepted.

http://rubyforge.org/tracker/index.php?func=detail&aid=26627&group_id=5650&a
tid=21859

In short, Kouhei added the '--order' option which lets you run tests in the
order in which the methods are defined within the test suite rather than
alphabetically.

It hasn't been released yet, but you can grab it from CVS (or wait until it
is released, though I'm not sure when that will be).

Regards,

Dan

 

[Ryan Davis]

Hey there! I had a question about the order tests are run in
Test::Unit. I have a class with about 10 tests and I'd like to be able
to better control the order they run in. Right now they just run
alphabetically [...]

order dependent tests are a bug. every single test should be able to
run and pass in isolation. eg:

% ruby -Ilib test/test_something.rb -n test_one_test

should always pass no matter what the file or test name is.

 

[Dylan]

Hey there! I had a question about the order tests are run in
Test::Unit. I have a class with about 10 tests and I'd like to be able
to better control the order they run in. Right now they just run
alphabetically [...]

order dependent tests are a bug. every single test should be able to  
run and pass in isolation. eg:

% ruby -Ilib test/test_something.rb -n test_one_test

should always pass no matter what the file or test name is.

Which is nice in theory, but not always the best option. In my case,
I'm testing a web app, and I need to test different things in
different logins, so I'd like to order it so I log in to a normal user
account, run some tests, then log in to an admin account, run some
other tests, etc... without having to re-login for every test, which
wastes a lot of time and doesn't give me anything in return other than
being able to say "Ooh, my tests are independent" (I already have a
separate test for the login system). If the login system doesn't work,
none of the tests are going to pass even if I had separate login
scripts for each one.

So I either need a way to order the tests so I can use certain logins
for certain tests, or I need a way to create only 1 XML file from
CI_reporter from multiple suites, then I can just set up 1 suite for
each set of tests (this would be preferred, although a way to order
the tests would still be helpful, at least as a placeholder to use
during test development).

@Daniel: Thanks I will check that out.

-Dylan

 

[Kouhei Sutou]

Hi,

I submitted a feature request for Test::Unix 2.x for something along these
lines and it was accepted.

http://rubyforge.org/tracker/index.php?func=detail&aid=26627&group_id=5650&a
tid=21859

In short, Kouhei added the '--order' option which lets you run tests in the
order in which the methods are defined within the test suite rather than
alphabetically.

It hasn't been released yet, but you can grab it from CVS (or wait until it
is released, though I'm not sure when that will be).

I'll release a new version of test-unit in RubyKaigi2009.
(I'll release it on today or tomorrow.)

 

[Kouhei Sutou]

Hi,

order dependent tests are a bug. every single test should be able to run and
pass in isolation. eg:

% ruby -Ilib test/test_something.rb -n test_one_test

should always pass no matter what the file or test name is.

I think so too. So I've also added "--order=random" option.


Thanks,

 

[Ryan Davis]

Hey there! I had a question about the order tests are run in
Test::Unit. I have a class with about 10 tests and I'd like to be
able
to better control the order they run in. Right now they just run
alphabetically [...]

order dependent tests are a bug. every single test should be able to
run and pass in isolation. eg:

% ruby -Ilib test/test_something.rb -n test_one_test

should always pass no matter what the file or test name is.

Which is nice in theory, but not always the best option. In my case,
I'm testing a web app, and I need to test different things in
different logins, so I'd like to order it so I log in to a normal user
account, run some tests, then log in to an admin account, run some
other tests, etc... without having to re-login for every test, which
wastes a lot of time and doesn't give me anything in return other than
being able to say "Ooh, my tests are independent" (I already have a
separate test for the login system). If the login system doesn't work,
none of the tests are going to pass even if I had separate login
scripts for each one.

So I either need a way to order the tests so I can use certain logins
for certain tests, or I need a way to create only 1 XML file from
CI_reporter from multiple suites [...]

...or you could deal with the real problem of having slow tests and
not the symptoms of it. I've talked about this a lot in the past[*],
so I'm not going to repeat myself here. I'll simply reassert:

_all_ tests should _always_ pass when run by themselves.

and I'll expand on that:

_all_ tests should run fast enough that they're a pleasure to run
_all_ the time.

[*] See my talks on Ruby Sadism, Electric Kool-ade Acid Testing, and
many others for related infos.

 

[Dylan]

On Jul 15, 2009, at 10:15 , Dylan wrote:
Hey there! I had a question about the order tests are run in
Test::Unit. I have a class with about 10 tests and I'd like to be  
able
to better control the order they run in. Right now they just run
alphabetically [...]
order dependent tests are a bug. every single test should be able to
run and pass in isolation. eg:
% ruby -Ilib test/test_something.rb -n test_one_test
should always pass no matter what the file or test name is.

Which is nice in theory, but not always the best option. In my case,
I'm testing a web app, and I need to test different things in
different logins, so I'd like to order it so I log in to a normal user
account, run some tests, then log in to an admin account, run some
other tests, etc... without having to re-login for every test, which
wastes a lot of time and doesn't give me anything in return other than
being able to say "Ooh, my tests are independent" (I already have a
separate test for the login system). If the login system doesn't work,
none of the tests are going to pass even if I had separate login
scripts for each one.

So I either need a way to order the tests so I can use certain logins
for certain tests, or I need a way to create only 1 XML file from
CI_reporter from multiple suites [...]

..or you could deal with the real problem of having slow tests and  
not the symptoms of it. I've talked about this a lot in the past[*],  
so I'm not going to repeat myself here. I'll simply reassert:

   _all_ tests should _always_ pass when run by themselves.

and I'll expand on that:

   _all_ tests should run fast enough that they're a pleasure to run  
_all_ the time.

[*] See my talks on Ruby Sadism, Electric Kool-ade Acid Testing, and  
many others for related infos.

I would love to do that, but I have no idea how. The "real problem" of
my slow tests is caused by the very real requirement of logging in
every time, were they to be truly independent. I'm not in charge of
the bandwidth to the server, I'm not in charge of the log-in code, and
I'm not in charge of the "Lets make the tests take less than a day to
run" requirement. I'm testing a web app, and web apps by their very
nature have to transmit data back and forth, which takes more time
than any local machine doing operations. In fact, the "slow" tests
aren't all that slow were it a real user in a real situation. Running
so many tests takes long enough as it is, and I still don't see the
benefit of independence in my case (even so, I would still prefer it,
as I've said, but in my situation the tradeoffs do not seem to be
worth it. Nothing further gets tested, they run just the same as if I
logged in every time unless there is some major screw up with cookies/
cache, and the other tests aren't going to screw up because of a
previous one (or on the off-chance they do it would be easily
traceable back to the actual problem). Yes, I do realize that there is
a chance that not re-logging in every time might screw up some test,
but the chance of it is remote enough, and the problems that would
occur if it did happen are small enough, that the added benefit of
being able to get results back relatively quickly is (to me) worth it.

I am in fact trying to solve your second rule by removing the first.
Were I to make them truly independent it would take well over 4-5
times as long to run, which means no one would bother running them/
waiting around for their results, which makes them useless. I would
welcome a solution that could fulfill both your rules there, but until
I find one I have to err on the side of usability.

-Dylan

 

[Rick DeNatale]

I would love to do that, but I have no idea how. The "real problem" of
my slow tests is caused by the very real requirement of logging in
every time, were they to be truly independent. I'm not in charge of
the bandwidth to the server, I'm not in charge of the log-in code, and
I'm not in charge of the "Lets make the tests take less than a day to
run" requirement. I'm testing a web app, and web apps by their very
nature have to transmit data back and forth, which takes more time
than any local machine doing operations.

It sounds to me that there's a confusion here between unit-testing and
full-stack testing.

For example if this were a rails app, the normal approach is to have
tests for models, separate tests for controllers, and integration
tests to test user interface flows.

And even in the last case, this is normally done not by transmitting
data back and forth, instead Rails provides special integration test
mocks for the http infrastructure which simulates actual http request
and response transmission.

And logging in (i.e. authorization) is one thing which is usually
mocked out for many if not all tests. Jim Weirich does a humorous
'skit' about mocking login, in which he has to help a co-worker who
was testing failed password validation using real access and his own
account, and kept getting locked out of the system.

Less traditional ruby web developers are using tools like Cucumber and
webrat to do integration/acceptance testing, but these are still
simulating infrastructure in different ways.

--
Rick DeNatale

Blog: http://talklikeaduck.denhaven2.com/
Twitter: http://twitter.com/RickDeNatale
WWR: http://www.workingwithrails.com/person/9021-rick-denatale
LinkedIn: http://www.linkedin.com/in/rickdenatale