Ruby Test Frameworks

[Michael Furmaniuk]

I'm looking for a Ruby Framework that I can use to aid in testing a
filter driver, we currently use a Perl based set of scripts that has
some wrapping and does system events for files, directories and
registry changes (it's multi-platform for Windows and *nixes).

Ideally, I want to start capturing data in a database and give a nice
simple framework around it, which is what led me to Rails and from that
to Ruby - and I thought being able to make our Harness more OO to allow
for more reuse and remove duplication (which there is a lot of in our
scripts). Plus one language to handle test results and the test harness
seemed more efficient. Most of the Ruby Test Frameworks that I have
looked through are all Web or GUI test frameworks, Test::Unit is nice
but I would like to know if there is a Framework for application and
system testing that I have missed. Has anyone used a framework for
application and system testing they could recommend, or even an existing
one that they have used successfully?

Some of my initial requirements are:
- Runs on Windows and Linux
- Supports calling external code/scripts, we have Java and C code that
is called for memory utilization and socket testing
- Can generate system events for create/modify/delete of files and
directories
- Has some hooks into Windows Registry, IIS and Linux RPM (Nice to
haves but not required)
- Can integrate with an IDE (Visual Studio would be nice)

Any help is appreciated.

Thanks!

 

[Brian Candler]

Most of the Ruby Test Frameworks that I have
looked through are all Web or GUI test frameworks, Test::Unit is nice
but I would like to know if there is a Framework for application and
system testing that I have missed. Has anyone used a framework for
application and system testing they could recommend, or even an existing
one that they have used successfully?

I don't know of anything specific to 'application and system testing'.
If I were you, I'd start with Test::Unit or its bigger brothers Shoulda
or Rspec, and see if it does what you actually need.

Some of my initial requirements are:
- Runs on Windows and Linux
- Supports calling external code/scripts, we have Java and C code that
is called for memory utilization and socket testing

You can call external programs using backticks, system() or IO.popen:
e.g.

def test_foo
result = `foo`
assert_equal 0, $? # exit code
assert_match /success/, result # output on stdout
end

You can call actual C using RubyInline or FFI.

- Can generate system events for create/modify/delete of files and
directories

You can do those actions using FileUtils. Or do you mean you want to be
*notified* of those sorts of events happening in the filesystem? I'm not
sure about that.

- Has some hooks into Windows Registry, IIS and Linux RPM (Nice to
haves but not required)

Not sure, I believe there is Win32 API (but I'm a Linux person).

- Can integrate with an IDE (Visual Studio would be nice)

I think Test::Unit has some graphical runners (but I'm a CLI person

 

[Richard Conroy]

I don't know of anything specific to 'application and system testing'.
If I were you, I'd start with Test::Unit or its bigger brothers Shoulda
or Rspec, and see if it does what you actually need.

For the simple stuff, dust is my tool of choice. Its a very simple
wrapper around
Test::Unit, so there is a shallow learning curve, but you get
meaningful test names,
and it is very easy to generate large blocks of tests from arrays of
inputs. Handy
for black box testing, testing regexes etc.

Shoulda and RSpec I think are tackling different needs. The learning
curve is greater
and there is more magic involved, but I suspect that test
organisation, and consolidation
of reporting is probably a greater priority than natural language test
description.

You can call external programs using backticks, system() or IO.popen:
e.g.

def test_foo
result = `foo`
assert_equal 0, $? # exit code
assert_match /success/, result # output on stdout
end

You can call actual C using RubyInline or FFI.

In addition, check out JtestR for plain old Java testing (it works great).

You can do those actions using FileUtils. Or do you mean you want to be
*notified* of those sorts of events happening in the filesystem? I'm not
sure about that.


Not sure, I believe there is Win32 API (but I'm a Linux person).

Theres a team integrating Ruby into visual studio, I can't recall offhand.

I think Test::Unit has some graphical runners (but I'm a CLI person

I haven't found a good solution for out of the box test reporting in Ruby.
RSpec has some great reporting, but I need something a bit more turnkey.

I currently use dust to test java code through JtestR on top of an existing
Java build system (so our test reporting is done through JUnit reports and
cobertura works for our code coverage of the java code).

 

[Brian Candler]

Shoulda and RSpec I think are tackling different needs. The learning
curve is greater
and there is more magic involved, but I suspect that test
organisation, and consolidation
of reporting is probably a greater priority than natural language test
description.

For Rspec I agree - I found it too contrived - but shoulda fitted my
needs much better. It gives you the meaningful test names, and also
setup/teardown contexts which avoids having to make a new class for
every set of fixtures. But you keep good old-fashioned assertions at the
bottom.

class Foo < Test::Unit::TestCase
context "Piping in FOO" do
setup do
File.delete("output.txt")
@res = `echo "FOO" | ./wibble`
end

should "generate output BAR" do
assert_match /BAR/, @res
end

should "create file output.txt" do
assert File.exist?("output.txt")
end

# etc
end
end

 

[Michael Furmaniuk]

Shoulda and RSpec I think are tackling different needs. The learning
curve is greater
and there is more magic involved, but I suspect that test
organisation, and consolidation
of reporting is probably a greater priority than natural language test
description.

Yes, natural language test description is nice, but if it comes at the
expense of reporting and organization I can deal with the odd
description. I had looked a little at RSpec but not in depth, and
Shoulda I saw referenced in a page somewhere but it did not seem a fit
for what I was looking for, I will check them more in depth.

In addition, check out JtestR for plain old Java testing (it works
great).

Thanks for these, and the other ideas, I will check them out!

 

[Yanik Magnan]

SSdtIGd1ZXNzaW5nIHByb2JhYmx5IHRoZSBJcm9uUnVieSBndXlzIGFyZSB0aGUgb25lcyBnZXR0
aW5nIFZpc3VhbApTdHVkaW8gd29ya2luZyB3aXRoIFJ1YnkuIDpQCgotLSBZYW5payBNYWduYW4g
4oCiIGh0dHA6Ly9yLWNoLm5ldAoKCgpPbiBUdWUsIERlYyA5LCAyMDA4IGF0IDM6NTEgUE0sIFJp
Y2hhcmQgQ29ucm95IDxyaWNoYXJkLmNvbnJveUBnbWFpbC5jb20+IHdyb3RlOgo+IFRoZXJlcyBh
IHRlYW0gaW50ZWdyYXRpbmcgUnVieSBpbnRvIHZpc3VhbCBzdHVkaW8sIEkgY2FuJ3QgcmVjYWxs
IG9mZmhhbmQuCg==