[ANN] testy.rb - ruby testing that's mad at the world

[Bil Kleb]

   . hi-jacking at_exit sucks ass

IIRC, PragDave suggested the at_exit trick to Talbott during
the elder days.

Later,

 

[Bil Kleb]

Just a thought: I've found myself doing that too, but in particular I've
ran an example externally to be sure it works, then ended up pasting it
into the rdoc. So it would be really cool if rdoc could integrate the
examples directly in the appropriate place(s). I want to be sure that my
examples actually run as advertised, and at the moment I risk them
becoming out of date, so I'm definitely with you on that point.

Sounds like you're pining for Python's doctest?

http://docs.python.org/library/doctest.html

Later,

 

[Brian Candler]

Sounds like you're pining for Python's doctest?

http://docs.python.org/library/doctest.html

Ah, now that's a really interesting way of thinking about
examples/testing: in the form of an irb session. You can write your
tests just by mucking about in irb, and when it makes sense, just paste
the output somewhere.

irb> foo = generate_foo
=> #<Foo:0xb7cd041c @attr1="hello", @attr2="world">
irb> foo.attr1
=> "hello"
irb> foo.attr2
=> "world"

Presumably you could avoid the fragile comparison on Object#inspect
output by deleting it from the transcript.

irb> foo = generate_foo
...
irb> foo.attr1
=> "hello"
irb> foo.attr2
=> "world"

Writing something which parses that an (re)runs it to verify the output
should be pretty straightforward.

It can also handle the 'assert_raises' case nicely.

irb> f.attr3
NoMethodError: undefined method `attr3' for ...

Ara, how about it?

 

[Michel Demazure]

Ah, now that's a really interesting way of thinking about
examples/testing: in the form of an irb session. You can write your
tests just by mucking about in irb, and when it makes sense, just paste
the output somewhere.

irb> foo = generate_foo
=> #<Foo:0xb7cd041c @attr1="hello", @attr2="world">
irb> foo.attr1
=> "hello"
irb> foo.attr2
=> "world"

This has been done : I found it quite some time ago on the Web and
polished it a bit :

#!/usr/bin/env ruby

# ---------------------------------------------------

# Copyright 2007 Clinton Forbes
# Modified Michel Demazure 11/07

# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with this program. If not, see http://www.gnu.org/licenses/.

# ---------------------------------------------------

# Put this in the script directory of your Rails app

# this script is inspired by the doctest feature in Python:
# (http://docs.python.org/lib/module-doctest.html)
# :although I never did get around to reading the original Python code.
#
# Feel free to improve upon this script, on the condition that you post
# a comment on
http://clintonforbes.blogspot.com/2007/08/doctest-for-ruby-and-rails.html
# and let me know about it so I can use it as well.
#
# to use, just put some doctests in your code and run this tool using
#
# script/doctest

# doctests look like this
=begin
#doctest Check that 1 + 1 = 2=> 6
=end

# or like this
#=begin
##doctest Test creating a Member
#>> m = Member.new()
#=> #nil, "name"=>nil, "hashed_password"=>nil, "admin"=>false,
"reset_key"=>nil, "newsletter_level"=>nil, "created_at"=>nil,
"email"=>nil}>
#>> m.name = "Clinton"
#=> "Clinton"
#>> m.admin = true
#=end

# or like this
=begin
#doctest Check that 1 + 2 = 3
irb(main):001:0> 1 + 2
=> 3
irb(main):002:0> 2 + 3
=> 5
=end

#get all of our Rails stuff
#require File.dirname(__FILE__) + '/../config/boot'
#require File.dirname(__FILE__) + '/../config/environment'

#build array of .rb files
#only looks in ./app directory by default. you can change this if you
keep
#extra code in other places (eg. ./vendor/plugins)

# CODE_REGEX caters for standard IRB prompt and Rails script/console
# prompt.
CODE_REGEX = Regexp.new(/(>>|irb.*?>) (.*)/)
RESULT_REGEX = Regexp.new(/=> (.*)/)

def get_ruby_files(dir_name)
ruby_file_names = []

Dir.foreach(dir_name) do |file_name|
unless file_name == '.' || file_name == '..'
full_name = File.join(dir_name, file_name)
if /.*\.rb$/ =~ full_name
ruby_file_names << full_name
elsif File.directory? full_name
sub_files = get_ruby_files(full_name)
ruby_file_names.concat(sub_files) unless sub_files.empty?
end
end
end

ruby_file_names
end

# When running tests, addresses of objects are never likely
# to be the same, so we wipe them out so tests don't fail
#
# for example: #
def normalize_result(input)
input.gsub(/:0x([a-f0-9]){8}/, ':0xXXXXXXXX')
end

def failure_report(statement, expected_result, result)
report = "\n FAILED" #add line number logic here
report << " Code: " << statement << "\n"
report << " Expected: " << expected_result << "\n"
report << " But got: " << result
end

def run_doc_tests(doc_test)
execution_context = binding()
statement, report = '', ''
wrong, passed = 0, 0
doc_test.split("\n").each do |line|
case line
when CODE_REGEX
statement << CODE_REGEX.match(line)[2]
when RESULT_REGEX
expected_result = normalize_result(RESULT_REGEX.match(line)[1])
result = normalize_result(eval(statement,
execution_context).inspect)
unless result == expected_result
report << failure_report(statement, expected_result, result)
wrong += 1
else
passed += 1
end
statement = ''
end
end
return passed, wrong, report
end

def process_ruby_file(file_name)
tests, succeeded, failed = 0, 0, 0
file_report = ''
code = File.read(file_name)
code.scan(/=begin\s#doctest ([^\n]*)\n(.*?)=end/m) do |doc_test|
file_report << "\n Testing '#{doc_test[0]}'..."
passed, wrong, report = run_doc_tests(doc_test[1])
file_report += (wrong == 0 ? "OK" : report)
tests += 1
succeeded += passed
failed += wrong
end
file_report = "Processing '#{file_name}'" + file_report unless
file_report.empty?
return tests, succeeded, failed, file_report
end

ruby_file_names = get_ruby_files(File.dirname(__FILE__))

total_report = "Looking for doctests in #{ruby_file_names.length}
files\n"
total_files, total_tests, total_succeeded, total_failed = 0, 0, 0, 0
ruby_file_names.each do |ruby_file_name|
tests, succeeded, failed, report = process_ruby_file(ruby_file_name)
total_files += 1 if tests > 0
total_tests += tests
total_succeeded += succeeded
total_failed += failed
total_report << report << "\n" unless report.empty?
end
total_report << "Total files: #{total_files}, total tests:
#{total_tests}, assertions succeeded: #{total_succeeded}, assertions
failed: #{total_failed}"
puts total_report

 

[ara.t.howard]

On Mar 30, 2009, at 2:12 AM, Brian Candler wrote:


( a LOT of good stuff to which i'll reply to selectively here)

Perhaps - but it's one rule that only needs to be learned once.

I notice that testy supports check <name>, <expected>, <actual> too.

Testy does (intentially) force you to name your tests, whereas
Test::Unit will happily let you write

check <expected>, <actual>

I really don't like having to name each assertion, maybe because I'm
lazy or maybe because it feels like DRY violation. I've already said
what I want to compare, why say it again?

hmmm. yeah i see that, but disagree that the effort isn't worth it
for the *next* programer.

Hmm, this is probably an argument *for* having a DSL for assertions
- to
make the assertions read as much like example code ("after running
this
example, you should see that A == B and C < D")

Neither

result.check "bar attribute", :expected => 123, :actual => res.bar

nor

assert_equal 123, res.bar, "bar attribute"

reads particularly well here, I think.

yeah i agree. i'm open to suggestion, just has to be very very simple.

Going too far this way down this path ends up with rspec, I think.

In fact, I don't really have a problem with writing

res.foo.should == 456

The trouble is the hundreds of arcane variations on this.

bingo! i really think the key is having *one* assertion method.

You solve this problem by only having a single test (Result#check),
and
indeed if rspec only had a single method (should_equal) that would be
fairly clean too. However this is going to lead to awkwardness when
you
want to test for something other than equality: e.g.

i dunno - ruby is pretty good at this

value = begin; object.call; rescue => e; e.class; end

result.check :error, SomeError, value

that seems perfectly fine to me. gives me an idea though - maybe
check should take a block

result.checkerror, SomeError){ something that raises an error }

and use the block to get the actual value as in

value = begin; block.call; rescue Object => e; e; end

result.check "foo should contain 'error'", foo, :=~, /error/

But again this is getting away from real ruby for the assertions, in
which case it isn't much better than

assert_match /error/, foo, "foo should contain 'error'"

assert_match /error/, foo # lazy/DRY version

check actually uses === for the comparison so you can do

result.check :instance_of, SomeClass, object

result.check :matches, /pattern/, string


i need to nail that down though.

Yes, parseable results and test management are extremely beneficial.
Those could be retro-fitted to Test::Unit though (or whatever its
replacement in ruby 1.9 is called)

Getting rid of the at_exit magic is also worth doing.

i actually thought of simply patching test/unit... but then there are
good test names, contexts, etc.

Nice, could perhaps show the (expected) result inline too?

well - here actual always === expected with my current impl. it's
essentially example code wrapped in assert_nothing_raised ;-)

I agree. Part of the problem is that when one thing is wrong making 20
tests fail, all with their respective backtraces, it can be very
hard to
see the wood for the trees. What would be nice would be a folding-type
display with perhaps one line for each failed assertion, and a [+] you
can click on to get the detail for that particular one.

funny you mention that as i also hate that. my first version of testy
actually just failed fast - if one test failed the code reported and
aborted. maybe i should consider going back to that? i am finding
that good output makes a ton of failures much easer - even using less
and searching is easier with testy that anything else.

I disagree there - not with the research, but the implied conclusion
that you should never use a large codebase. Shoulda works well, and
I've
not once found a bizarre behaviour in the testing framework itself
that
I've had to debug, so I trust it.

shoulda does work well - i stole it's context concept just yesterday ;-)

Yeah, but how many lines of Rails framework?

i used to be on the ramaze list alot too you know ;-)


a @ http://codeforpeople.com/

 

[Phrogz]

Perhaps - but it's one rule that only needs to be learned once.

I disagree. I 'learn' it each time I look at it, and then I forget it.
I think possibly because I think it's backwards.

The same goes for alias_method for me. I cannot tell you how many
times I've had to look up the order of old_name/new_name. And with
this, it's certainly because I think the values are backwards. (I just
had to look up the order to be sure.)

 

[Yossef Mendelssohn]

Quite a bit of discussion since I looked at this last. It's almost as
if people care about testing.

i'm open to suggestion on format though. =A0requirements are

=A0 =A0 =A0 =A0 . readable by humans
=A0 =A0 =A0 =A0 =A0. easily parsed by computers

basically that means some yaml format. =A0honestly open to suggestion =A0
here...

Agreed on YAML. That's just a really simple way to go if you're going
to stick to those two (very sensible) requirements. Maybe JSON would
work as well. I think the trouble I had with the specific example is
that it's listed as a failure, yet shows 'a' matching. Maybe instead
of

returning unexpected results:
failure:
expect:
a: 42
b: forty-two
actual:
a: 42
b: 42.0

you could have something like

returning unexpected results:
status: failure
vars:
matched:
- a
unmatched:
expect:
b: forty-two
actual:
b: 42.0

It doesn't have to be exactly like that, of course, but I'd call
attention to specifically the status being something you can easily
check by using the 'status' key instead of looking for 'failure'. Also
that the variables are split into good and bad.

The status thing, admittedly, is more of a point if you're going to
have YAML reports of tests that passed as well. One of the things I
like about RSpec and bacon is the specdoc format, so if you have
sensibly-named contexts and examples, you can get a document
explaining the behavior of the code under test. In fact, I added this
output format to shoulda in the pre-git hullabaloo. I don't think it
ever made it in.

i have major issues with points two and three wrst to most ruby =A0
testing frameworks. =A0one of the main points of testy is to combine =A0
examples with testing. =A0rspec and all the others do not serve as =A0
examples unless you are a ruby master. =A0that is to say they introduce = =A0
too many additions to the code that's supposed to be an example to =A0
really preserve it's 'exampleness'. =A0and of course the output is =A0
utterly useless to normal humans. =A0if a framework provides 1000 =A0
asset_xxxxxxxx methods ad nausea then the point of the code - it's =A0
level of example-good-ness - is lost to mere mortals

The examples I tend to give are just very high-level overviews, mostly
showing the API and drawing the reader into other documentation, the
specs, or finally the code if they really want to see everything my
lib/module/gem/whathaveyou can do. As such, they're usually not very
helpful as far as specs go.

I agree that having a glut of assertions/checks isn't useful, and is
fact hiding a very important fact when it comes to BDD (or really
testing in general): If something is painful to test, it should be
changed. What I love about BDD is having that something using my code
and defining its API as I go, so I don't get stuck in nasty test-land
with shitty implementation-specific tests, or at least not as easily.
So that pain, like any pain, is a signal that something is wrong, and
it should be fixed. It shouldn't be hidden behind a special assertion
or matcher or any other rug-sweeping activity unless there's good
reason, like the code that's hurting you to test isn't available to
you to fix.

this will summarize where my thoughts are on that

cfp:~/redfission > find vendor/gems/{faker,mocha,thoughtbot}* -type f|
xargs -n1 cat|wc -l
=A0 =A0 24255

cfp:~/redfission > find app -type f|xargs -n1 cat|wc -l
=A0 =A0 =A01828

rspec and co might be fine but seriously, the above is insane right?

It's bordering on nuts. It'd be better to show LoC than just wc -l, of
course, and are all the thoughtbot gems for your tests? Also, someone
else mentioned flay/flog/reek output, which could be illuminating. And
someone also brought up the app framework. After all, what you're
doing here is comparing your app code to your test framework. Seems
like you should be comparing app to test, or app framework to test
framework, or app to test with their respective frameworks. (And don't
forget to include Test::Unit for the testing frameworks that are
simply a layer on top of that.)

And I'd like to once again bring up bacon. I say I live with a few
methods on Object and Kernel to be able to test without going crazy,
and I mean it. I don't care how much how much you say assert_equal
makes sense or result.check 'some name', :expect =3D> x, :actual =3D> y is
great, I'm never going to get confused about some_value.should =3D=3D 5.
It's readable, and it's close to English. Where I think bacon wins out
huge over RSpec is that bacon keeps all those methods on the Should
object instead of making them global. It's a small change to syntax
(some_obj.should.respond_tometh) vs. some_obj.should respond_to
meth)), but it makes a world of difference as far as sensibility
goes.

Oh, and bacon is < 300 LoC, with facon at 365 (a line of code for
every day in the year!).

 

[ara.t.howard]

I disagree. I 'learn' it each time I look at it, and then I forget it.
I think possibly because I think it's backwards.

The same goes for alias_method for me. I cannot tell you how many
times I've had to look up the order of old_name/new_name. And with
this, it's certainly because I think the values are backwards. (I just
had to look up the order to be sure.)

100% agree on both counts. i personally would always use the options
approach expected, :actual =>) because of that.

a @ http://codeforpeople.com/

 

[The Higgs bozo]

100% agree on both counts. i personally would always use the options
approach expected, :actual =>) because of that.

Ha, and I thought it was only me with alias_method and assert_equal
dyslexia. With alias_method, I find myself expecting it to act like
'cp' or 'ln', and I have to keep remembering it's the opposite of that,
even after a hundred times of re-remembering. And assert_equal order is
the opposite to the usual order in English, which I guess is a selling
point of rspec.

Along the same lines, I always ask irb about methods(true) vs
methods(false). I'm like, OK I just want what's recently defined --
true? Doh. OK then, false, I only want the recent definitions. False
-- do not include super. See? I am even forgetting, right now, as I
type this, which is which.

"I curse you in ways multifarious!"
"OK, will that be Curses.methods(true) or Curses.methods(false)?"
"ARGGH!"

 

[Brian Candler]

shoulda does work well - i stole it's context concept just yesterday ;-)

Looks like you're not the only one:
http://github.com/citrusbyte/contest

A nice thing about tiny test frameworks like this (and yours) is that
they are easily vendorized, so the app is no longer dependent on a
specific test framework and/or version being available.

 

[Joel VanderWerf]

Who was just fussing about "too many lines of code in the framework"?


Right:

report = journal do
my_code()
end

Inside journal{}, everything the Ruby VM does gets packed into a report.
Then you rip the report to get back to your details. That's what assert{
2.0 } _can't_ do, so it goes in two passes.

The first pass is a normal block.call, to detect success or failure.
This is so any bugs in the second pass don't throw the test run.

The second pass puts the block into Ripper to decompile it, and then the
fun starts.

So you have to be careful in the block to avoid side effects, right?
Otherwise pass1 and pass2 might start with different state.

 

[Phlip]

So you have to be careful in the block to avoid side effects, right?
Otherwise pass1 and pass2 might start with different state.

No, I just document: Don't put anything in the block with a side-effect. The
money-line (the line that calls your production code to test it), for example,
should never be inside the assert{}.

The first pass is just a raw block.call, so the assertion must always get its
actual status right...