Embedded vs. Non-embedded Tests

T

Trans

As far as I know, Facets is the only large project that uses embedded
unit tests. The tests are housed in block comments, eg.

=begin test
...
=end

I have a command line tool that can run the embedded test directly.
For deployment, I have another tool that extracts all the embedded
tests and copies them to stand-alone test/ files so others can run
them. It works fairly well. And the advantage of course is that the
unit tests are right there with the libs they test.

However, considering that just about everyone else is putting units
test in their on test files, I'm wondering if I actually missing out
on superior advantages to that approach. If not why haven't others
pursued better tools for supporting embedded tests?

Thanks,
T.
 
R

Rick DeNatale

As far as I know, Facets is the only large project that uses embedded
unit tests. The tests are housed in block comments, eg.

=begin test
...
=end

I have a command line tool that can run the embedded test directly.
For deployment, I have another tool that extracts all the embedded
tests and copies them to stand-alone test/ files so others can run
them. It works fairly well. And the advantage of course is that the
unit tests are right there with the libs they test.

However, considering that just about everyone else is putting units
test in their on test files, I'm wondering if I actually missing out
on superior advantages to that approach. If not why haven't others
pursued better tools for supporting embedded tests?

I think that you're swimming against the tide.

Personally, it just feels right to me to separate the test code from
the code under test, most test equipment belongs in the garage instead
of the trunk of the car.

And when it gets beyond just unit testing individual classes, it's not
apparent to me which file an embedded test should call home.

From a practical tool perspective, there are lots of things which
expect a relationship between code files and test files. Two which
spring immediately to mind are autotest and rcov.
 
R

Rick DeNatale

A lot of diagnostics now takes place in modern cars, however. Tire pressure
checks, engine wear, oil, etc. Sure, you'd only work and make changes to the
car in the garage, but a lot of the "testing" is a hidden ongoing day to day
process.

For the most part, that's instrumentation, not testing.

In the case where sensor information is used to affect the operation
of the vehicle, such as
engine managment, abs, or traction control, it's no more testing
equipment than carburettors, brakes, and master cylinders. Yes, data
gets logged, but that's not testing, any more than a server log file
is.
And if you get onto, say, Formula 1 cars.. some "repairs" can even take
place on the road and the telemetry provides almost as much info to the
engineers while the car is running out on the track as they can get in the
garage. I've always wondered if/how to make software as easy to diagnose
with real-time telemetry and the ability to make changes to a running
program's parameters..

Yes, but most the testing equipment is not in the car, the engineers
in the pits have laptops, and the heavy stuff is in places like Woking
or Maranello, connected through the 'internets.'
 
T

Trans

I think that you're swimming against the tide.

Personally, it just feels right to me to separate the test code from
the code under test, most test equipment belongs in the garage instead
of the trunk of the car.

Interesting analogy. We do put some gages right on the dash. But
that's more like embedding the tests directly into code --eg. rescue
and try clauses. So maybe you're analogy is a good one. I can see some
reasons for doing so, it certainly make it easier to send some one a
test, that they can try against their own implementation, for
instance. But then I start to wonder about API documentation too.
Should that also be separated out into dedicated files? Why tests but
not docs?
And when it gets beyond just unit testing individual classes, it's not
apparent to me which file an embedded test should call home.

That's true. Embedded tests only apply for unit test. Integration
tests still need to be is separate files anyway.
From a practical tool perspective, there are lots of things which
expect a relationship between code files and test files. Two which
spring immediately to mind are autotest and rcov.

That's part of what I was wondering. I've never tried these tools. I
guess I've always suspected they'd be too magical to introduce into an
already established project, and thus lead to more trouble than
they're worth --though I'm sure they're quite useful if one starts
using them from the get-go. Am I wrong about that? But even so,
couldn't such tools be adapted to handle embedded tests too?

T.
 
R

Rick DeNatale

Interesting analogy. We do put some gages right on the dash. But
that's more like embedding the tests directly into code --eg. rescue
and try clauses. So maybe you're analogy is a good one. I can see some
reasons for doing so, it certainly make it easier to send some one a
test, that they can try against their own implementation, for
instance. But then I start to wonder about API documentation too.
Should that also be separated out into dedicated files? Why tests but
not docs?

Well , it probably seems more obvious to most to put the doc with the
code (assuming that there is any doc <G>). Note that we've just now
seemed to get tools like dcov.

It's long been pretty standard process to have tools which extract doc
from code. On the other hand there are some arguments for keeping
even these two separate. In "The Psychology of Computer Programming"
Gerry Weinberg talked about the trap of "debugging" the comments
instead of the code, and advocated tools which would not just extract
code commentary but REMOVE it.

On the other hand without having the doc with the code and tools like
rdoc we'd probably have even less documentation than we do now, just a
guess.
 
S

Stefan Rusterholz

Peter said:
And if you get onto, say, Formula 1 cars.. some "repairs" can even take
place on the road and the telemetry provides almost as much info to the
engineers while the car is running out on the track as they can get in
the
garage. I've always wondered if/how to make software as easy to diagnose
with real-time telemetry and the ability to make changes to a running
program's parameters..

Take a look at Eiffels Design By Contract. I'm a huge fan of it. Allows
you to move most testing code into the contract, the unit tests are then
merely running your code with contracts enabled.
For an idea on how that works, take a look at the movies with "Design by
Contract" in the title here: http://eiffel.com/developers/presentations/
Would be a very nice feature if ruby had that, even though I'd introduce
additional syntax and I've no idea how to integrate it nicely.

Regards
Stefan
 
A

Alex Young

Stefan said:
Take a look at Eiffels Design By Contract. I'm a huge fan of it. Allows
you to move most testing code into the contract, the unit tests are then
merely running your code with contracts enabled.
For an idea on how that works, take a look at the movies with "Design by
Contract" in the title here: http://eiffel.com/developers/presentations/
Would be a very nice feature if ruby had that, even though I'd introduce
additional syntax and I've no idea how to integrate it nicely.
Allow me to shamelessly copy and paste from an old post of mine:

$ cat testable.rb
module Test; module Unit; class TestCase; end; end; end

module Testable
def test_case(&block)
Class.new(Test::Unit::TestCase).module_eval &block
end
end

$ cat builtin_tests.rb
require 'testable'

class MyWorkingClass
extend Testable

def foo(a,b)
return a+b
end

test_case {
def test_foo_null
assert_equal 2, MyWorkingClass.new.foo(1,1)
end
}
end

if __FILE__ == $0
puts MyWorkingClass.new.foo(1,1)
end

$ ruby builtin_tests.rb
2
$ testrb builtin_tests.rb
Loaded suite builtin_tests.rb
Started
 
S

Stefan Rusterholz

Alex said:
Allow me to shamelessly copy and paste from an old post of mine:

$ cat testable.rb
module Test; module Unit; class TestCase; end; end; end

module Testable
def test_case(&block)
Class.new(Test::Unit::TestCase).module_eval &block
end
end

$ cat builtin_tests.rb
require 'testable'

class MyWorkingClass
extend Testable

def foo(a,b)
return a+b
end

test_case {
def test_foo_null
assert_equal 2, MyWorkingClass.new.foo(1,1)
end
}
end

if __FILE__ == $0
puts MyWorkingClass.new.foo(1,1)
end

$ ruby builtin_tests.rb
2
$ testrb builtin_tests.rb
Loaded suite builtin_tests.rb
Started
.
Finished in 0.00095 seconds.

1 tests, 1 assertions, 0 failures, 0 errors
$


Something like that?

No, the whole approach is different. I'd say the movies I linked to can
explain it far better than I can.
With DBC, if contracts are enabled (in Eiffel you can disable contracts
to get better performance), your method is tested everytime you call the
method. It tests pre-conditions and/or post-conditions (providing you
the old state for your post-conditions). Also after every call to a
method the class invariant is tested to validate your object. That's the
basic idea.

One way that could probably be implemented (though, only as of ruby 1.9
due to &block params) might look this:

class DayOfTime
extend DBC
invariant {
@hour.between?(0,23)
@min.between?(0,59)
@sec.between?(0,59)
}

define:)min=) {
pre_condition { |value|
value.kind_of?(Numeric)
value.between?(0,59)
}
body { |value|
@value = value
}
post_condition { |old, value|
@hour == old.hour
@min == value
@sec == old.second
}
}
end

Regards
Stefan
 
T

Trans

Take a look at Eiffels Design By Contract. I'm a huge fan of it. Allows
you to move most testing code into the contract, the unit tests are then
merely running your code with contracts enabled.
For an idea on how that works, take a look at the movies with "Design by
Contract" in the title here:http://eiffel.com/developers/presentations/
Would be a very nice feature if ruby had that, even though I'd introduce
additional syntax and I've no idea how to integrate it nicely.

http://jroller.com/dscataglini/entry/jim_weirich_test_drivern_development

T.
 
M

micathom

Would be a very nice feature if ruby had that, even though I'd introduce
additional syntax and I've no idea how to integrate it nicely.

There are a few libraries around that try to do this (google for "ruby
dbc" or similar). Most of these efforts are not entirely convincing.
Unfortunately, the ruby community seems somewhat dogmatic in arguing
against dbc. Maybe because it's not enough of a duck.
 
S

Stefan Rusterholz

micathom said:
There are a few libraries around that try to do this (google for "ruby
dbc" or similar). Most of these efforts are not entirely convincing.
Unfortunately, the ruby community seems somewhat dogmatic in arguing
against dbc. Maybe because it's not enough of a duck.

I know, but I found none of them convincing enough. Might change with
ruby2, as there are more possibilities then.
IMHO DBC is orthogonal to duck typing. It changes just what you test
for. Instead having a contract that states "argument must be a duck" you
just will have "argument must quack like a duck". I fail to see how that
goes against duck typing.

Regards
Stefan
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,733
Messages
2,569,439
Members
44,829
Latest member
PIXThurman

Latest Threads

Top