unit testing advice

[Michael Guterl]

I wish I had learned this stuff ten years ago.

For almost my entire career people have assumed that I can look at
their code and tell what it is supposed to do. I can't; no-one can,
with any accuracy. All I can do is look at code and say what it
*does*. Which is probably not the same thing, or I wouldn't be
looking at the code in the first place...

But if there are automated tests, *they* will tell you what the code
should do. And, better yet, if the programmers have been using them,
they will be accurate and up to date. They might not be complete, of
course, but two out of three is a hell of a good hit rate in this
subject.

As far as I am concerned, that's good enough to spark a religious conversion.


As for test-driven-design, I think it's a fine ideal. Mostly when I
code I know in advance where I'm going, and in that case it pays to
use TDD. But sometimes I haven't the faintest idea what I'm doing; I
don't know how many classes I need or what jobs go in what classes --
or even if the idea is at all workable. I suspect that TDD isn't of
much use in those times.

Check out Test-Driven Development By Example by Kent Beck. It brought
my understanding of TDD to a new level.

Regards,
Michael Guterl

 

[James Britt]

Hi --

On Sun, 3 Aug 2008, James Britt wrote:


That may be the ideal for some people, but not for me when I'm
modeling a domain and especially when I'm working out a database
schema. I don't consider unit tests to have superseded the other tools
that support those activities, including blank pieces of paper and
index cards and so on. I guess you could write unit tests for a Rails
app before creating the models, if you mocked up the objects'
attributes, their associated objects, and so on (since there would
presumably be no database yet), and then rename your files so they
don't get clobbered when you generate the test files... but it seems
like it would be terribly arduous, with no real gain, and I don't
think I've ever seen anyone do it.

http://www.infoq.com/interviews/Dave-Astels-and-Steven-Baker#

Quote:

I guess you don't use the Rails code generation then.

Steven: No, not at all. I use TDD to generate code manually.

So you don't even do model generation with Rails?

Steven: Not at all. Especially when I'm pairing with somebody who's
never used Rails before or who has never done TDD before I find it
especially important not to rely on the generator to give me test stubs,
to give me things that I should know or that I should be providing for
myself. When the failings specifications tell me that I need it, because
that would be contradicting what I'm trying to teach.





--
James Britt

www.happycamperstudios.com - Wicked Cool Coding
www.jamesbritt.com - Playing with Better Toys
www.ruby-doc.org - Ruby Help & Documentation
www.rubystuff.com - The Ruby Store for Ruby Stuff

 

[David A. Black]

Hi --

http://www.infoq.com/interviews/Dave-Astels-and-Steven-Baker#

Quote:

I guess you don't use the Rails code generation then.

Steven: No, not at all. I use TDD to generate code manually.

So you don't even do model generation with Rails?

Steven: Not at all. Especially when I'm pairing with somebody who's never
used Rails before or who has never done TDD before I find it especially
important not to rely on the generator to give me test stubs, to give me
things that I should know or that I should be providing for myself. When the
failings specifications tell me that I need it, because that would be
contradicting what I'm trying to teach.

OK, now I've seen one person I imagine there are more. Luckily we
don't need a consensus; we can all do what we're comfortable with.
It's interesting that Steve does it this way, and I'd like to see it
in person.

It doesn't appeal to me not to use the model generator. For one thing,
I assume that it would mean having to run sequential migrations, where
you wrote a test or spec that tested for the presence of, say, a
first_name attribute on the user; then add that column to the
database; then test for last_name; then add that column; and so forth.
I don't mean to sound like I'm parodying what Steve is describing. I
assume that this, or something like it, is what you'd end up doing.


I'm not sure what Steve means by test stubs in this context. The model
generation just gives you a file with an empty class definition for
your tests, and I think something similar happens with RSpec. (Just to
clarify: I'm talking about model generation, not scaffold generation,
which does give you all sorts of tests and which I dislike strongly
and consider manifestly counter-productive.)


David

 

[Phlip]

http://www.infoq.com/interviews/Dave-Astels-and-Steven-Baker#

Quote:

I guess you don't use the Rails code generation then.

Steven: No, not at all. I use TDD to generate code manually.

So you don't even do model generation with Rails?

Steven: Not at all. Especially when I'm pairing with somebody who's
never used Rails before or who has never done TDD before I find it
especially important not to rely on the generator to give me test stubs,
to give me things that I should know or that I should be providing for
myself. When the failings specifications tell me that I need it, because
that would be contradicting what I'm trying to teach.

I don't know if we are discussing the same thing. This Rails command...

script/generate model MyModel

....does not create "stubs". It creates all the files that back up a Model,
including its my_model_test.rb and fixtures/my_models.yml files.

I can't imagine not using it, because all it saves you is a lot of rote typing.
Writing a test that fails because "fixtures/my_models.yml" does not exist is not
TDD, because that's not production code.

I often write a test that fails because MyModel does not exist yet, to start the
cycle.

I think Dave and Steve are discussing some tool that reads Ruby classes and
spits out stubbed test cases for each method.

 

[Phlip]

It doesn't appeal to me not to use the model generator. For one thing,
I assume that it would mean having to run sequential migrations, where
you wrote a test or spec that tested for the presence of, say, a
first_name attribute on the user; then add that column to the
database; then test for last_name; then add that column; and so forth.
I don't mean to sound like I'm parodying what Steve is describing. I
assume that this, or something like it, is what you'd end up doing.

Parenthetically, you are "allowed" to edit a pre-existing migration file, if you
can keep track of which VERSION your production database has.

Within that loop, yes, write a test that fails because MyModel has no has_many
NotherModel, then pass it by adding a column :nother_model_id to the migration.

 

[James Britt]

James Britt wrote:

I don't know if we are discussing the same thing. This Rails command...

script/generate model MyModel

...does not create "stubs". It creates all the files that back up a
Model, including its my_model_test.rb and fixtures/my_models.yml files.

That's exactly what is being talked about.

And the issue is that it creates a model file before the developer has
written a failing test indicating the need for a model file.


One might assume that the developer is going to end up in the same place
anyways, so the code might as well be automatically generated, but it's
contrary to "write the spec, have it fail, then write the code to make
the spec pass".

I'm not arguing in favor of dogmatic testing practices, justing pointing
out that there is at least one person who actually walks the walk.

(Besides, when a developer selects Rails as their toolkit they've
already made many code choices long before there are any tests to drive
those decisions, so the TDD/BDD purity thing is sort of a moot point.)

--
James Britt

www.happycamperstudios.com - Wicked Cool Coding
www.jamesbritt.com - Playing with Better Toys
www.ruby-doc.org - Ruby Help & Documentation
www.rubystuff.com - The Ruby Store for Ruby Stuff

 

[David A. Black]

Hi --

Parenthetically, you are "allowed" to edit a pre-existing migration file, if
you can keep track of which VERSION your production database has.

Yes, but then you get into migrating backwards, which I increasingly
think is almost never appropriate. (Though not quite never, maybe.)

Within that loop, yes, write a test that fails because MyModel has no
has_many NotherModel, then pass it by adding a column :nother_model_id to the
migration.

I'm afraid I don't see the advantage of that, at least in every case,
over bootstrapping at least part of a database schema and model
structure first. Certainly the kind of thing you're describing does
happen, but if I'm creating an Order model and I know that it's going
to belong to Customer, I'm not averse to saying so up front. Also,
with a large table, you could get into an awful lot of migrations.

It all comes out in the wash, so to speak, since once the project is
bootstrapped in some way, the different approaches to bootstrapping
tend to converge.


David

 

[David A. Black]

Hi --

That's exactly what is being talked about.

And the issue is that it creates a model file before the developer has
written a failing test indicating the need for a model file.


One might assume that the developer is going to end up in the same place
anyways, so the code might as well be automatically generated, but it's
contrary to "write the spec, have it fail, then write the code to make the
spec pass".

I'm not arguing in favor of dogmatic testing practices, justing pointing out
that there is at least one person who actually walks the walk.

(Besides, when a developer selects Rails as their toolkit they've already
made many code choices long before there are any tests to drive those
decisions, so the TDD/BDD purity thing is sort of a moot point.)

I don't think anything in Rails militates against a test-driven
approach, though -- in fact, by creating test files for you, and
having the default 'rake' command be to run your tests, it creates a
very hospitable environment for testing. If you don't write tests,
you're always reminded of their (non)existence by the presence of the
files.


David

 

[James Gray]

And the issue is that it creates a model file before the developer
has written a failing test indicating the need for a model file.

One might assume that the developer is going to end up in the same
place anyways, so the code might as well be automatically generated,
but it's contrary to "write the spec, have it fail, then write the
code to make the spec pass".

I'm not arguing in favor of dogmatic testing practices, justing
pointing out that there is at least one person who actually walks
the walk.

(Besides, when a developer selects Rails as their toolkit they've
already made many code choices long before there are any tests to
drive those decisions, so the TDD/BDD purity thing is sort of a moot
point.)

Yeah, I think this is a good example of the dogma getting in the way
of common sense. To use Rails and not allow it to generate files and
by extension satisfy it's own conventions for you just sounds silly to
me.

My opinion is that the goal of testing is to add some degree of
protection to the software you build, not to straight-jacket you as a
programmer.

James Edward Gray II

 

[James Britt]

On Aug 3, 2008, at 12:03 PM, James Britt wrote:

Yeah, I think this is a good example of the dogma getting in the way of
common sense. To use Rails and not allow it to generate files and by
extension satisfy it's own conventions for you just sounds silly to me.

My opinion is that the goal of testing is to add some degree of
protection to the software you build, not to straight-jacket you as a
programmer.

I've been wondering about resistance to "purist" TDD/BDD, and if there's
some aspect of "listen to your body" to observe. For example, when
exercising or dieting, there are assorted rules or regimes to follow,
but if something is hurting, or if you've constant cravings for some
food or another, then maybe that's an important sign to deviate from the
original plan, and not simply a lack of will power or a failure to "do
it right."

Rather than people applying ad hoc approaches to Test-second
Development, maybe there are some better practices to be shared for what
seems to be a common situation.


--
James Britt

www.happycamperstudios.com - Wicked Cool Coding
www.jamesbritt.com - Playing with Better Toys
www.ruby-doc.org - Ruby Help & Documentation
www.rubystuff.com - The Ruby Store for Ruby Stuff

 

[Phlip]

That's exactly what is being talked about.

And the issue is that it creates a model file before the developer has
written a failing test indicating the need for a model file.

So write the failing test, then run script/generate. Manually writing all the
files that script/generate creates is just heroism.

 

[James Gray]

I've been wondering about resistance to "purist" TDD/BDD, and if
there's some aspect of "listen to your body" to observe. For
example, when exercising or dieting, there are assorted rules or
regimes to follow, but if something is hurting, or if you've
constant cravings for some food or another, then maybe that's an
important sign to deviate from the original plan, and not simply a
lack of will power or a failure to "do it right."

Yeah, I think you are making a lot of sense here.

For example, I haven't been able to get into RSpec. I know a lot of
people really like it and it's gained a lot of popularity recently,
but it's not for me.

The interface just "feels" wrong to me. To give a random example, I
can't imagine how I would explain to a programming student something
like, "Now we can check that valid?() method you just wrote using
be_valid()." I would much rather be saying, "OK, let's call valid?()
a few times and see if we get the results we expect."

Even if RSpec becomes the one true way to handle Ruby testing, I'm
going to have to respectfully decline to follow suit. I'm glad people
like it; I'm glad it's bringing new people to testing; I'm glad it's
encouraging such good practices. It's just not for me.

James Edward Gray II

 

[Shadowfirebird]

Well, I'm not a professional Ruby developer, but I *am* someone that
writes code for a living, and I've been an IT contractor.

And while I love the idea of TDD, if I were using it professionally
then I simply wouldn't be able to afford to be that dogmatic. Even if
I always, always, wrote the tests first, sooner or later I would run
up against code written by somewhere else who didn't -- and as a
contractor, of course, I never got to choose what programming and
testing methodologies were being used at all. ("You want me to indent
every other line by four tabs? Okay, you're paying.")

I'm not criticising anyone for being gung-ho about DTT; I would simply
advise everyone to learn how to write tests second, as well.

Shadowfirebird.

So write the failing test, then run script/generate. Manually writing all
the files that script/generate creates is just heroism.

--
Me, I imagine places that I have never seen / The colored lights in
fountains, blue and green / And I imagine places that I will never go
/ Behind these clouds that hang here dark and low
But it's there when I'm holding you / There when I'm sleeping too /
There when there's nothing left of me / Hanging out behind the
burned-out factories / Out of reach but leading me / Into the
beautiful sea

 

[Gregory Brown]

Yeah, I think you are making a lot of sense here.

For example, I haven't been able to get into RSpec. I know a lot of people
really like it and it's gained a lot of popularity recently, but it's not
for me.

The interface just "feels" wrong to me. To give a random example, I can't
imagine how I would explain to a programming student something like, "Now we
can check that valid?() method you just wrote using be_valid()." I would
much rather be saying, "OK, let's call valid?() a few times and see if we
get the results we expect."

I like RSpec in a lot of ways, but this stuff annoys me too. The
trouble is the lack of bare assertions and the dogmatic assumption
that introducing them could be evil. But I don't know, my inner
Rubyist and non-conformist asks me, why shouldn't I just be able to do
this?

class Thing
def valid?
false
end
end

def assert(thing)
(!!thing).should be_true
end

describe "A thing" do
it "should be valid" do
@thing = Thing.new
assert @thing.valid?
end
end

It's not like this code is hard to follow, but it's annoying that I
need to create a custom hack to support the most basic thing (a simple
assert).

But hacking in some less magic alternatives to RSpec isn't exactly
appealing. Things like Shoulda and test/spec and friends already do
that. What I'd really want in Test::Unit to make it feel like a good
solution 'out of the box' is:

* Some standard mocking library - Mocha is very pretty IMO, but any would do

* Full Test::Unit compatibility to play nice with the massive amount
of pre-existing tests out there

* Flexibility to extend the framework to go in the direction *I* want
it to. I'm personally quite sick of people telling me that I'm doing
something wrong because it doesn't read like English, or because I use
a setup method that creates a 'dependency' between the tests. Though
these are valid conversations to have, people need to stop acting like
there are established laws, and should give us the freedom to bend our
tools to our work flow, not the other way around.

* The ability to specify my test cases as strings rather than methods
like test_something_should_do_something

* The ability to have contexts and nested contexts.

* Nice built in extras to do pretty HTML reports, coverage reports, etc.

I know there are some efforts heading in this direction, but I feel
like standard Ruby should really ship with something that accomplishes
all of the above. I don't hate BDD style syntax, but it's the last
reason why I use RSpec. The lack of compatibility with other testing
tools creates a divide that I don't like, too.

It mostly seems that people who are accustomed to Test::Unit are more
tolerant and willing to write specs if necessary, but that BDD
advocates who are convinced they've learned "The One True Way" of
doing things really have a strong aversion to going back to xunit
style syntax. Seems like a mental hangup to me.

Anyway, what I've said applies only to a small percentage of people
I've encountered in the community, but they have seemed to be quite
vocal.

I am mostly thinking that RSpec owes a huge chunk of its success to
its glittering side-features, and not the BDD syntax magic. I wonder
if others feel the same way.

-greg

 

[David A. Black]

Hi --

I've been wondering about resistance to "purist" TDD/BDD, and if there's some
aspect of "listen to your body" to observe. For example, when exercising or
dieting, there are assorted rules or regimes to follow, but if something is
hurting, or if you've constant cravings for some food or another, then maybe
that's an important sign to deviate from the original plan, and not simply a
lack of will power or a failure to "do it right."

I definitely think so. We're all (in theory) discerning, careful
practioners who are constantly monitoring the conditions under which
we're working, dealing with exceptional situations, re-evaluating our
technologies, and so forth. All you're saying, as I read it, is that
we should not stop behaving like this, just because the technology in
question is test-related. That makes perfect sense to me. I never
signed on to switch off my critical faculties for test-writing
purposes.

Rather than people applying ad hoc approaches to Test-second Development,
maybe there are some better practices to be shared for what seems to be a
common situation.

It's common but I think it's also probably always going to involve
some ad hoc thinking, since the circumstances under which one might be
retro-fitting tests can vary so widely. I like the idea of thinking it
through systematically, though (and I don't claim to have done so at
this stage).


David

 

[Dave Bass]

So should we write code first, or tests first?

Tests are themselves bits of code, so they may have bugs in them. No
really, I've put them there myself so I know. It's galling to discover
that your "real" code is correct but the test is failing because you got
it wrong. Duh, I was looking for the bug in completely the wrong place!

This realisation puts code and tests on a more parallel footing. The
upshot is that code and tests should be written in parallel as far as
possible; certainly at method level. Most of the time I like to write a
test or two to exercise the few lines of code I've just written. By few
I mean typically 5..10.

I like to write my code first, tests second. Otherwise how do I know
what I'm supposed to be testing? (There's no spec, I make things up as I
go along. This is a hobby, not work.)

Other people probably do things differently.

Dave

 

[James Gray]

I like to write my code first, tests second. Otherwise how do I know
what I'm supposed to be testing? (There's no spec, I make things up
as I
go along. This is a hobby, not work.)

I actually prefer to do the tests first. This let's me think through
a question like, "How do I want to use this code?" I find that helps
prevent me from designing awful interfaces.

At times though, I will write some code and then cover it in tests.
I'm not a super dogmatic person.

James Edward Gray II

 

[Avdi Grimm]

I'm afraid I don't see the advantage of that, at least in every case,
over bootstrapping at least part of a database schema and model
structure first. Certainly the kind of thing you're describing does
happen, but if I'm creating an Order model and I know that it's going
to belong to Customer, I'm not averse to saying so up front. Also,
with a large table, you could get into an awful lot of migrations.

I've seen a lot of Rails coders completely lose track of the fact that
"Model" means "Domain Model" because of thinking like that. To the
point of getting confused when you start talking about models that
don't derive from ActiveRecord::Base. They've begun to identify
"Model" with "table in the database" rather than "model of a domain
entity".

I challenge any Rails developer to start a a web app without a
database, leaving persistence for iteration 2. It can be an
eye-opening experience, forcing you to think more about domain
concerns and less about keeping MySQL happy.

--
Avdi

Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com

 

[David A. Black]

Hi --

I've seen a lot of Rails coders completely lose track of the fact that
"Model" means "Domain Model" because of thinking like that. To the
point of getting confused when you start talking about models that
don't derive from ActiveRecord::Base. They've begun to identify
"Model" with "table in the database" rather than "model of a domain
entity".

I repeat: *if* I'm creating an Order model and I know that it's going
to belong to Customer, etc. That's not the only possible scenario, but
it can happen. Obviously it has no bearing on non-AR models, and so
forth. There's definitely an issue with over-identification of models
and tables, but we've been talking rather specifically about the case
where the associations and table structure are already known (or at
least a piece of them), and whether or not it makes sense, in the
presence of that knowledge, to refrain from implementing those things
until there are failing tests for them. The question of whether to use
belongs_to and so forth in the first place is important but different.

I challenge any Rails developer to start a a web app without a
database, leaving persistence for iteration 2. It can be an
eye-opening experience, forcing you to think more about domain
concerns and less about keeping MySQL happy.

I usually do it sort of the other way around: model the domain first,
then think about how that will play out in a database schema, and then
start to think about it as a Web app. I often spend a lot of time in
the console before I create a controller. Some people like to work
view-first. I'm the opposite.

I've never thought about keeping MySQL happy, I have to admit. I
expect it to keep me happy


David

 

[David A. Black]

Hi --

I've seen a lot of Rails coders completely lose track of the fact that
"Model" means "Domain Model" because of thinking like that. To the
point of getting confused when you start talking about models that
don't derive from ActiveRecord::Base. They've begun to identify
"Model" with "table in the database" rather than "model of a domain
entity".

Just another follow-up (see also my earlier post, which addresses your
concerns directly): there's a new screencast at railscasts.com today,
on exactly that topic: models that are not ActiveRecord models. It's
here:

http://media.railscasts.com/videos/121_non_active_record_model.mov

I'm glad to see it, as this is something that people often don't
realize is possible, and when they do it can be quite a revelation.


David