Bugtracking & UnitTests == good?

[martinankerl at eml dot cc]

Hi all! I am afraid this post is a bit offtopic. If you are not
interested in Bugtracking and Unit Tests, read no further!

At the moment I am playing with the idea of writing a simple bugtracking
system in Ruby. I thought that it would be a cool idea to integrate Unit
Tests with such a system: Every reported bug should get a unit test
assigned. If a unit test fails, the bug is automatically reopened. When
the tests is successful, it is closed. I have not really thought this
through, I am just asking if people here think this is be a good idea or
not.

martinus

 

[Sean O'Dell]

Hi all! I am afraid this post is a bit offtopic. If you are not
interested in Bugtracking and Unit Tests, read no further!

At the moment I am playing with the idea of writing a simple bugtracking
system in Ruby. I thought that it would be a cool idea to integrate Unit
Tests with such a system: Every reported bug should get a unit test
assigned. If a unit test fails, the bug is automatically reopened. When
the tests is successful, it is closed. I have not really thought this
through, I am just asking if people here think this is be a good idea or
not.

Actually, this does sound like a great idea. Idea: allow the users who submit
bugs to provide the unit test to prove the bug's existence.

Sean O'Dell

 

[Jamis Buck]

Actually, this does sound like a great idea. Idea: allow the users who submit
bugs to provide the unit test to prove the bug's existence.

Sean O'Dell

It would have to be handled delicately. Otherwise, this sounds like a
great way to discourage people from submitting bugs. I can think of
lots of people, many of whom I work with, who would rather just work
around the bug than report it if the bug reporting process was too
cumbersome.

- Jamis

--
Jamis Buck
(e-mail address removed)
http://www.jamisbuck.org/jamis

"I use octal until I get to 8, and then I switch to decimal."

 

[Sean O'Dell]

It would have to be handled delicately. Otherwise, this sounds like a
great way to discourage people from submitting bugs. I can think of
lots of people, many of whom I work with, who would rather just work
around the bug than report it if the bug reporting process was too
cumbersome.

On the other hand, it would save a lot of time trying to decipher cryptic bug
reports, or reports that misinterpret the nature of the bug itself, making
actual location of the problem difficult. I would say at least half, if not
more, of all bug reports I get don't come with enough information for me to
locate the problem, or I simply cannot reproduce the bug as described.

Sean O'Dell

 

[Jamis Buck]

On the other hand, it would save a lot of time trying to decipher cryptic bug
reports, or reports that misinterpret the nature of the bug itself, making
actual location of the problem difficult. I would say at least half, if not
more, of all bug reports I get don't come with enough information for me to
locate the problem, or I simply cannot reproduce the bug as described.

Sean O'Dell

To be sure, there is a trade off. In this case, it's a trade off between
having really good reports for the bugs that get reported, but missing
many real bugs that would have been reported if the process were less
cumbersome, and having poor-to-mediocre bug reports that, when
reproducible, cover a larger span of bugs.

I know exactly what you mean, Sean--I've recieved the same quality of
bug reports for many projects, and it can be frustrating. But I'd prefer
to know that people have having problems, than to say, "gee, no one is
reporting bugs, therefore my software must be working as expected," when
that is not really the case.

--
Jamis Buck
(e-mail address removed)
http://www.jamisbuck.org/jamis

"I use octal until I get to 8, and then I switch to decimal."

 

[Randy Lawrence]

Hi all! I am afraid this post is a bit offtopic. If you are not
interested in Bugtracking and Unit Tests, read no further!

At the moment I am playing with the idea of writing a simple bugtracking
system in Ruby. I thought that it would be a cool idea to integrate Unit
Tests with such a system: Every reported bug should get a unit test
assigned. If a unit test fails, the bug is automatically reopened. When
the tests is successful, it is closed. I have not really thought this
through, I am just asking if people here think this is be a good idea or
not.

martinus

The best bug/issue tracking tool I've found is Roundup which is written
in Python:

http://roundup.sf.net

It is incredibly easy to customize both the data structure, page layout
and reports. Currently using it for todos, issue tracking and project
management (3 different configurations).

If you can do something like this in Ruby, I'm sure it'll be a hit!

 

[Sean O'Dell]

To be sure, there is a trade off. In this case, it's a trade off between
having really good reports for the bugs that get reported, but missing
many real bugs that would have been reported if the process were less
cumbersome, and having poor-to-mediocre bug reports that, when
reproducible, cover a larger span of bugs.

You've cut out my quote where I said "allow the user"; I didn't say or imply
"force the user." There are many users who can report bugs but simply don't
have the programming skill required to submit a unit test. That said, to a
person with the skill to provide a unit test, it would be a simple process to
write "x feature bombs, check this out" followed by a short unit test.

Sean O'Dell

 

[Lennon Day-Reynolds]

The thing I like about the idea of having actual code which is being
used to test bugs in an issue tracker is that you have a "live" suite
of tests which could catch the re-appearance of bugs even before a
user experiences them.

How about this: each new bug reporter has a status method
automatically associated with it, which is a test case for that issue.
The default status code always returns an error, so before an issue
can be closed, someone must either manually write a test case which
can then be checked to determine the issue status, or they must
explicitly reassign a different "stock" status method which does not
raise an exception (and possibly returns some symbolic value like
"WorkingAsDesigned", or "NotReproduceable", which can go into a full
test report).

Unforunately, it seems like it would require a huge amount of trust in
your users, or tightly-restricted permissions for who could write such
cases, since you would be (presumably) accepting live code through the
web, with all of the security implications that causes.

Lennon

 

[Charles Mills]

Hi all! I am afraid this post is a bit offtopic. If you are not
interested in Bugtracking and Unit Tests, read no further!

At the moment I am playing with the idea of writing a simple
bugtracking system in Ruby. I thought that it would be a cool idea to
integrate Unit Tests with such a system: Every reported bug should get
a unit test assigned. If a unit test fails, the bug is automatically
reopened. When the tests is successful, it is closed. I have not
really thought this through, I am just asking if people here think
this is be a good idea or not.

martinus

Seems like a great way to prevent bugs from coming back to life. The
Pragmatic Programmer talks about the importance of adding tests for
each bug that is discovered. If you had a test script for every bug
that popped up you would end up with lots of files. Although I guess a
test script could just be a string/record in a database which is used
to generate a temp file when the tests actually needs to be run (or
just pipes a script into ruby). Using a database would also help solve
the problem of creating a link between the bug report and the unit
test....
Sounds like a major project. I would be interested in the results.
-Charlie

 

[Sean O'Dell]

How about this: each new bug reporter has a status method
automatically associated with it, which is a test case for that issue.
The default status code always returns an error, so before an issue
can be closed, someone must either manually write a test case which
can then be checked to determine the issue status, or they must
explicitly reassign a different "stock" status method which does not
raise an exception (and possibly returns some symbolic value like
"WorkingAsDesigned", or "NotReproduceable", which can go into a full
test report).

This is an excellent idea, even if combined with a system that allows users to
submit their own unit tests. The user-submitted unit test could be disable
until a developer has looked at it for security and then it could be added to
the battery of tests.

Unforunately, it seems like it would require a huge amount of trust in
your users, or tightly-restricted permissions for who could write such
cases, since you would be (presumably) accepting live code through the
web, with all of the security implications that causes.

I think perhaps the cadre of developers for a given project could all be given
the ability to review and "enable" user-submitted unit tests.

Sean O'Dell

 

[Lothar Scholz]

Hello Randy,


RL> The best bug/issue tracking tool I've found is Roundup which is written
RL> in Python:

RL> http://roundup.sf.net

I found that a AskSam is the best program for tracking
bugs and features. And for 99% of all projects that are done with
scripting languages a "bugs.txt" is an appropriate bugtracking tool.

And to be honest i stopped unit tests, because the current libraries
GUI, networking, filesystems do not support them well enough. Hope
that this will improve in the future.

 

[Patrick May]

And to be honest i stopped unit tests, because the current libraries
GUI, networking, filesystems do not support them well enough. Hope
that this will improve in the future.

Most things aren't that bad. Mocking up files, mocking up command line
interactions, these are cheap. GUI's are tough to test. I think there
is alot of hope for web libraries though. On the protocol level, HTTP
behaves alot like a command-line. It's a laborious, rather than
impossible, task to make HTTP easy to test.

Cheers,

Patrick

 

[Bret Pettichord]

If a unit test fails, the bug is automatically reopened.

What if the test fails for a different reason that that which triggered the
original bug report? Shouldn't that be a new bug report?

I'm not quite sure what the benefit of automatically reopening a bug would
be anyway. If you don't check in code unless it passes all unit tests, then
you'd never have to worry about this situation in the first place.

I generally think that it is more worthwhile to expend effort on tests than
on a bug tracking system.


______________________________________
Bret Pettichord, Software Tester
Consultant - www.pettichord.com
Author - www.testinglessons.com
Blogger - www.io.com/~wazmo/blog

Scripting Web Tests Tutorial
XP Agile Universe, Aug 15, Calgary
http://www.xpuniverse.com/schedule/T15

 

[Martin Ankerl]

What if the test fails for a different reason that that which triggered the

original bug report? Shouldn't that be a new bug report?

I finally had a bit time to look a bit further into this project. The
problem you state is, that the unit test might not directly related to
the report: it can cover more or less of the problem. There actually
seems to be an M to N relation between bugs and unit tests. This
complicates things, but I find this very interesting: now there is
information available that enables me to list all the tests that
relate to a bug, and the other way round. I am not yet shure what this
information is good for, but I will think of something

I'm not quite sure what the benefit of automatically reopening a bug would
be anyway. If you don't check in code unless it passes all unit tests, then
you'd never have to worry about this situation in the first place.

That's true. But I am not shure that this is always possible. There
might be bug reports that take a lot of time to fix. Here's again the
problem that unit tests do not directly relate to bug reports. Hm, I
need more time for this

I generally think that it is more worthwhile to expend effort on tests than
on a bug tracking system.

Well, I want to spend effort on the combination of both. My goal is to
make both more convenient and useful, and maybe create something
that's interesting, and new.

Martin