unittest: Proposal to add failUnlessNear

[Antoon Pardon]

I have been working with unittests lately and found that the
self.failUnlessAlmostEqual, isn't as usefull as it could be.

My main problem is, that it is only usefull with objects
that can be converted to floats, while there are a whole
bunch of objects that can be almost equal but not so
convertable. The first example coming to mind being
complex numbers.

A secondary objection is that you are limited to
a very specific set of tolerances. If for instance
you want to verify that two numbers differ at most
by 0.0003 you can't specify that.

So I propose to add the following


def failUnlessNear(self, first, second, tolerance=1e-7, msg=None, norm=abs):
"""Fail if the two objects are too far appart as determined
by the distance between them and the tolerance allowed.
"""
if norm(second-first) > tolerance:
raise self.failureException, \
(msg or '%s != %s within %s tolerance' % (`first`, `second`, `tolerance`))

 

[Jeff Epler]

I have been working with unittests lately and found that the
self.failUnlessAlmostEqual, isn't as usefull as it could be.

I'd like to hear more about the uses you envision about this, and
especially about how to choose "tolerance" properly.

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFA++LfJd01MZaTXX0RAmGkAJ48vek3HG/xN38Qrhc9dTNAOXlhiwCgqACB
6wvCcYIp0ko8m8dJWDgp9E8=
=qbcT
-----END PGP SIGNATURE-----

 

[Bengt Richter]

I have been working with unittests lately and found that the
self.failUnlessAlmostEqual, isn't as usefull as it could be.

My main problem is, that it is only usefull with objects
that can be converted to floats, while there are a whole
bunch of objects that can be almost equal but not so
convertable. The first example coming to mind being
complex numbers.

A secondary objection is that you are limited to
a very specific set of tolerances. If for instance
you want to verify that two numbers differ at most
by 0.0003 you can't specify that.

So I propose to add the following


def failUnlessNear(self, first, second, tolerance=1e-7, msg=None, norm=abs):
"""Fail if the two objects are too far appart as determined
by the distance between them and the tolerance allowed.
"""
if norm(second-first) > tolerance:
raise self.failureException, \
(msg or '%s != %s within %s tolerance' % (`first`, `second`, `tolerance`))

How about a more general solution? E.g., how about an optional keyword arg
cmp=comparison_function
passed to failUnlessAlmostEqual? (I'm guessing here, no time to look at it)
E.g., that way for mixed numbers including complex you could (if you thought
it made sense, which I am not necessarily arguing ;-) use e.g.

def special_cmp(x,y): # untested!
diff = complex(x)-complex(y)
return max(cmp(abs(diff.real), tolerance), cmp(abs(diff.imag), tolerance))

and pass cmp=special_cmp as the kwarg.

For special objects, you could define other kinds of nearness, and raise
appropriate informative exceptions if you get non-comparable arguments.

(obviously tolerance has to be defined, so if you want to vary it conveniently,
you could pass it to a factory function that does the above def and returns
it with tolerance captured in a closure referred to by special_cmp).

Or some other way. The point is you get to define the cmp= function as you please
without modifying the framework (once the optional kwarg is implemented).)

gotta go...

Regards,
Bengt Richter

 

[John Roth]

I have been working with unittests lately and found that the
self.failUnlessAlmostEqual, isn't as usefull as it could be.

My main problem is, that it is only usefull with objects
that can be converted to floats, while there are a whole
bunch of objects that can be almost equal but not so
convertable. The first example coming to mind being
complex numbers.

A secondary objection is that you are limited to
a very specific set of tolerances. If for instance
you want to verify that two numbers differ at most
by 0.0003 you can't specify that.

Float and complex have two very different issues.
Since I'm maintaining the Python port of FIT, I
had to deal with them. What I did for float was
use the character form of the float to determine
the precision. The default routine appends a
character 5 and then uses this to calculate the
bounds within which the other value must be found.

An example may help. Let's say I want to
test against '6.23e14'. This method would
establish bounds of 6.235e14 and 6.225e14
and then test whether the other number was
within those bounds. (Note that the '5' is
appended before the numbers are converted
to float, however the second bound is calculated
after the conversion.)

There's also a way of prescribing something other
than '5' to append. This approach is due to
Ward Cunningham; what I did was move it
from the ScientificDouble class to the base
float type adapter. (The Java source can be
located in a number of places, including
fit.c2.com and www.fitnesse.org. The Python
source is in the files section of the Extremeprogramming
and fitnesse Yahoo groups. You're welcome
to copy the approach, but since Ward put it
under the GPL, I don't think it can be directly
dropped into unittest.)

Complex, on the other hand, should be dealt
with by calculating the distance between the
two values in the i,j plane
and then seeing if this distance is within a given
bound. The bound should, of course, be a
separately specifiable parameter.

One major issue is that these are very type specific;
the methods should be called something like
"failUnlessFloatWithinTolerance" and
"failUnlessComplexWithinTolerance".

John Roth

 

[John Roth]

How about a more general solution? E.g., how about an optional keyword arg
cmp=comparison_function
passed to failUnlessAlmostEqual? (I'm guessing here, no time to look at it)
E.g., that way for mixed numbers including complex you could (if you thought
it made sense, which I am not necessarily arguing ;-) use e.g.

def special_cmp(x,y): # untested!
diff = complex(x)-complex(y)
return max(cmp(abs(diff.real), tolerance), cmp(abs(diff.imag), tolerance))

and pass cmp=special_cmp as the kwarg.

For special objects, you could define other kinds of nearness, and raise
appropriate informative exceptions if you get non-comparable arguments.

(obviously tolerance has to be defined, so if you want to vary it conveniently,
you could pass it to a factory function that does the above def and returns
it with tolerance captured in a closure referred to by special_cmp).

Or some other way. The point is you get to define the cmp= function as you please
without modifying the framework (once the optional kwarg is implemented).)

I think this is overkill. I find myself using a raw assert much of the time
rather than unittest's specific test methods; it works just as well and I
don't have to remember the blasted syntax!

The only reason for doing float and complex specially is that they are
basic Python types, and while it's trivial to do the correct floating point
test (assert abs(x - y) < tolerance) in each instance, it's repetitive code
that isn't that easy to generalize correctly (at least without having to
specify the tolerance on each call). See my other response for the
general solution I used in FIT.

John Roth

 

[Tim Peters]

[John Roth]

I think this is overkill. I find myself using a raw assert much of the time
rather than unittest's specific test methods; it works just as well and I
don't have to remember the blasted syntax!

It doesn't work as well, because assert stmts "vanish" when running
tests with -O. Luckily,

self.assert_(expression)

works fine in a unittest, with or without -O.

self.assertEqual(e1, e2)

is usually a lot better than

self.assert_(e1 == e2)

too, because the former displays the unequal values when it fails.

 

[Antoon Pardon]

--8/UBlNHSEJa6utmr
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline



I'd like to hear more about the uses you envision about this,

It can be used with every vectorspace that defines a distance
between vectors.

If someone wants to implement quaternions, he can use this
method to test whether the result is close enough to
what it should be.

Same with vectors or simple complex calculations.

and
especially about how to choose "tolerance" properly.

Sorry, but that would require a course in numerical
analysis.

 

[Antoon Pardon]

Float and complex have two very different issues.

Only if you want them to be. Floats are a subset
of complex and as such can be treated exactly
the same.

Since I'm maintaining the Python port of FIT, I
had to deal with them. What I did for float was
use the character form of the float to determine
the precision. The default routine appends a
character 5 and then uses this to calculate the
bounds within which the other value must be found.

An example may help. Let's say I want to
test against '6.23e14'. This method would
establish bounds of 6.235e14 and 6.225e14
and then test whether the other number was
within those bounds. (Note that the '5' is
appended before the numbers are converted
to float, however the second bound is calculated
after the conversion.)

Why don't you just test whether the second
number lies within a fault tolerance of
5e11 of 6.23e14

There's also a way of prescribing something other
than '5' to append. This approach is due to
Ward Cunningham; what I did was move it
from the ScientificDouble class to the base
float type adapter. (The Java source can be
located in a number of places, including
fit.c2.com and www.fitnesse.org. The Python
source is in the files section of the Extremeprogramming
and fitnesse Yahoo groups. You're welcome
to copy the approach, but since Ward put it
under the GPL, I don't think it can be directly
dropped into unittest.)

Complex, on the other hand, should be dealt
with by calculating the distance between the
two values in the i,j plane
and then seeing if this distance is within a given
bound. The bound should, of course, be a
separately specifiable parameter.

There is no reason floats should be treated differently.
If you want to, be my guest, but there really is no
need, and doing so when you are working with both
just complicates things needlessly.

One major issue is that these are very type specific;
the methods should be called something like
"failUnlessFloatWithinTolerance" and
"failUnlessComplexWithinTolerance".

Why? calculating the distance between numbers and
checking whether this distance is within bounds
works just as good for floats as it does for
complex numbers. We don't need an absForFloat
and an absForComplex either. So why do you want
to split this up?

 

[Roy Smith]

Only if you want them to be. Floats are a subset
of complex and as such can be treated exactly
the same.

Why do you say floats are a subset of complex? There's a reasonable
argument that could be made that int/float and scalar/complex are
orthogonal properties. No reason why you shouldn't be able to have a
set of numbers which are 2-tuples with real and imaginary parts, each of
which has an integral value. The set would, I believe, be closed for
the same operations that integers are (addition, subtraction, and
multiplication).

Of course, I'm talking theoretically. In the current Python, complex
numbers do indeed appear to have floating point components.

 

[Antoon Pardon]

Piggy backing.

I think the ability to define a norm, being the length of the number,
vector or other entities is all that is needed. The reason I think so is that
mathematcally, the norm of a vector and the distance between two vectors
are closely linked. The norm of a vector is the distance between this
vector and the null vector and the distance between two vectors is the
norm of the difference vector.

We could of course leave this relationship behind, but then I think
providing a distance function is enough, something like the following
should suffice.

def failUnlessNear(self, first, second, tolerance=1e-7, msg=None,
distance=lambda x,y :abs(x-y)):
"""Fail if the two objects are too far appart as determined
by the distance between them and the tolerance allowed.
"""
if distance(second,first) > tolerance:
raise self.failureException, \
(msg or '%s != %s within %s tolerance' % (`first`, `second`, `tolerance`))

I'm sorry but I have a hard time figuring out what you want to do here.
First of all, your special_cmp normally has no access to the tolerance.
So either you want a third parameter or you are mixing things that
should go in special_cmp with things you want to happen in
failUnlessNear. Using my distance approach, I think you wanted this:

def dist(x,y):

diff = complex(x)-complex(y)
return max(abs(diff.real) , abs(diff.imag))


Well I have a hard time imaganing a usefull distance function whose
results don't form a total order. If I have a tolerance t and an
actual distance d, but d and t don't compare then the question whether
the two objects are near enough is unanswerable.

Well mostly I'm all for choosing the more general solution. But in this
case I sure would like to know if you have an actual example in mind.

The reason I ask is that this is stuff that comes from mathematics and
this is AFAIK a general framework that is used for distances on all
kind of mathematical objects. I can of course be ignorant of a more
general framework in mathematics to deal with this and of course it
is possible that a more general approach is usefull although it hasn't
caught the interest of the mathematical community, that is why I ask
for an example.

 

[Antoon Pardon]

Why do you say floats are a subset of complex?

because if x is a float it is equal to x + 0j.

So whatever algorithm you use to see whether
two complex numbers a and b are near enough,
you can use that same algorithm to see whether
two floats f and g are near enough by using
f + 0j and g + 0j for a and b.

more specifically you can use the test:

abs(a - b) < tolerance.

for complex numbers as well as floats, and
for every numberlike object you wish to
define that has a __sub__ and an __abs__
method.

There's a reasonable
argument that could be made that int/float and scalar/complex are
orthogonal properties. No reason why you shouldn't be able to have a
set of numbers which are 2-tuples with real and imaginary parts, each of
which has an integral value. The set would, I believe, be closed for
the same operations that integers are (addition, subtraction, and
multiplication).

Indeed no reason, but you wouldn't call them complex and I would
call the integers a subset of these defined numbers for the same
reason as I called the floats a subset of complex.

 

[Jeff Epler]

Sorry, but that would require a course in numerical
analysis.

Oh, that's too bad. I bet this will end up being misused by a lot of
pepole, then, because few of us have had courses in numerical analysis.

Oh well, as long as it's there for you and for Tim Peters, I guess it's
no loss for the rest of us.

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFA/QiwJd01MZaTXX0RAhfzAKCZU7M1+QheQzdV1cW93oz2Z9H9sQCbByjG
xP45gdjQyPDeSz8ruSW5MLs=
=I90K
-----END PGP SIGNATURE-----

 

[Antoon Pardon]

--TB36FDmn/VVEgNH/
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline




Oh, that's too bad. I bet this will end up being misused by a lot of
pepole, then, because few of us have had courses in numerical analysis.

Well then I suspect that assertAlmostEqual, is being missued by a lot
of people, because properly choosing the right number of places to round to,
is about the same difficulty as properly choosing a tolerance.

 

[Michael Hudson]

[John Roth]

I think this is overkill. I find myself using a raw assert much of the time
rather than unittest's specific test methods; it works just as well and I
don't have to remember the blasted syntax!

It doesn't work as well, because assert stmts "vanish" when running
tests with -O. Luckily,

self.assert_(expression)

works fine in a unittest, with or without -O.

self.assertEqual(e1, e2)

is usually a lot better than

self.assert_(e1 == e2)

too, because the former displays the unequal values when it fails.

Have you seen Holger Krekel's utest?

$ cat test.py
def test_something():
assert "abc"[::-1] == "cab"
$ python2.3 ./std/bin/utest test.py
inserting /home/mwh/src/pypy
_______________________________________________________________________________
______________________________ TESTS STARTING ______________________________
_______________________________________________________________________________
F

_______________________________________________________________________________
________________________________ Test Failure ________________________________

def test_something():

assert "abc"[::-1] == "cab"

-------------------------------------------------------------------------------
assert 'cba' == 'cab'
============================== 1 TESTS FINISHED ==============================
0.06 seconds (0 passed, 1 failed, 0 skipped)

Doesn't seem to be the easiest to install package I've ever used,
mind...

Cheers,
mwh

 

[Thomas Heller]

Have you seen Holger Krekel's utest?

[...]

Nice!
Does it support ruuning test cases in a separate process? That's what
I currently need, and currently implementing on top of unittest...

Thomas

 

[Michael Hudson]

Have you seen Holger Krekel's utest?

[...]

Nice!
Does it support ruuning test cases in a separate process? That's what
I currently need, and currently implementing on top of unittest...

I don't know but I doubt it.

Cheers,
mwh

 

[John Roth]

[John Roth]

I think this is overkill. I find myself using a raw assert much of the time
rather than unittest's specific test methods; it works just as well and I
don't have to remember the blasted syntax!

It doesn't work as well, because assert stmts "vanish" when running
tests with -O.

I'm not sure why I'd ever want to bother compiling
my test modules with -O. (In fact, I'm not sure what
-O does any more. I'm sure it does something besides
removing asserts, but I don't know what.)

John Roth

 

[John Roth]

Only if you want them to be. Floats are a subset
of complex and as such can be treated exactly
the same.

Floats are not a subset of complex. I concur
with Roy Smith's comments on that issue. There's
no reason why complex numbers cannot be
represented by a pair of integers, or rationals,
or the new floating decimal type, or whatever.

In addition, you're going to have to do significant
type checking if you want to combine floats and
complex in the same method call. That's just
poor design.

Why don't you just test whether the second
number lies within a fault tolerance of
5e11 of 6.23e14

I take it you've never worked with FIT. It's intended
to be used by non-technical people on a project, such
as the on-site customer, business analysts and so forth.
If I did something as silly as requiring them to insert
a tolerance that is blatently obvious to anyone looking
at the number involved, I'd get a change request so fast
that the edges would char from the friction.

It would also violate the interface agreement that all
ports of FIT are supposed to conform to.

John Roth

 

[John Roth]

Why don't you just test whether the second
number lies within a fault tolerance of
5e11 of 6.23e14

For one blatently obvious reason. It's harder
to calculate that in the character form than it
is to insert the '5', and you don't have the
information to tell what you wanted if you
convert from character to float first.

My objective here is to ***not have to***
separately enter a precision. The semantics
of saying I want something to be treated as equal
if it's "close enough" to 6.23e14 is that it should
be closer to that than to either 6.24e14 or
6.22e14.

John Roth

 

[Christopher T King]

In addition, you're going to have to do significant
type checking if you want to combine floats and
complex in the same method call. That's just
poor design.

Why would you have to do that? The Python operators, as well as every
function in cmath, transparently support both floats and complex numbers
(as they should). Every operation between floats and complex numbers is
well-defined.