unittest: assertRaises() with an instance instead of a type

[Ulrich Eckhardt]

Hi!

I'm currently writing some tests for the error handling of some code. In
this scenario, I must make sure that both the correct exception is
raised and that the contained error code is correct:


try:
foo()
self.fail('exception not raised')
catch MyException as e:
self.assertEqual(e.errorcode, SOME_FOO_ERROR)
catch Exception:
self.fail('unexpected exception raised')


This is tedious to write and read. The docs mention this alternative:


with self.assertRaises(MyException) as cm:
foo()
self.assertEqual(cm.the_exception.errorcode, SOME_FOO_ERROR)


This is shorter, but I think there's an alternative syntax possible that
would be even better:


with self.assertRaises(MyException(SOME_FOO_ERROR)):
foo()


Here, assertRaises() is not called with an exception type but with an
exception instance. I'd implement it something like this:


def assertRaises(self, exception, ...):
# divide input parameter into type and instance
if isinstance(exception, Exception):
exception_type = type(exception)
else:
exception_type = exception
exception = None
# call testee and verify results
try:
...call function here...
except exception_type as e:
if not exception is None:
self.assertEqual(e, exception)


This of course requires the exception to be equality-comparable.


Questions here:
1. Does this sound like a useful extension or am I missing another
obvious solution to my problem?
2. The assertRaises() sketch above tries to auto-detect whether the
given parameter is the type or an instance. Given the highly dynamic
nature of Python, an object can be both instance and type, is the above
detection mechanism reliable?


Of course I'm open for other suggestions to solve my problem. One that I
thought of but which I haven't really looked into was to modify __init__
or __new__ of my exception class to return an instance of a derived
class that uniquely identifies the error. I.e.
MyException(SOME_FOO_ERROR) would not create a MyException instance but
a MyFooErrorException instance (which has MyException as a baseclass).
In that case, the existing code that checks for the right exception type
would suffice for my needs.


Cheers everybody!

Uli

 

[Steven D'Aprano]

Hi!

I'm currently writing some tests for the error handling of some code. In
this scenario, I must make sure that both the correct exception is
raised and that the contained error code is correct:


try:
foo()
self.fail('exception not raised')
catch MyException as e:
self.assertEqual(e.errorcode, SOME_FOO_ERROR)
catch Exception:
self.fail('unexpected exception raised')

First off, that is not Python code. "catch Exception" gives a syntax
error.

Secondly, that is not the right way to do this unit test. You are testing
two distinct things, so you should write it as two separate tests:


def testFooRaisesException(self):
# Test that foo() raises an exception.
self.assertRaises(MyException, foo)


If foo does *not* raise an exception, the unittest framework will handle
the failure for you. If it raises a different exception, the framework
will also handle that too.

Then write a second test to check the exception code:

def testFooExceptionCode(self):
# Test that foo()'s exception has the right error code.
try:
foo()
except MyException as err:
self.assertEquals(err.errorcode, SOME_FOO_ERROR)


Again, let the framework handle any unexpected cases.

If you have lots of functions to test, write a helper function:

def catch(exception, func, *args, **kwargs):
try:
func(*args, **kwargs)
except exception as err:
return err
raise RuntimeError('no error raised')


and then the test becomes:

def testFooExceptionCode(self):
# Test that foo()'s exception has the right error code.
self.assertEquals(
catch(MyException, foo).errorcode, SOME_FOO_ERROR
)



(By the way, I have to question the design of an exception with error
codes. That seems pretty poor design to me. Normally the exception *type*
acts as equivalent to an error code.)

 

[Terry Reedy]

Hi!

I'm currently writing some tests for the error handling of some code. In
this scenario, I must make sure that both the correct exception is
raised and that the contained error code is correct:


try:
foo()
self.fail('exception not raised')
catch MyException as e:
self.assertEqual(e.errorcode, SOME_FOO_ERROR)
catch Exception:
self.fail('unexpected exception raised')


This is tedious to write and read. The docs mention this alternative:


with self.assertRaises(MyException) as cm:
foo()
self.assertEqual(cm.the_exception.errorcode, SOME_FOO_ERROR)

Exceptions can have multiple attributes. This allows the tester to
exactly specify what attributes to test.

This is shorter, but I think there's an alternative syntax possible that
would be even better:

with self.assertRaises(MyException(SOME_FOO_ERROR)):
foo()

I presume that if this worked the way you want, all attributes would
have to match. The message part of builtin exceptions is allowed to
change, so hard-coding an exact expected message makes tests fragile.
This is a problem with doctest.

Here, assertRaises() is not called with an exception type but with an
exception instance. I'd implement it something like this:

def assertRaises(self, exception, ...):
# divide input parameter into type and instance
if isinstance(exception, Exception):
exception_type = type(exception)
else:
exception_type = exception
exception = None
# call testee and verify results
try:
...call function here...
except exception_type as e:
if not exception is None:
self.assertEqual(e, exception)

Did you use tabs? They do not get preserved indefinitely, so they are
bad for posting.

This of course requires the exception to be equality-comparable.

Equality comparison is by id. So this code will not do what you want.

You can, of course, write a custom AssertX subclass that at least works
for your custom exception class.

 

[Steven D'Aprano]

Have a look at Python's built-in OSError. The various errors from the
operating system can only be distinguished by the numeric code the OS
returns, so that's what to test on in one's unit tests.

I'm familiar with OSError. It is necessary because OSError is a high-
level interface to low-level C errors. I wouldn't call it a good design
though, I certainly wouldn't choose it if we were developing an error
system from scratch and weren't constrained by compatibility with a more
primitive error model (error codes instead of exceptions).

The new, revamped exception hierarchy in Python 3.3 will rationalise much
(but not all) for this, unifying IOError and OSError and making error
codes much less relevant:


http://www.python.org/dev/peps/pep-3151/

 

[Peter Otten]

Have a look at Python's built-in OSError. The various errors from the
operating system can only be distinguished by the numeric code the OS
returns, so that's what to test on in one's unit tests.

The core devs are working to fix that:

$ python3.2 -c'open("does-not-exist")'
Traceback (most recent call last):
File "<string>", line 1, in <module>
IOError: [Errno 2] No such file or directory: 'does-not-exist'
$ python3.3 -c'open("does-not-exist")'
Traceback (most recent call last):
File "<string>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'does-not-exist'

$ python3.2 -c'open("unwritable", "w")'
Traceback (most recent call last):
File "<string>", line 1, in <module>
IOError: [Errno 13] Permission denied: 'unwritable'
$ python3.3 -c'open("unwritable", "w")'
Traceback (most recent call last):
File "<string>", line 1, in <module>
PermissionError: [Errno 13] Permission denied: 'unwritable'

http://www.python.org/dev/peps/pep-3151/

 

[Ulrich Eckhardt]

Am 28.03.2012 20:07, schrieb Steven D'Aprano:

First off, that is not Python code. "catch Exception" gives a syntax
error.

Old C++ habits... :|

Secondly, that is not the right way to do this unit test. You are testing
two distinct things, so you should write it as two separate tests: [..code..]
If foo does *not* raise an exception, the unittest framework will handle
the failure for you. If it raises a different exception, the framework
will also handle that too.

Then write a second test to check the exception code: [...]
Again, let the framework handle any unexpected cases.

Sorry, you got it wrong, it should be three tests:
1. Make sure foo() raises an exception.
2. Make sure foo() raises the right exception.
3. Make sure the errorcode in the exception is right.

Or maybe you should in between verify that the exception raised actually
contains an errorcode? And that the errorcode can be equality-compared
to the expected value? :>

Sorry, I disagree that these steps should be separated. It would blow up
the code required for testing, increasing maintenance burdens. Which
leads back to a solution that uses a utility function, like the one you
suggested or the one I was looking for initially.

(By the way, I have to question the design of an exception with error
codes. That seems pretty poor design to me. Normally the exception *type*
acts as equivalent to an error code.)

True. Normally. I'd adapting to a legacy system though, similar to
OSError, and that system simply emits error codes which the easiest way
to handle is by wrapping them.


Cheers!

Uli

 

[Ulrich Eckhardt]

Am 28.03.2012 20:26, schrieb Terry Reedy:

On 3/28/2012 8:28 AM, Ulrich Eckhardt wrote: [...]

# call testee and verify results
try:
...call function here...
except exception_type as e:
if not exception is None:
self.assertEqual(e, exception)

Did you use tabs? They do not get preserved indefinitely, so they are
bad for posting.

I didn't consciously use tabs, actually I would rather avoid them. That
said, my posting looks correctly indented in my "sent" folder and also
in the copy received from my newsserver. What could also have an
influence is line endings. I'm using Thunderbird on win32 here, acting
as news client to comp.lang.python. Or maybe it's your software (or
maybe some software in between) that fails to preserve formatting.

*shrug*

Uli

 

[Ulrich Eckhardt]

Am 28.03.2012 20:26, schrieb Terry Reedy:

I presume that if this worked the way you want, all attributes would
have to match. The message part of builtin exceptions is allowed to
change, so hard-coding an exact expected message makes tests fragile.
This is a problem with doctest.

I would have assumed that comparing two exceptions leaves out messages
that are intended for the user, not as part of the API. However, my
expectations aren't met anyway, because ...

Equality comparison is by id. So this code will not do what you want.

False

Yikes! That was unexpected and completely changes my idea. Any clue
whether this is intentional? Is identity the fallback when no equality
is defined for two objects?

Thanks for your feedback!

Uli

 

[Peter Otten]

True. Normally. I'd adapting to a legacy system though, similar to
OSError, and that system simply emits error codes which the easiest way
to handle is by wrapping them.

If you have

err = some_func()
if err:
raise MyException(err)

the effort to convert it to

exc = lookup_exception(some_func())
if exc:
raise exc

is small. A fancy way is to use a decorator:

#untested
def code_to_exception(table):
def deco(f):
def g(*args, **kw):
err = f(*args, **kw)
exc = table[err]
if exc is not None:
raise exc
return g
return f

class MyError(Exception): pass
class HyperspaceBypassError(MyError): pass

@code_to_exception({42: HyperspaceBypassError, 0: None})
def some_func(...):
# ...

 

[Roy Smith]

I didn't consciously use tabs, actually I would rather avoid them. That
said, my posting looks correctly indented in my "sent" folder and also
in the copy received from my newsserver. What could also have an
influence is line endings. I'm using Thunderbird on win32 here, acting
as news client to comp.lang.python. Or maybe it's your software (or
maybe some software in between) that fails to preserve formatting.

*shrug*

Oh noes! The line eater bug is back!

 

[Terry Reedy]

False

Yikes! That was unexpected and completely changes my idea. Any clue
whether this is intentional? Is identity the fallback when no equality
is defined for two objects?

Yes. The Library Reference 4.3. Comparisons (for built-in classes) puts
is this way.
"Objects of different types, except different numeric types, never
compare equal. Furthermore, some types (for example, function objects)
support only a degenerate notion of comparison where any two objects of
that type are unequal." In other words, 'a==b' is the same as 'a is b'.
That is also the default for user-defined classes, but I am not sure
where that is documented, if at all.

 

[Dave Angel]

Am 28.03.2012 20:26, schrieb Terry Reedy:

On 3/28/2012 8:28 AM, Ulrich Eckhardt wrote: [...]

# call testee and verify results
try:
...call function here...
except exception_type as e:
if not exception is None:
self.assertEqual(e, exception)

Did you use tabs? They do not get preserved indefinitely, so they are
bad for posting.

I didn't consciously use tabs, actually I would rather avoid them.
That said, my posting looks correctly indented in my "sent" folder and
also in the copy received from my newsserver. What could also have an
influence is line endings. I'm using Thunderbird on win32 here, acting
as news client to comp.lang.python. Or maybe it's your software (or
maybe some software in between) that fails to preserve formatting.

*shrug*

Uli

More likely, you failed to tell Thunderbird to send it as text. Html
messages will read differently on html aware readers than on the
standard text readers. They also take maybe triple the space and bandwidth.

In thunderbird 3.1.19
In Edit->Preferences, Composition->general Configure Text Format
Behavior -> SendOptions In that dialog, under Text Format, choose
Convert the message to plain text. Then in the tab called "Plain text
domains", add python.org

 

[Terry Reedy]

Am 28.03.2012 20:26, schrieb Terry Reedy:

On 3/28/2012 8:28 AM, Ulrich Eckhardt wrote: [...]

# call testee and verify results
try:
...call function here...
except exception_type as e:
if not exception is None:
self.assertEqual(e, exception)

Did you use tabs? They do not get preserved indefinitely, so they are
bad for posting.

I didn't consciously use tabs, actually I would rather avoid them. That
said, my posting looks correctly indented in my "sent" folder and also
in the copy received from my newsserver. What could also have an
influence is line endings. I'm using Thunderbird on win32 here, acting
as news client to comp.lang.python.

I am using Thunderbird, win64, as news client for gmane. The post looked
fine as originally received. The indents only disappeared when I hit
reply and the >s were added. That does not happen, in general, for other
messages. Unfortunately I cannot go back and read that message as
received because the new version of Tbird is misbehaving and deleting
read messages on close even though I asked to keep them 6 months. I will
look immediately when I next see indents disappearing.

 

[Ethan Furman]

Secondly, that is not the right way to do this unit test. You are testing
two distinct things, so you should write it as two separate tests:

I have to disagree -- I do not see the advantage of writing a second
test that *will* fail if the first test fails as opposed to bundling
both tests together, and having one failure.

~Ethan~

 

[Steven D'Aprano]

Am 28.03.2012 20:07, schrieb Steven D'Aprano:

Secondly, that is not the right way to do this unit test. You are
testing two distinct things, so you should write it as two separate
tests: [..code..]
If foo does *not* raise an exception, the unittest framework will
handle the failure for you. If it raises a different exception, the
framework will also handle that too.

Then write a second test to check the exception code: [...]
Again, let the framework handle any unexpected cases.

Sorry, you got it wrong, it should be three tests: 1. Make sure foo()
raises an exception. 2. Make sure foo() raises the right exception. 3.
Make sure the errorcode in the exception is right.

Or maybe you should in between verify that the exception raised actually
contains an errorcode? And that the errorcode can be equality-compared
to the expected value? :>

Of course you are free to slice it even finer if you like:

testFooWillRaiseSomethingButIDontKnowWhat
testFooWillRaiseMyException
testFooWillRaiseMyExceptionWithErrorcode
testFooWillRaiseMyExceptionWithErrorcodeWhichSupportsEquality
testFooWillRaiseMyExceptionWithErrorcodeEqualToFooError

Five tests

To the degree that the decision of how finely to slice tests is a matter
of personal judgement and/or taste, I was wrong to say "that is not the
right way". I should have said "that is not how I would do that test".

I believe that a single test is too coarse, and three or more tests is
too fine, but two tests is just right. Let me explain how I come to that
judgement.

If you take a test-driven development approach, the right way to test
this is to write testFooWillFail once you decide that foo() should raise
MyException but before foo() actually does so. You would write the test,
the test would fail, and you would fix foo() to ensure it raises the
exception. Then you leave the now passing test in place to detect
regressions.

Then you do the same for the errorcode. Hence two tests.

Since running tests is (usually) cheap, you never bother going back to
remove tests which are made redundant by later tests. You only remove
them if they are made redundant by chances to the code. So even though
the first test is made redundant by the second (if the first fails, so
will the second), you don't remove it.

Why not? Because it guards against regressions. Suppose I decide that
errorcode is no longer needed, so I remove the test for errorcode. If I
had earlier also removed the independent test for MyException being
raised, I've now lost my only check against regressions in foo().

So: never remove tests just because they are redundant. Only remove them
when they are obsolete due to changes in the code being tested.

Even when I don't actually write the tests in advance of the code, I
still write them as if I were. That usually makes it easy for me to
decide how fine grained the tests should be: since there was never a
moment when I thought MyException should have an errorcode attribute, but
not know what that attribute would be, I don't need a *separate* test for
the existence of errorcode.

(I would only add such a separate test if there was a bug that sometimes
the errorcode does not exist. That would be a regression test.)

The question of the exception type is a little more subtle. There *is* a
moment when I knew that foo() should raise an exception, but before I
decided what that exception would be. ValueError? TypeError? Something
else? I can write the test before making that decision:

def testFooRaises(self):
try:
foo()
except: # catch anything
pass
else:
self.fail("foo didn't raise")


However, the next step is broken: I have to modify foo() to raise an
exception, and there is no "raise" equivalent to the bare "except", no
way to raise an exception without specifying an exception type.

I can use a bare raise, but only in response to an existing exception. So
to raise an exception at all, I need to decide what exception that will
be. Even if I start with a placeholder "raise BaseException", and test
for that, when I go back and change the code to "raise MyException" I
should change the test, not create a new test.

Hence there is no point is testing for "any exception, I don't care what"
since I can't write code corresponding to that test case. Hence, I end up
with two tests, not three and certainly not five.

 

[Steven D'Aprano]

I have to disagree -- I do not see the advantage of writing a second
test that *will* fail if the first test fails as opposed to bundling
both tests together, and having one failure.

Using that reasoning, your test suite should contain *one* ginormous test
containing everything:

def testDoesMyApplicationWorkPerfectly(self):
# TEST ALL THE THINGS!!!
...


since *any* failure in any part will cause cascading failures in every
other part of the software which relies on that part. If you have a tree
of dependencies, a failure in the root of the tree will cause everything
to fail, and so by your reasoning, everything should be in a single test.

I do not agree with that reasoning, even when the tree consists of two
items: an exception and an exception attribute.

The problem of cascading test failures is a real one. But I don't believe
that the solution is to combine multiple conceptual tests into a single
test. In this case, the code being tested covers two different concepts:

1. foo() will raise MyException. Hence one test for this.

2. When foo() raises MyException, the exception instance will include an
errorcode attribute with a certain value. This is conceptually
separate from #1 above, even though it depends on it.

Why is it conceptually separate? Because there may be cases where the
caller cares about foo() raising MyException, but doesn't care about the
errorcode. Hence errorcode is dependent but separate, and hence a
separate test.

 

[Ulrich Eckhardt]

Am 29.03.2012 17:25, schrieb Terry Reedy:

I am using Thunderbird, win64, as news client for gmane. The post looked
fine as originally received. The indents only disappeared when I hit
reply and the >s were added.

I can confirm this misbehaviour of Thunderbird (version 11.0 here), it
strips the leading spaces when you begin a reply.

Uli

 

[Dave Angel]

Am 29.03.2012 17:25, schrieb Terry Reedy:

I can confirm this misbehaviour of Thunderbird (version 11.0 here), it
strips the leading spaces when you begin a reply.

Uli

But since it doesn't do it on all messages, have you also confirmed that
it does it for a text message? My experience seems to be that only the
html messages are messed up that way.

of course, it could be lots of other things, like which gateways did the
message go through, was it originally sent via the google-mars bridge, etc.

 

[Ethan Furman]

To the degree that the decision of how finely to slice tests is a matter
of personal judgement and/or taste, I was wrong to say "that is not the
right way". I should have said "that is not how I would do that test".

I believe that a single test is too coarse, and three or more tests is
too fine, but two tests is just right. Let me explain how I come to that
judgement.

If you take a test-driven development approach, the right way to test
this is to write testFooWillFail once you decide that foo() should raise
MyException but before foo() actually does so. You would write the test,
the test would fail, and you would fix foo() to ensure it raises the
exception. Then you leave the now passing test in place to detect
regressions.

Then you do the same for the errorcode. Hence two tests.
[snip]

So: never remove tests just because they are redundant. Only remove them
when they are obsolete due to changes in the code being tested.

Very persuasive argument -- I now find myself disposed to writing two
tests (not three, nor five .

~Ethan~

 

[Ulrich Eckhardt]

Am 30.03.2012 14:47, schrieb Dave Angel:

But since it doesn't do it on all messages, have you also confirmed that
it does it for a text message? My experience seems to be that only the
html messages are messed up that way.

I can't find any HTML in what I posted, so HTML is not the problem. A
difference could be the content type. I had in my posting:

Content-Type: text/plain; charset=ISO-8859-15; format=flowed

Another one titled "Pipelining in Python", where TB doesn't mess up the
formatting, has:

Content-Type: text/plain; charset=ISO-8859-1

Searching the web turned up [1], the gist is that "format=flowed" means
that your mailer is allowed to move linebreaks and quotation signs (">
") as it wants. Which is not what I meant. Some more search turned up
[2], which tells us how to disable this. Go to the settings, advanced
section and find the button that fires up the raw configuration editor.
There, locate the key mailnews.send_plaintext_flowed and change the
according value to false.

# Checking...
if this.worked:
hurray("I didn't even have to close the message in writing")


Uli


[1] http://joeclark.org/ffaq.html
[2] http://www.firstpr.com.au/web-mail/Mozilla-mail/