Java - Junit distinguish exceptions

R

Ronny Mandal

Hello, I am relatively new to JUnit. I am instrumenting already
existing tests, what I want to do is to record which exceptions that
(could) occure when running the tests. One sometimes expected
exception is java.lang.SecurityException, i.e. it might be thrown, but
not necessarily. The other does not matter, all that I am interested
in is that an exception has occured.

So my results would be Pass / Fail / SecurityException (Other
exception would normally be recorded as an error. I cannot use
@Test(expected=...), because the test would fail if no exception was
thrown. It is preferrable not to duplicate the test methods, where one
has an expected exception.

Any suggestions is greatly appreciated.


Regards,

Ronny Mandal
 
R

Ronny Mandal

Hello, I am relatively new to JUnit. I am instrumenting already
existing tests, what I want to do is to record which exceptions that
(could) occure when running the tests. One sometimes expected
exception is java.lang.SecurityException, i.e. it might be thrown, but
not necessarily. The other does not matter, all that I am interested
in is that an exception has occured.

So my results would be Pass / Fail / SecurityException (Other
exception would normally be recorded as an error. I cannot use
@Test(expected=...), because the test would fail if no exception was
thrown. It is preferrable not to duplicate the test methods, where one
has an expected exception.

Any suggestions is greatly appreciated.
Strange thing, the answer always emerges when the question is asked.
WHen an exception occurs, a stack trace is yielded. Capture this,
parse it and suddenly the task is done!

/Ronny Mandal
 
L

Lew

Strange thing, the answer always emerges when the question is asked.
WHen an exception occurs, a stack trace is yielded. Capture this,
parse it and suddenly the task is done!

I was planning to suggest:
http://junit.sourceforge.net/doc/faq/faq.htm#tests_7

You can manually catch an exception within the test:

@test public void testFoo()
{
try
{
foo();
}
catch ( ExpectedException exc )
{
logger.log( "ExpectedException occurred", exc );
// eat exception so test passes
}
assertTrue( true );
}
 
J

John

Hello, I am relatively new to JUnit. I am instrumenting already
existing tests, what I want to do is to record which exceptions that
(could) occure when running the tests. One sometimes expected
exception is java.lang.SecurityException, i.e. it might be thrown, but
not necessarily. The other does not matter, all that I am interested
in is that an exception has occured.

So my results would be Pass / Fail / SecurityException (Other
exception would normally be recorded as an error. I cannot use
@Test(expected=...), because the test would fail if no exception was
thrown. It is preferrable not to duplicate the test methods, where one
has an expected exception.

Any suggestions is greatly appreciated.

Regards,

Ronny Mandal

JUnit has a built in way to tell whether an exception was thrown when
it should have been:

@Test(expected=IllegalArgumentException.class)
public final void testThatShouldThrowIllegalArgumentException()
{
assertEquals(1.0, methodUnderTest(illegalValue), 0);
}

If the methodUnderTest(illegalValue) throws an
IllegalArgumentException,
then the test passes. If it doesn't throw the exception, then the test
fails.

Hope this helps,

John
 
L

Lew

Ronny said:
JUnit has a built in way to tell whether an exception was thrown when
it should have been:

     @Test(expected=IllegalArgumentException.class)
     public final void testThatShouldThrowIllegalArgumentException()
     {
         assertEquals(1.0, methodUnderTest(illegalValue), 0);
     }

If the methodUnderTest(illegalValue) throws an
IllegalArgumentException,
then the test passes. If it doesn't throw the exception, then the test
fails.

He wants it to succeed when it throws no exception at all.
 
M

Mike Schilling

Lew said:
He wants it to succeed when it throws no exception at all.

@Test(expected=SecurityException.class)
public final void testThatShouldThrowIllegalArgumentException()
{
assertEquals(1, methodThatMayThrowASecurityException());
throw new SecurityException("Success!");
}
 
M

markspace

JUnit has a built in way to tell whether an exception was thrown when
it should have been:

@Test(expected=IllegalArgumentException.class)


That is very cool, thanks for pointing that out. I've always just
caught the exception like Lew. One advantage of the latter method is I
think that you can test a large number of vectors in a loop. Whereas
using the built in feature necessitates that you exit the test immediately.

Here's an actual live example:

for( String test: testVectors ) {
ByteArrayInputStream ins = new ByteArrayInputStream(
test.getBytes() );
MultiBufferCharSeq instance = new MultiBufferCharSeq( ins, null );

Class<IllegalArgumentException> expResult =
IllegalArgumentException.class;
Class<?> result = null;

try {
char dummy = instance.setIndex( -1 );
} catch( IllegalArgumentException ex ) {
result = ex.getClass();
}
assertSame( expResult, result );
}
}
 
T

Tom Anderson

That is very cool, thanks for pointing that out. I've always just caught the
exception like Lew.

I used to catch exceptions, then i switched to expected=, but now i catch
exceptions again. The expected= method is very easy to go wrong with,
because it necessarily covers the whole method. Consider:

@Test(expected=SecurityException.class)
public void processingAListOfForbiddenFilesThrowsAnException() throws IOException {
File list = File.createTempFile("test", ".txt");
Writer out = new FileWriter(list);
out.write("/root/forbiddenfile.txt\n");
out.close();
new FileProcessor().processFilesInListFile(list); // should throw SecurityException
}

If one of the first four lines manages to throw a SecurityException, this
test will pass, when really, we want it to be an error. If we write it
like this:

@Test
public void processingAListOfForbiddenFilesThrowsAnException() throws IOException {
File list = File.createTempFile("test", ".txt");
Writer out = new FileWriter(list);
out.write("/root/forbiddenfile.txt\n");
out.close();
try {
new FileProcessor().processFilesInListFile(list); // should throw SecurityException
fail("processFilesInListFile should have thrown a SecurityException");
}
catch(SecurityException e) {
// success!
}
}

We don't have that problem.

Admittedly, this problem only manifests itself when your tests include
multiple calls to 'real' code. If you're making zealous use of mocking,
then it is much less of a problem.

Still, i like the fact that the explicit try-catch makes it obvious
exactly where the exception is expected to be thrown. It's one less thing
you have to know about tests, since the testing is handled using a normal
java structure.
One advantage of the latter method is I think that you can test a large
number of vectors in a loop. Whereas using the built in feature
necessitates that you exit the test immediately.

Here's an actual live example:

for( String test: testVectors ) {
ByteArrayInputStream ins = new ByteArrayInputStream(
test.getBytes() );
MultiBufferCharSeq instance = new MultiBufferCharSeq( ins, null );

Class<IllegalArgumentException> expResult =
IllegalArgumentException.class;
Class<?> result = null;

try {
char dummy = instance.setIndex( -1 );
} catch( IllegalArgumentException ex ) {
result = ex.getClass();
}
assertSame( expResult, result );
}
}

I don't see why you couldn't do this with expected=.

tom
 
T

Tom Anderson

Firstly, this doesn't sound like a good test at all. An exception 'might
be' thrown, and that's okay? Why? Why is it okay for the test to do
different things at different times?
Strange thing, the answer always emerges when the question is asked.
WHen an exception occurs, a stack trace is yielded. Capture this,
parse it and suddenly the task is done!

That doesn't sound like a good solution at all. Where are you doing the
capturing and parsing? In the test? In general, you can't really rely on
the stack trace in an exception, since it can vary at the whim of the
optimiser. Plus, parsing stack traces is just a hacky thing to do.

Now, in advance of an answer to my question above, i'm going to make an
assumption about what's going on here. I think that you're running these
tests in different contexts; sometimes, the test can work, but sometimes,
something it needs to do is blocked by the security manager, and you get
the SecurityException. The reason you don't consider this a failure is
that it means the test could not be carried out, not that the test failed.
In that case, you can make use of a somewhat obscure JUnit feature called
'assumptions': an assumpion is just like an assertion, except that if it
fails, then rather than causing the test to fail, it causes it to be
ignored. See:

http://junit.sourceforge.net/javadoc/org/junit/Assume.html

You can write your test like this:

import static org.junit.Assume.assumeNoException;

@test
public void testStuff() {
try {
// whatever
}
catch (SecurityException e) {
assumeNoException(e);
}
}

If you get a SecurityException, the test will be ignored.

A warning: be careful with these. Remember that an ignored test is not
testing anything. Every test that ends up being ignored is something about
your system that is not being tested. Ignores are in a way more dangerous
than failures, because they're so easy to overlook: if you have ninety
passing tests and ten ignored, with no failures, you tend to think your
system is okay, whereas in reality, the features covered by the ten
ignored tests could be disastrously buggy. In general, i think it's better
to have test which cannot be carried out fail rather than be ignored,
because it makes you aware of this problem. Consequently, i'd say the only
time ignored tests are useful is if you're monitoring their number; if an
ignored tests is a cause for concern in the same way as a failed test,
then you can make good use of them. But for most people, ignored tests are
just ignored.

tom
 
M

markspace

I don't see why you couldn't do this with expected=.


Because I need to test that ALL of the vectors throw the exception, and
the FIRST one will terminate the loop if I don't catch the exception.
 
T

Tom Anderson

Because I need to test that ALL of the vectors throw the exception, and
the FIRST one will terminate the loop if I don't catch the exception.

I completely missed that when i read your code. Apologies.

You could still use expected= if you made the test parameterized:

http://www.junit.org/apidocs/org/junit/runners/Parameterized.html

Which would give you a separate test execution per vector; you might find
that gave you more detailed output as well. However, parameterized tests
involve rather a lot of boilerplate, and i very rarely find they're worth
it.

tom
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,733
Messages
2,569,439
Members
44,829
Latest member
PIXThurman

Latest Threads

Top