How to unit test random seeds?

P

Patricia Shanahan

I have an application that uses two random number generator instances.
The input parameters can specify the seed for either or both of them. If
a seed is not provided in the input, I use a SecureRandom's generateSeed().

I can obviously unit test the processing of specified seeds. The problem
is how to test my use of generateSeed(). Any ideas?

Patricia
 
M

Mike Schilling

Patricia Shanahan said:
I have an application that uses two random number generator
instances.
The input parameters can specify the seed for either or both of
them. If
a seed is not provided in the input, I use a SecureRandom's
generateSeed().

I can obviously unit test the processing of specified seeds. The
problem
is how to test my use of generateSeed(). Any ideas?

What are you testing for? Obviously the detailed results will vary,
but perhaps it's worth verifying that that algorithm as a whole works
for, say, 100 random seed values.
 
P

Patricia Shanahan

Mike said:
What are you testing for? Obviously the detailed results will vary,
but perhaps it's worth verifying that that algorithm as a whole works
for, say, 100 random seed values.
 
P

Patricia Shanahan

Mike said:
What are you testing for? Obviously the detailed results will vary,
but perhaps it's worth verifying that that algorithm as a whole works
for, say, 100 random seed values.

In general, I am unit testing code that processes parameters and manages
default values. Specifically, the question is whether my implementation
uses generateSeed correctly.

There are several conditions that each have a 1 in 2^64 probability,
such as the two seeds, if both defaulted, being equal, or either seed
being zero. I could test for those things and have a vanishingly small
probability of rejecting a correct result.

Patricia
 
E

Eric Sosman

Patricia said:
I have an application that uses two random number generator instances.
The input parameters can specify the seed for either or both of them. If
a seed is not provided in the input, I use a SecureRandom's generateSeed().

I can obviously unit test the processing of specified seeds. The problem
is how to test my use of generateSeed(). Any ideas?

Idea (maybe not the greatest, and definitely not original):
Some features of a program are amenable to testing, others are
better suited to inspection. If your test proves that "no seed"
does in fact wind up using SecureRandom, just assume SecureRandom
does its job properly.
 
P

Patricia Shanahan

Eric said:
Idea (maybe not the greatest, and definitely not original):
Some features of a program are amenable to testing, others are
better suited to inspection. If your test proves that "no seed"
does in fact wind up using SecureRandom, just assume SecureRandom
does its job properly.

I do assume SecureRandom does its job properly. The question I would
like to test is whether I am using it correctly, as well as the more
general question of how to test one's use of pseudo-random features.

This thread is a last resort before declaring it untestable, at least at
the unit test level, and depending on desk check and system test
instead. At the system test level, I can check questions like whether
repeated runs without specified seeds do behave differently in the
proper ways.

Patricia
 
D

Daniel Dyer

I do assume SecureRandom does its job properly. The question I would
like to test is whether I am using it correctly, as well as the more
general question of how to test one's use of pseudo-random features.

I'm not sure that I am certain what you are asking. Do you want to test
under which circumstances your code makes calls to generateSeed()? In
other words, do you want to assert that generateSeed is called when you
expect it to be called?

Or are you asking how to write tests when you don't know which seed value
your RNGs are using (and therefore can't be certain of the expected
outcomes)?

Dan.
 
P

Patricia Shanahan

Daniel said:
I'm not sure that I am certain what you are asking. Do you want to test
under which circumstances your code makes calls to generateSeed()? In
other words, do you want to assert that generateSeed is called when you
expect it to be called?

After the call to generateSeed their is also some arithmetic to turn its
byte[] into a long for use by Random's constructor. My missing unit test
is for the process of actually calling generateSeed, converting the
result, and constructing the Random.
Or are you asking how to write tests when you don't know which seed
value your RNGs are using (and therefore can't be certain of the
expected outcomes)?

The easiest solution to that is to not do it. For example, I force both
seeds in my regression tests, so they use the same RNG sequences every
time they run. That works for everything except testing the using of
generateSeed to seed a java.util.Random.

Patricia
 
D

Daniel Dyer

I'm not sure that I am certain what you are asking. Do you want to
test under which circumstances your code makes calls to
generateSeed()? In other words, do you want to assert that
generateSeed is called when you expect it to be called?

After the call to generateSeed their is also some arithmetic to turn its
byte[] into a long for use by Random's constructor. My missing unit test
is for the process of actually calling generateSeed, converting the
result, and constructing the Random.

OK, I think I see what your problem is. You want to validate your logic
but can't formulate the assertions because you don't know what
generateSeed is going to return. Is that correct?

Obviously I am unfamiliar with the structure of your code, but the only
thing that comes to mind is to introduce another layer of indirection.
For the RNGs in my Uncommons Maths library, I have a SeedGenerator
strategy interface. This allows me to use seed data from various sources
(/dev/random and random.org, as well as SecureRandom). If you had a
similar arrangement you would be able to substitute your true seed
generator with a mock that returned a known value, purely for the purpose
of testing your arithmetic. You'd have to decide whether the added
complexity is worth it to get the test coverage that you are looking for.

Dan.
 
P

Patricia Shanahan

Daniel said:
I do assume SecureRandom does its job properly. The question I would
like to test is whether I am using it correctly, as well as the more
general question of how to test one's use of pseudo-random features.
I'm not sure that I am certain what you are asking. Do you want to
test under which circumstances your code makes calls to
generateSeed()? In other words, do you want to assert that
generateSeed is called when you expect it to be called?

After the call to generateSeed their is also some arithmetic to turn its
byte[] into a long for use by Random's constructor. My missing unit test
is for the process of actually calling generateSeed, converting the
result, and constructing the Random.

OK, I think I see what your problem is. You want to validate your logic
but can't formulate the assertions because you don't know what
generateSeed is going to return. Is that correct?

That's it.
Obviously I am unfamiliar with the structure of your code, but the only
thing that comes to mind is to introduce another layer of indirection.
For the RNGs in my Uncommons Maths library, I have a SeedGenerator
strategy interface. This allows me to use seed data from various
sources (/dev/random and random.org, as well as SecureRandom). If you
had a similar arrangement you would be able to substitute your true seed
generator with a mock that returned a known value, purely for the
purpose of testing your arithmetic. You'd have to decide whether the
added complexity is worth it to get the test coverage that you are
looking for.

Interesting. That would solve my unit test problem. As you say, there is
a tradeoff between test coverage and layers of indirection.

Thanks,

Patricia
 
P

Patricia Shanahan

Patricia said:
Daniel Dyer wrote: ....

Interesting. That would solve my unit test problem. As you say, there is
a tradeoff between test coverage and layers of indirection.

I thought some more about this, and it can be made very unobtrusive as a
test feature.

I defined a SeedGenerator interface inside my seed-processing class. It
also has a static anonymous inner class object that implements
SeedGenerator using SecureRandom. The seed-processing class has two
constructors, the normal one that defaults to its own SecureRandom
generator, and a package-scope one for testing that allows the caller to
specify the SeedGenerator. I made the interface extremely simple to code
using SecureRandom.

That makes the seed generator selection invisible for normal uses of the
class, but lets its unit test control the seeds completely.

Patricia
 
A

Arne Vajhøj

Patricia said:
After the call to generateSeed their is also some arithmetic to turn its
byte[] into a long for use by Random's constructor. My missing unit test
is for the process of actually calling generateSeed, converting the
result, and constructing the Random.

If you are willing to use heavy artillery to kill a fly, then you
could run your unit test with byte code instrumentation and
replace the SecureRandom class byte code with a class of yours
that delivered a known result. That way you could test that
SecureRandom is being called and that your "post processing"
of the seed is correct.

Arne
 
P

Patricia Shanahan

Arne said:
Patricia said:
After the call to generateSeed their is also some arithmetic to turn its
byte[] into a long for use by Random's constructor. My missing unit test
is for the process of actually calling generateSeed, converting the
result, and constructing the Random.

If you are willing to use heavy artillery to kill a fly, then you
could run your unit test with byte code instrumentation and
replace the SecureRandom class byte code with a class of yours
that delivered a known result. That way you could test that
SecureRandom is being called and that your "post processing"
of the seed is correct.

Yes, I can see that is possible. However, based on earlier suggestions
I've reached a compromise that at least keeps me happy. I use a nested
interface SeedGenerator that is defined so that its implementation in
terms of SecureRandom is trivial. The unit test substitutes its own
SeedGenerator implementation that forces known seed values.

There is still a tiny untested code path that your suggestion would
remove, but I don't think it is worth the added complication.

Patricia
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top