Test-driven development of random algorithms

[Steven D'Aprano]

I'm working on some functions that, essentially, return randomly generated
strings. Here's a basic example:

def rstr():
"""Return a random string based on a pseudo
normally-distributed random number.
"""
x = 0.0
for i in range(12):
x += random.random()
return str(int(x)+6))

I want to do test-driven development. What should I do? Generally, any
test I do of the form

assert rst() == '1'

will fail more often than not (about 85% of the time, by my estimate). An
easy work around would be to do this:

assert rstr() in [str(n) for n in range(-6, 6)]

but (1) that doesn't scale very well (what if rstr() could return one of
a billion different strings?) and (2) there could be bugs which only show
up probabilistically, e.g. if I've got the algorithm wrong, rstr() might
return '6' once in a while.

Does anyone have generic advice for the testing and development of this
sort of function?

 

[Robert Kern]

I'm working on some functions that, essentially, return randomly generated
strings. Here's a basic example:

def rstr():
"""Return a random string based on a pseudo
normally-distributed random number.
"""
x = 0.0
for i in range(12):
x += random.random()
return str(int(x)+6))

I want to do test-driven development. What should I do? Generally, any
test I do of the form

assert rst() == '1'

will fail more often than not (about 85% of the time, by my estimate). An
easy work around would be to do this:

assert rstr() in [str(n) for n in range(-6, 6)]

but (1) that doesn't scale very well (what if rstr() could return one of
a billion different strings?) and (2) there could be bugs which only show
up probabilistically, e.g. if I've got the algorithm wrong, rstr() might
return '6' once in a while.

Does anyone have generic advice for the testing and development of this
sort of function?

"Design for Testability". In library code, never call the functions in the
random module. Always take as an argument a random.Random instance. When
testing, you can seed your own Random instance and all of your numbers will be
the same for every test run.

This kind of design is A Good Thing(TM) outside of unit tests, too. They aren't
the only places where one might want to have full control over the sequence of
random numbers.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Ben Finney]

"Design for Testability". In library code, never call the functions
in the random module. Always take as an argument a random.Random
instance. When testing, you can seed your own Random instance and
all of your numbers will be the same for every test run.

Even better, you can pass a stub Random instance (that fakes it) or a
mock Random instance (that fakes it, and allows subsequent assertion
that the client code used it as expected). This way, you can ensure
that your fake Random actually gives a sequence of numbers designed to
quickly cover the extremes and corner cases, as well as some normal
cases.

This applies to any externality (databases, file systems, input
devices): in your unit tests, dont pass the real externality. Pass a
convincing fake that will behave entirely predictably, but will
nevertheless exercise the functionality needed for the unit tests.

This is one of the main differences between unit tests and other
kinds. With unit tests, you want each test to exercise as narrow a set
of the client behaviour as feasible. This means eliminating anything
else as a possible source of problems. With other tests -- stress
tests, acceptance tests, and so on -- you want to exercise the entire
application stack, or some significant chunk of it.