PEP 299 and unit testing

[Ben Finney]

Howdy all,

PEP 299 <URL:http://www.python.org/dev/peps/pep-0299> details an
enhancement for entry points to Python programs: a module attribute
(named '__main__') that will be automatically called if the module is
run as a program.

The PEP has status "Rejected", citing backward-compatibility issues,
and Guido's pronouncement that "It's not worth the change (in docs,
user habits, etc.) and there's nothing particularly broken."

I don't deny the backward-compatibility issues in the cited
discussion, but I'd like to point out one thing that is broken by
this: unit testing of program modules.


Unit tests need to import a module and introspectively test small
units from the module to verify their behaviour in isolation. The
boundary of a unit test is the code that's actually in the module
under test: any functional code in that module needs to be tested by
the module's unit test, any code not in that module is outside the
scope of that unit test module.

The logical extension of this is to put *all* functional code into
discrete units, including the "main line" code that gets executed when
the module is run as a program. This leads to code of the type
discussed in PEP 299:

def main(argv):
""" Do the main stuff of this program """
parse_commandline(argv)
try:
do_interesting_things()
except SystemExit, e:
exitcode = e.code
return exitcode

if __name__ == "__main__":
import sys
exitcode = main(sys.argv)
sys.exit(exitcode)

This allows the module's 'main' function to be called as a discrete
unit from the unit test module; the unit test passes in 'argv' as
desired, and fakes out other units that aren't being tested.

What it doesn't allow is for the testing of the 'if __name__ ==
"__main__":' clause itself. No matter how simple we make that, it's
still functional code that can contain errors, be they obvious or
subtle; yet it's code that *can't* be touched by the unit test (by
design, it doesn't execute when the module is imported), leading to
errors that won't be caught as early or easily as they might.

So, I'd argue that "nothing particularly broken" isn't true: unit
testing is flawed in this scenario. It means that even the simple
metric of statement-level test coverage can't ever get to 100%, which
is a problem since it defeats a simple goal of "get all functional
code covered by unit tests".


On the other hand, if PEP 299 *were* implemented (and the
backward-compatibility issues solved), the above could be written as:

def __main__(argv):
""" Do the main stuff of this program """
parse_commandline(argv)
try:
do_interesting_things()
except SystemExit, e:
exitcode = e.code
return exitcode

with no module-level 'if __name__' test at all, and therefore no
functional code unreachable by the unit test module. The effect of the
program is the same, but the invocation of the '__main__' function
isn't left to be implemented in every single program, separately and
subject to error in every case. Instead, it becomes part of the
*external* environment of the module, and is trivially outside the
scope of a unit test module for that program.

 

[Steven Bethard]

What it doesn't allow is for the testing of the 'if __name__ ==
"__main__":' clause itself. No matter how simple we make that, it's
still functional code that can contain errors, be they obvious or
subtle; yet it's code that *can't* be touched by the unit test (by
design, it doesn't execute when the module is imported), leading to
errors that won't be caught as early or easily as they might.

You could always use runpy.run_module.

STeVe

 

[Ben Finney]

You could always use runpy.run_module.

For values of "always" that include Python 2.5, of course. (I'm still
coding to Python 2.4, until 2.5 is more widespread.)

Thanks! I was unaware of that module. It does seem to nicely address
the issue I discussed.

 

[Steven Bethard]

For values of "always" that include Python 2.5, of course. (I'm still
coding to Python 2.4, until 2.5 is more widespread.)

Thanks! I was unaware of that module. It does seem to nicely address
the issue I discussed.

You might try the runpy module as-is with Python 2.4. I don't know if
it works, but it's pure Python so it's worth a try.

STeVe

 

[Ben Finney]

You might try the runpy module as-is with Python 2.4. I don't know
if it works, but it's pure Python so it's worth a try.

Drat. It uses (by explicit design) "the standard import mechanism" to
load the module, which means it doesn't work for exactly the thing I'm
trying to do: load a program file *not* named with a '.py' suffix.

I've long been able to load my program modules from no-suffix
filenames (or indeed any non-standard filenames) with this function::

def make_module_from_file(module_name, file_name):
""" Make a new module object from the code in specified file """

from types import ModuleType
module = ModuleType(module_name)

module_file = open(file_name, 'r')
exec module_file in module.__dict__
sys.modules[module_name] = module

return module

Unfortunately, it seems that "module is already present with name
'foo' in 'sys.modules'" is insufficient for the Python import
mechanism. The module loader used by 'runpy' still complains that it
can't find the module, which is no surprise because its filename is
not that of a library module.

Perhaps I need to delve into the details of the import mechanism
myself :-(

 

[Ben Finney]

Thanks! I was unaware of that module. It does seem to nicely address
the issue I discussed.

Thinking about it further: I don't think it does address the issue.

Running the *entire* module code again in a single step (as
'run_module' seems to do) would happily overwrite any instrumented
faked attributes of the module that were inserted for the purpose of
unit testing, rendering it useless for unit test purposes.

The issue here is that there is an irreducible amount of functional
code inside the module that cannot be unit tested without running the
entire program with all its side effects.

PEP 299 promises to make that specific small-but-significant code
become an implementation detail in the language runtime, which would
mean it would no longer be prone to errors in the modules themselves,
and thus no longer the topic of a unit test on those modules. I think
100% statement coverage is not possible in Python programs without
this, or something that achieves the same thing.