Usual practice: running/testing modules in a package

[alito]

Hi all,

I am new to using packages to group my modules. I can't figure out
how to run a module that uses relative imports without writing a
wrapper that imports that module. Everything I try it complains that
I am attempting a relative import in a non-package.

eg
~/python/testpackage$ ls
config.py importer.py __init__.py

~/python/testpackage$ cat importer.py
from . import config
config.hello()

~/python/testpackage$ cat config.py
def hello():
print 'hello'

__init__.py is empty

~/python/testpackage$ python -V
Python 2.5.2

~/python/testpackage$ echo $PYTHONPATH
..:/home/ale/python/libs:/home/ale/lib/python/:

~/python/testpackage$ python importer.py
Traceback (most recent call last):
File "importer.py", line 1, in <module>
from . import config
ValueError: Attempted relative import in non-package

Ok, fair enough. There's something about running with -m for running
modules:

~/python/testpackage$ python -m importer
Traceback (most recent call last):
File "/usr/lib/python2.5/runpy.py", line 95, in run_module
filename, loader, alter_sys)
File "/usr/lib/python2.5/runpy.py", line 52, in _run_module_code
mod_name, mod_fname, mod_loader)
File "/usr/lib/python2.5/runpy.py", line 32, in _run_code
exec code in run_globals
File "/home/ale/python/testpackage/importer.py", line 1, in <module>
from . import config
ValueError: Attempted relative import in non-package

Ok, maybe it can't see the __init__.py because it is in the current
directory. Go up one.

~/python$ python testpackage/importer.py
Traceback (most recent call last):
File "testpackage/importer.py", line 1, in <module>
from . import config
ValueError: Attempted relative import in non-package

No. What about:
~/python$ python -m testpackage.importer
Traceback (most recent call last):
File "/usr/lib/python2.5/runpy.py", line 95, in run_module
filename, loader, alter_sys)
File "/usr/lib/python2.5/runpy.py", line 52, in _run_module_code
mod_name, mod_fname, mod_loader)
File "/usr/lib/python2.5/runpy.py", line 32, in _run_code
exec code in run_globals
File "/home/ale/python/testpackage/importer.py", line 1, in <module>
from . import config
ValueError: Attempted relative import in non-package

That one is very puzzling. It knows that testpackage is a package
because I am invoking it with testpackage.importer, but still thinks
it isn't a package.

A wrapper on the level up works:

~/python$ cat importercaller.py
from testpackage import config
config.hello()

~/python$ python importercaller.py
hello

So, how do I run these modules without writing a wrapper script for
each one? My main use actually would be to get pylint to analyse
them, but it seems to use the python import system, so it fails
whenever python fails. If anyone can tell me how to get pychecker to
even be able to use a wrapper, it would also be very beneficial.

 

[Gabriel Genellina]

Hi all,

I am new to using packages to group my modules. I can't figure out
how to run a module that uses relative imports without writing a
wrapper that imports that module. Everything I try it complains that
I am attempting a relative import in a non-package.

eg
~/python/testpackage$ ls
config.py importer.py __init__.py

~/python/testpackage$ cat importer.py
from . import config
config.hello()

~/python/testpackage$ cat config.py
def hello():
print 'hello'

__init__.py is empty

~/python/testpackage$ python importer.py
Traceback (most recent call last):
File "importer.py", line 1, in <module>
from . import config
ValueError: Attempted relative import in non-package

[...more attempts...]

So, how do I run these modules without writing a wrapper script for
each one? My main use actually would be to get pylint to analyse
them, but it seems to use the python import system, so it fails
whenever python fails. If anyone can tell me how to get pychecker to
even be able to use a wrapper, it would also be very beneficial.

A package is a library, meant to be imported by some other code. Your main script (or the testing code) is a program, it uses (i.e. imports) the library.
I usually put the test code or the main entry point one level above the package in the development environment. This way, "import testpackage" works almost the same as it would in the production environment (where the package is installed, probably in site-packages)

 

[med.swl]

A package is a library, meant to be imported by some other code. Your main script (or the testing code) is a program, it uses (i.e. imports) the library.

You are right that a module is a library and its main use is to be
imported from another module or program. But is a common practice in
Python to introduce testing code as part as the module itself. Even
there is an option “-m” to do that.
So (in my humbled opinion), that is one error Python must fix.

 

[Gabriel Genellina]

You are right that a module is a library and its main use is to be
imported from another module or program. But is a common practice in
Python to introduce testing code as part as the module itself. Even
there is an option “-m†to do that.
So (in my humbled opinion), that is one error Python must fix.

A *module* or a *package*? The OP asked how to test a package, and a
package is clearly a library. In the final application, it will be
imported. So the test code should mimic the same environment, and import
the package.

The easiest way to import a package is to run a script from its container
directory: the current directory is already in sys.path so the package
directory is found directly without messing with sys.path, PYTHONPATH
variable and such. This is my suggested approach, but there are many other
alternatives. Anyway, the idea is to import the package to test it.

If you don't import the package and just run some script from inside it,
how would Python know that it belongs to a package? Looking for a
__init__.py in the same directory? what if __init__.py contains code? when
should it be executed?

 

[Gabriel Genellina]

Thats what I usually do to solve the "problem", but for my taste it's
better to write the test code of a module inside it.
The code I write in "__init__.py" is related to structures of
initializations, globals at package level.

if __name__ == "__main__":
pass # Here I test only concepts related to the entire package,
not to any module.

I think you misunderstood the comment.
Suppose this setup:

SomeDir/
SomePackage/
__init__.py
module1.py
module2.py
test.py

When someone executes test.py directly, currently Python doesn't know that it is contained inside a package, and some forms of relative imports don't work.
To make relative imports work, Python should be aware that test.py is contained inside a package. Looking for an __init__.py file at the same directory may reveal that there is a package - but that's not enough, because when a package is actually imported, its __init__.py is executed and a new module object is placed in sys.modules.
So, when test.py is run directly, what to do with the code in __init__.py? should it be executed, and when? I don't know the actual reasons, but this seems enough trouble to me to NOT automatically recognize a package unless someone actually imports it.

The application will import the package anyway, so why would the test code not do the same thing? I want to mimic the production environment as closely as possible in the testing environment. And the easiest way to do that is to "import SomePackage" in a script placed at SomeDir. The actual tests may reside in test.py, of course, but now Python *knows* that test.py lives inside a package and relative imports work fine now.

 

[Gabriel Genellina]

--- El dom 24-ago-08, Medardo Rodriguez (Merchise Group)
<[email protected]> escribió:

(please keep posting on this list)

On Sun, Aug 24, 2008 at 1:18 AM, Gabriel Genellina

I didn't. The problem is my bad English

I'm not a native English speaker either...

What you say in this mail is valid, my point is not related
to the way
we organize a python package in submodules and how it's
executed the
"__init__.py" to initialize it or to define
structures a package
level.
My problem is about style. It is my wish to implement
module tests in
the same module after "if __name__ ==
'__main__':" and to test each
module in every moment I feel nice to do it (I widely use
"Test-driven
development").
Of course I move the test code to package
"__init__.py" when it
depends on relative imports. Maybe the solution is to
formalize a way
in order IDE plug-ins, like Eclipse PyDev could make this
easy.

I see. Unfortunately it doesn't work in this situation:

- you want to test a module inside a package
- the module uses relative imports
- you want the test code to reside on the same module, using "if __name__
== '__main__'"

Because if you run the module as a script (to execute the "if __name__ ==
...." section) Python can't recognize that the module is contained inside a
package, and in that case relative imports don't work.
You may avoid any of the last two conditions:

- don't use relative imports. That might work, but leaves another problem:
in the production environment, the module *will* be part of a package. In
the testing environment, it is not. The difference may be relevant in some
cases (relative imports are just one of the differences).

- don't put the test code on the same module. Use another script that
first imports the package and then executes the test. As you said,
probably some Eclipse plugin can help automate this setup - I don't know,
I don't use Eclipse myself. PyScripter does support it
<http://pyscripter.googlecode.com/>

 

[Casey]

A wrapper on the level up works:

~/python$ cat importercaller.py
from testpackage import config
config.hello()

~/python$ python importercaller.py
hello

So, how do I run these modules without writing a wrapper script for
each one?

I just ran into this problem trying to run a package from a local
directory. I agree creating another script just to import the local
package is a bad solution. Without another option, I would recommend
using a Makefile and creating a target to import the package and
kickoff the execution:

hello:
python -c 'from testpackage import config; config.hello()'

and then "make hello" to execute the command.

 

[Casey]

It looks like PEP 366 [http://www.python.org/dev/peps/pep-0366/] will
solve this issue.