Understanding relative imports in package - and running pytest withrelative imports?

[Victor Hooi]

Hi,

Ok, this is a topic that I've never really understood properly, so I'd like to find out what's the "proper" way of doing things.

Say I have a directory structure like this:

furniture/
__init__.py
chair/
__init__.py
config.yaml
build_chair.py
common/
__init__.py
shared.py
table/
__init__.py
config.yaml
create_table.sql
build_table.py

The package is called furniture, and we have modules chair, common and table underneath that.

build_chair.py and build_table.py are supposed to import from common/shared.py using relative imports. e.g.:

from ..common.shared import supplies

However, if you then try to run the scripts build_chair.py, or build_table.py, they'll complain about:

ValueError: Attempted relative import in non-package

After some Googling:

http://stackoverflow.com/questions/...ative-import-in-non-package-even-with-init-py
http://stackoverflow.com/questions/72852/how-to-do-relative-imports-in-python
http://stackoverflow.com/questions/...-in-non-package-error-in-spite-of-having-init
http://stackoverflow.com/questions/...-non-package-although-packaes-with-init-py-in
http://melitamihaljevic.blogspot.com.au/2013/04/python-relative-imports-hard-way.html

The advice seems to be either to run it from the parent directory of furniture with:

python -m furniture.chair.build_chair

Or to have a main.py outside of the package directory and run that, and have it import things.

However, I don't see having a separate single main.py outside my package would work with keeping my code tidy/organised, and or how it'd work with the other files (config.yaml, or create_table.sql) which are associated with each script?

A third way I thought of way just to create a setup.py and install the package into site-packages - and then everything will work? However, I don't think that solves my problem of understanding how things work, or getting my directory structure right.

Although apparently running a script inside a package is an anti-pattern? (https://mail.python.org/pipermail/python-3000/2007-April/006793.html)

How would you guys organise the code above?

Also, if I have tests (say with pyttest), inside furniture/table/tests/test_table.py, how would I run these as well? If I run py.test from there, I get the same:

$ py.test
....
from ..table.build_table import Table
E ValueError: Attempted relative import in non-package
....

(Above is just an extract).

Assuming I use pytest, where should my tests be in the directory structure, and how should I be running them?

Cheers,
Victor

 

[Devin Jeanpierre]

The advice seems to be either to run it from the parent directory of furniture with:

python -m furniture.chair.build_chair

Yes. More pedantically, run it from somewhere such that the furniture
package is importable. For example, if you install furniture, then
it's accessible everywhere.

However, I don't see having a separate single main.py outside my package would work with keeping my code tidy/organised, and or how it'd work with the other files (config.yaml, or create_table.sql) which are associated with each script?

The other files are contained within the package. As long as
build_table looks in the right places, they'll still be accessible.
(e.g. http://docs.python.org/2/library/pkgutil#pkgutil.get_data ).

A third way I thought of way just to create a setup.py and install the package into site-packages - and then everything will work? However, I don't think that solves my problem of understanding how things work, or getting my directory structure right.

No, installing furniture won't change anything about how you need to
run it. Either you run a separate script that imports it and runs the
right main function, or you use python -m furniture.foo.bar

Although apparently running a script inside a package is an anti-pattern? (https://mail.python.org/pipermail/python-3000/2007-April/006793.html)

It's not an antipattern, it's just plain broken. Some things stop
working as you'd expect, like exception handling.

How would you guys organise the code above?

If I really cared about the command line interface, I would probably
reorganize something like this:

furniture/
- __init__.py
- __main__.py
- common.py
* chair/
- __init__.py
- build.py
- create.sql
- config.yaml
* furniture/
- __init__.py
- build.py
- create.sql
- config.yaml

And then create chairs by running `python -m furniture build chair`.
furniture/__main__.py would dynamically import furniture.chair.build
and run furniture.chair.build.main().

There would also be an executable I might install separately,
furniture, the contents of which are "from furniture import __main__;
__main__.main()"

Also, if I have tests (say with pyttest), inside furniture/table/tests/test_table.py, how would I run these as well? If I run py.test from there, I get the same:

$ py.test
....
from ..table.build_table import Table
E ValueError: Attempted relative import in non-package
....

(Above is just an extract).

Assuming I use pytest, where should my tests be in the directory structure, and how should I be running them?

As far as I understand py.test and nose, they are designed such that
you don't put your tests as subpackages. If you want to put your tests
as subpackages, use a different unit testing framework, like unittest.
Then you can just do `python -m unittest discover .`

If you do want to use py.test (or nose), then either there is some
flag you need to pass in that I'm not aware of, or your directory
structure should be:

my-project/
- README
- LICENSE
- setup.py
* furniture/
...
* tests/
- test_chair.py
- test_common.py
- test_table.py

Note that tests is not a package either. py.test and nose work by
loading files, NOT modules. So they won't work with loading test
submodules in a package.

For this and other reasons, I personally stay away from nose and py.test.

-- Devin