Organising packages/modules - importing functions from a common.py ina separate directory?

[Victor Hooi]

Hi,

I have a collection of Python scripts I'm using to load various bits of data into a database.

I'd like to move some of the common functions (e.g. to setup loggers, reading in configuration etc.) into a common file, and import them from there.

I've created empty __init__.py files, and my current directory structure looks something like this:

foo_loading/
__init__.py
common/
common_foo.py
em_load/
__init__.py
config.yaml
sync_em.py
pg_load/
__init__.py
config.yaml
sync_pg.py

So from within the sync_em.py script, I'm trying to import a function from foo_loading/common/common_foo.py.

from ..common.common_foo import setup_foo_logging

I get the error:

ValueError: Attempted relative import in non-package

If I change directories to the parent of "foo_loading", then run

python -m foo_loading.em_load.sync_em sync_em.py

it works. However, this seems a bit roundabout, and I suspect I'm not doing things correctly.

Ideally, I want a user to be able to just run sync_em.py from it's own directory, and have it correctly import the logging/config modules from common_foo.py, and just work.

What is the correct way to achieve this?

Secondly, if I want to move all of the config.yaml files to a common foo_loading/config.yaml, or even foo_loading/config/config.yaml, what is the correct way to access this from within the scripts? Should I just be using "../", or is there a better way?

Cheers,
Victor

 

[Victor Hooi]

Hi,

Ok, so I should be using absolute imports, not relative imports.

Hmm, I just tried to use absolute imports, and it can't seem to locate the modules:

In the file "foo_loading/em_load/sync_em.py", I have:

from common.common_bex import setup_foo_logging

When I try to run that script:

python sync_em.py

I get:

ImportError: No module named common.common_foo

I've also tried adding "foo_loading" (the package name):

from foo_loading.common.common_bex import setup_foo_logging

Same error:

ImportError: No module named foo_loading.common.bex_common

Any thoughts?

Cheers,
Victor

 

[Victor Hooi]

Hi,

If I try to use:

from .common.common_foo import setup_foo_logging

I get:

ValueError: Attempted relative import in non-package

And the absolute imports don't seem to be able to find the right modules.

Is it something to do with the fact I'm running the sync_em.py script from the "foo_loading/em_load" directory?

I thought I could just refer to the full path, and it'd find it, but evidently not...hmm.

Cheers,
Victor

 

[Victor Hooi]

Hi,

Hmm, this post on SO seems to suggest that importing from another sibling directory in a package ins't actually possibly in Python without some ugly hacks?

http://stackoverflow.com/questions/6323860/sibling-package-imports

Did I read the above correctly?

Is there another way I can structure my code so that I can run the sync_em.py and sync_pg.py scripts, and they can pull common functions from somewhere?

Cheers,
Victor

 

[Peter Otten]

Hi,

Hmm, this post on SO seems to suggest that importing from another sibling
directory in a package ins't actually possibly in Python without some ugly
hacks?

http://stackoverflow.com/questions/6323860/sibling-package-imports

Did I read the above correctly?
Yes.

Is there another way I can structure my code so that I can run the
sync_em.py and sync_pg.py scripts, and they can pull common functions from
somewhere?

The packages you are trying to access in your original post

foo_loading/
__init__.py
common/
common_foo.py
em_load/
__init__.py
config.yaml
sync_em.py
pg_load/
__init__.py
config.yaml
sync_pg.py

aren't actually siblings in the sense of the stackoverflow topic above, they
are subpackages of foo_loading, and as you already found out

So from within the sync_em.py script, I'm trying to import a function from foo_loading/common/common_foo.py.

from ..common.common_foo import setup_foo_logging

I get the error:

ValueError: Attempted relative import in non-package

If I change directories to the parent of "foo_loading", then run

python -m foo_loading.em_load.sync_em sync_em.py

it works. However, this seems a bit roundabout, and I suspect I'm not doing things correctly.

Ideally, I want a user to be able to just run sync_em.py from it's own

directory, and have it correctly import the logging/config modules from
common_foo.py, and just work.

What is the correct way to achieve this?

you can access them as long as the *parent* directory of foo_loading is in
sys.path through PYTHONPATH, or as the working directory, or any other
means. However, if you step into the package, e. g.

$ cd foo_loading
$ python -c 'import common'

then from Python's point of view 'common' is a toplevel package rather than
the intended 'foo_loading.common', and intra-package imports will break.

To preserve your sanity I therefore recommend that you

(1) avoid to put package directories into sys.path
(1a) avoid to cd into a package
(2) put scripts you plan to invoke directly rather than import outside the
package.

 

[Victor Hooi]

Hi,

Wait - err, subpackage != module, right? Do you think you could explain what a sub-package is please? I tried Googling, and couldn't seem to find the term in this context.

Also, so you're saying to put the actual script that I want to invoke *outside* the Python package.

Do you mean something like this:

sync_em.py
sync_pg.py
foo_loading/
__init__.py
common/
common_foo.py
em_load/
__init__.py
config.yaml
em.py
pg_load/
__init__.py
config.yaml
pg.py

and the sync_em.py and sync_pg.py would just be thin wrappers pulling in things from em.py and pg.py? Is that a recommended approach to organise the code?

Would it make any difference if I actually packaged it up so you could install it in site-packages? Could I then call modules from other modules within the package?

Cheers,
Victor

 

[Peter Otten]

Wait - err, subpackage != module, right? Do you think you could explain
what a sub-package is please? I tried Googling, and couldn't seem to find
the term in this context.

In analogy to subdirectory I em_load and pg_load -- and common if you add an
__init__.py would be sub-packages, provided only the parent of foo_loading
is in sys.path and you import them with

import foo_loading.pg_load

etc.

Also, so you're saying to put the actual script that I want to invoke
*outside* the Python package.

Do you mean something like this:


and the sync_em.py and sync_pg.py would just be thin wrappers pulling in
things from em.py and pg.py? Is that a recommended approach to organise
the code?

I don't know. I prefer it that way.

Would it make any difference if I actually packaged it up so you could
install it in site-packages? Could I then call modules from other modules
within the package?

If you mean "import", yes, installing is one way to get it into sys.path.

 

[Peter Otten]

Wait - err, subpackage != module, right? Do you think you could explain
what a sub-package is please? I tried Googling, and couldn't seem to find
the term in this context.

[second attempt]

In analogy to the term "subdirectory" em_load and pg_load -- and common if
you add an __init__.py -- would be sub-packages, provided only the parent of
foo_loading is in sys.path and you import them with

import foo_loading.pg_load

etc.