How to improve the usability of nested packages

  • Thread starter Michael Schwarz
  • Start date
M

Michael Schwarz

I need some guidance on how to structure a large library with lots of
packages and sub-packages. Using the library should be as effortless
as possible. It's your average beast of a catch-all for sharing code
across applications at our company. Let's call it "the_library". In my
attempt to structure the library, I was following two principles:

- The complete library is contained in a single package. This is to
avoid polluting the top-level namespace.
- Only modules and sub-packages directly under the top-level package
should be imported directly. This means that any class or function in
the library is accessed using the same qualified name everywhere
inside the library or the application. This makes moving code around
easier.

Following this, using a module from the library is pretty
straight-forward. A typical file in the application code could start
with:

from the_library import sip, rtp, sdp

This works from any module or script in the library or application.
Then I decided to split the "sip" module into smaller modules, e.g.
"message", "transaction", "dialog", all contained in a package named
"sip". Ideally, an application would still import the sip package
using the import above and then, for example, access the "DialogID"
class using "sip.dialog.DialogID". Currently this is only possible
when also explicitly importing the "dialog" module:

from the_library import sip
import the_library.sip.dialog

This is ugly and seems unnecessary to me as, for example, having all
the modules in the "sip" package available using a single import would
not pollute the local namespace. So I tried to enable this by
importing all the modules in the "sip" package from the package's
"__init__.py":

from . import message, transaction, dialog

… which doesn't work. Some of the modules reference other modules in
the same package. I'm not talking about cyclic references, but, for
example, the "dialog" module uses the "transaction" module. The
problem is that the "dialog" module uses the same mechanism shown
above to import the other modules from it's package. This means that
modules and packages are imported in this order:

- Application code executes "from the_library import sip"
- the_library/__init__.py is executed. No imports here.
- the_library/sip/__init__.py executes "from . import [...], dialog"
- the_library/sip/dialog.py executes "from the_library import sip"

During the last import a problem arises: The module object for the
package "the_library" does not yet have a "sip" member (as it is still
executing the import) and so the import fails. It is still possible to
import the "transaction" module directly from the "dialog" module
using:

from . import transaction

But this would make the "transaction" module available under a
different qualified name as anywhere else (where it's accessed using
"sip.transaction").

What path would you take to circumvent this problem? Would you break
the rule that any module should be accessed using the same way, no
matter from where it is accessed, or would you maybe structure the
library entirely different?

Thanks for any suggestions!

Michael
 
R

Rouslan Korneychuk

… which doesn't work. Some of the modules reference other modules in
the same package. I'm not talking about cyclic references, but, for
example, the "dialog" module uses the "transaction" module. The
problem is that the "dialog" module uses the same mechanism shown
above to import the other modules from it's package. This means that
modules and packages are imported in this order:

- Application code executes "from the_library import sip"
- the_library/__init__.py is executed. No imports here.
- the_library/sip/__init__.py executes "from . import [...], dialog"
- the_library/sip/dialog.py executes "from the_library import sip"


In a way, you do have a cyclic reference. If you think of "import sip"
as "from sip import __init__ as sip", it should become apparent.


Anyway, I see two ways around this. One is to rename the package and
create a module under the_library with the old package name that imports
what you want. To keep using the same names, every module in the package
can import siblings like so (where sippkg is the new package name):
import the_library.sippkg.transaction
sip = the_library.sippkg


The second way is to not use import at all to load sibling modules
(except in __init__.py), and instead use:
sip = sys.modules['the_library.sip']

This will work as long as you are careful to load the modules from
__init__.py in the correct order, where each module's dependencies must
appear before the module, in the import list.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,904
Latest member
HealthyVisionsCBDPrice

Latest Threads

Top