How to improve the usability of nested packages

Discussion in 'Python' started by Michael Schwarz, Nov 2, 2012.

  1. I need some guidance on how to structure a large library with lots of
    packages and sub-packages. Using the library should be as effortless
    as possible. It's your average beast of a catch-all for sharing code
    across applications at our company. Let's call it "the_library". In my
    attempt to structure the library, I was following two principles:

    - The complete library is contained in a single package. This is to
    avoid polluting the top-level namespace.
    - Only modules and sub-packages directly under the top-level package
    should be imported directly. This means that any class or function in
    the library is accessed using the same qualified name everywhere
    inside the library or the application. This makes moving code around
    easier.

    Following this, using a module from the library is pretty
    straight-forward. A typical file in the application code could start
    with:

    from the_library import sip, rtp, sdp

    This works from any module or script in the library or application.
    Then I decided to split the "sip" module into smaller modules, e.g.
    "message", "transaction", "dialog", all contained in a package named
    "sip". Ideally, an application would still import the sip package
    using the import above and then, for example, access the "DialogID"
    class using "sip.dialog.DialogID". Currently this is only possible
    when also explicitly importing the "dialog" module:

    from the_library import sip
    import the_library.sip.dialog

    This is ugly and seems unnecessary to me as, for example, having all
    the modules in the "sip" package available using a single import would
    not pollute the local namespace. So I tried to enable this by
    importing all the modules in the "sip" package from the package's
    "__init__.py":

    from . import message, transaction, dialog

    … which doesn't work. Some of the modules reference other modules in
    the same package. I'm not talking about cyclic references, but, for
    example, the "dialog" module uses the "transaction" module. The
    problem is that the "dialog" module uses the same mechanism shown
    above to import the other modules from it's package. This means that
    modules and packages are imported in this order:

    - Application code executes "from the_library import sip"
    - the_library/__init__.py is executed. No imports here.
    - the_library/sip/__init__.py executes "from . import [...], dialog"
    - the_library/sip/dialog.py executes "from the_library import sip"

    During the last import a problem arises: The module object for the
    package "the_library" does not yet have a "sip" member (as it is still
    executing the import) and so the import fails. It is still possible to
    import the "transaction" module directly from the "dialog" module
    using:

    from . import transaction

    But this would make the "transaction" module available under a
    different qualified name as anywhere else (where it's accessed using
    "sip.transaction").

    What path would you take to circumvent this problem? Would you break
    the rule that any module should be accessed using the same way, no
    matter from where it is accessed, or would you maybe structure the
    library entirely different?

    Thanks for any suggestions!

    Michael
     
    Michael Schwarz, Nov 2, 2012
    #1
    1. Advertising

  2. On 11/02/2012 12:11 PM, Michael Schwarz wrote:
    > … which doesn't work. Some of the modules reference other modules in
    > the same package. I'm not talking about cyclic references, but, for
    > example, the "dialog" module uses the "transaction" module. The
    > problem is that the "dialog" module uses the same mechanism shown
    > above to import the other modules from it's package. This means that
    > modules and packages are imported in this order:
    >
    > - Application code executes "from the_library import sip"
    > - the_library/__init__.py is executed. No imports here.
    > - the_library/sip/__init__.py executes "from . import [...], dialog"
    > - the_library/sip/dialog.py executes "from the_library import sip"



    In a way, you do have a cyclic reference. If you think of "import sip"
    as "from sip import __init__ as sip", it should become apparent.


    Anyway, I see two ways around this. One is to rename the package and
    create a module under the_library with the old package name that imports
    what you want. To keep using the same names, every module in the package
    can import siblings like so (where sippkg is the new package name):
    import the_library.sippkg.transaction
    sip = the_library.sippkg


    The second way is to not use import at all to load sibling modules
    (except in __init__.py), and instead use:
    sip = sys.modules['the_library.sip']

    This will work as long as you are careful to load the modules from
    __init__.py in the correct order, where each module's dependencies must
    appear before the module, in the import list.
     
    Rouslan Korneychuk, Nov 6, 2012
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Paul  Smith
    Replies:
    0
    Views:
    745
    Paul Smith
    Nov 18, 2003
  2. Esmail
    Replies:
    8
    Views:
    332
    Esmail
    Mar 21, 2009
  3. David Lyon
    Replies:
    19
    Views:
    598
    David Lyon
    Apr 24, 2009
  4. Terry Reedy
    Replies:
    0
    Views:
    146
    Terry Reedy
    Nov 2, 2012
  5. Stefan H. Holek
    Replies:
    0
    Views:
    145
    Stefan H. Holek
    Nov 2, 2012
Loading...

Share This Page