import packet.module without importing packet.__init__ ?

Discussion in 'Python' started by Gelonida N, Sep 11, 2011.

  1. Gelonida N

    Gelonida N Guest

    Hi,

    I am little shaky with how exactly python imports packages / modules etc.

    Is it possible to import a module from a packet without importing its
    __init__.py ?


    Full example:
    ==============

    # application.py ---------------------
    print "starting application"
    import mypacket.module1

    # mypacket.__init__.py ---------------------
    print "importing __init__"


    # mypacket.module1.py ---------------------
    print "importing module1"



    The output, that I get with python 2.6 is
    $ python application.py
    > starting application
    > importing __init__
    > importing module1


    Under certain circumstances I'd like to avoid the implicit import
    of __init__.py

    Due to other constrains of my project I am obliged to have a
    non empty __init__.py
     
    Gelonida N, Sep 11, 2011
    #1
    1. Advertising

  2. Gelonida N wrote:


    > Is it possible to import a module from a packet without importing its
    > __init__.py ?


    Untested, but I think so. But you shouldn't. The solution (if it does work,
    as I said I haven't tested it) is a hack.

    Suppose you have a library like this:

    modules/
    +-- spam.py
    +-- ham.py
    +-- package/
    +-- __init__.py
    +-- cheese.py

    The directory "modules" is in your Python path, so you can "import spam"
    or "import package". The trick is to add "modules/package" to the Python
    path so that "import cheese" will find cheese.py directly.

    You can do that by manipulating sys.path, or by setting the environment
    variable PYTHONPATH.

    Another hack would be to add a hard link to the top level:

    modules/
    +-- spam.py
    +-- ham.py
    +-- cheese.py <------------+
    +-- package/ |
    +-- __init__.py |
    +-- cheese.py <--------+


    This is not a copy, it is a hard link: the same file appears in literally
    two places. (But note that not all file systems support hard linking.) Soft
    links or shortcuts might also work. Now you can "import cheese".

    But generally, you should design your package so that it doesn't matter
    whether or not __init__.py is imported first. I see three reasnable
    situations:

    (1) package.cheese should rely on package, in which case it requires
    package.__init__ to be imported first. (Sometimes I put code common to the
    entire package in __init__.py.)

    (2) Just don't worry about the fact that package.__init__ gets imported
    first. Do you have any idea how many modules get imported when Python
    starts up? What's one more?

    (3) If cheese.py truly is independent of package, then it shouldn't be part
    of package. Just make it a stand alone module outside the package
    directory, and be done. (Your installer can still treat cheese.py as a
    dependency of package, and ensure that they are both installed.)



    --
    Steven
     
    Steven D'Aprano, Sep 11, 2011
    #2
    1. Advertising

  3. Gelonida N

    Gelonida N Guest

    Hi Steven,

    Thanks for your answer.

    On 09/11/2011 02:56 AM, Steven D'Aprano wrote:
    > Gelonida N wrote:
    >> Is it possible to import a module from a packet without importing its
    >> __init__.py ?

    >
    > Untested, but I think so. But you shouldn't. The solution (if it does work,
    > as I said I haven't tested it) is a hack.
    >
    > Suppose you have a library like this:

    .. . .
    .. . .
    > The directory "modules" is in your Python path, so you can "import spam"
    > or "import package". The trick is to add "modules/package" to the Python
    > path so that "import cheese" will find cheese.py directly.
    >


    Yes it's a hack, but good to know it exists. Just in case.

    > Another hack would be to add a hard link to the top level:
    >


    As you said: Hard links would be a little annoying on some file systems.

    >
    > But generally, you should design your package so that it doesn't matter
    > whether or not __init__.py is imported first. I see three reasnable
    > situations:
    >
    > (1) package.cheese should rely on package, in which case it requires
    > package.__init__ to be imported first. (Sometimes I put code common to the
    > entire package in __init__.py.)
    >
    > (2) Just don't worry about the fact that package.__init__ gets imported
    > first. Do you have any idea how many modules get imported when Python
    > starts up? What's one more?
    >
    > (3) If cheese.py truly is independent of package, then it shouldn't be part
    > of package. Just make it a stand alone module outside the package
    > directory, and be done. (Your installer can still treat cheese.py as a
    > dependency of package, and ensure that they are both installed.)
    >



    There's still something, that I am philosophycally missing.

    Wy do I have to import the entire tree if I'm just interested in a leave.

    this is not always desired.

    let's imagine

    kitchen.__init__
    kitchen.knives
    kitchen.pans
    kitchen.pots


    or alternatively
    os
    os.path
    os.wheteverelse

    Let's assume, that kitchen/__init__.py is importing the sub modules
    knives, pans and pots

    then I understand of course, that the next lines would import the entire
    kitchen

    > import kitchen # will import all activities
    > kitchen.pots.boil(egg)



    But if I know that I just want to boil some eggs I would really
    appreciate if I could just say

    > from kitchen.pot import boil
    > boil(eggs)

    without having the side effect of getting pans and knives also loaded
    into memory.

    Moving pots out into a separate package doesn't really feel right.
    I agree, that a 10 level deep package tree is not really desirable, but
    having every package flat on the top level doesn't sound that good either.

    So it seems, that developers of huge packages shouldn't really do a lot
    in __init__.py.

    Otherwise any user would always 'suck in' the entiry package with all
    its sub modules.




    To give more details about why I asked this question:

    The real setup looks a little different and there are some issues, that
    I am having:

    One package in question is a huge django application.

    Within the package is one module defining some constants and tiny
    functions, which I would like to reuse from some command line tools,
    which should start quickly without loading any of the django specific
    modules. (the startup penalty and the memory overhead would be noticable)

    I am using rpc4django, which expects, that __init__.py of the django
    application exports some all rpc functions
    which will basically import 95% of the django application and the entire
    django frame work (none of which were required by my command tool,
    support utility for this application)


    I could of course create a separate package just for this tiny sub
    module, but somehow it doesn't feel right to me.


    The second issue is a little more complicated. basically I end up having
    circular dependencies.

    as I have two applications that send a few customs django signals forth
    an back.

    If I wouldn't import __init__.py or if I moved the declaration of the
    signals into a third package then circular dependencies would disappear.


    So it seems, that if I can't avoid loading __init__.py in
    a nice way I have to create one or two tiny packages outside of my
    django application,
    or change the python path as you proposed (increases of course name
    space clashes)

    or to find a way for rpc4django of loading / locating the rpc functions
    without putting them in __init__.py change the rpc functions such, that
    they will only import the other sub modules when being called the first
    time (proxies / delayed imports, . . .)
     
    Gelonida N, Sep 11, 2011
    #3
  4. Gelonida N wrote:


    > There's still something, that I am philosophycally missing.
    >
    > Wy do I have to import the entire tree if I'm just interested in a leave.


    You don't. Python just imports the branches leading to the leaf, not the
    entire tree.

    [...]
    > But if I know that I just want to boil some eggs I would really
    > appreciate if I could just say
    >
    >> from kitchen.pot import boil
    >> boil(eggs)

    > without having the side effect of getting pans and knives also loaded
    > into memory.


    In your example, you stated that kitchen explicitly imports kitchen.pans and
    kitchen.knives. So in that case, take it up with the designer of the
    kitchen package -- it was his decision to import them, just like he
    imported re and math and httplib. (For example.)

    If kitchen is your package, then it is your choice: if you want to have
    control over when sub-packages get imported, then don't automatically
    import them in __init__.py. If you want them to be automatically imported,
    like os and os.path, then automatically import them. If you don't, don't.

    But note that you cannot expect to import kitchen.pot without importing
    kitchen first.


    > One package in question is a huge django application.
    >
    > Within the package is one module defining some constants and tiny
    > functions, which I would like to reuse from some command line tools,
    > which should start quickly without loading any of the django specific
    > modules. (the startup penalty and the memory overhead would be noticable)
    >
    > I am using rpc4django, which expects, that __init__.py of the django
    > application exports some all rpc functions
    > which will basically import 95% of the django application and the entire
    > django frame work (none of which were required by my command tool,
    > support utility for this application)
    >
    >
    > I could of course create a separate package just for this tiny sub
    > module, but somehow it doesn't feel right to me.


    Why should it be a package? Just make it a stand-alone module.

    Or use path manipulation from your command line tool to import it on its
    own. It might be somewhat of a hack, but if need to do it, do so.

    Or do something like this:

    my_app/
    +-- __init__.py # lightweight, nearly empty
    +-- cmd_tools.py
    +-- module1.py
    +-- module2.py
    +-- rpc/
    +-- __init__.py # huge, imports my_app.module1, my_app.module2, etc.


    then point rpc4django at my_app.rpc instead of my_app.



    --
    Steven
     
    Steven D'Aprano, Sep 11, 2011
    #4
  5. Gelonida N

    Gelonida N Guest

    Hi Steven,

    Thanks again for your answer.


    On 09/11/2011 06:51 AM, Steven D'Aprano wrote:
    > Gelonida N wrote:


    >
    > In your example, you stated that kitchen explicitly imports kitchen.pans and
    > kitchen.knives. So in that case, take it up with the designer of the
    > kitchen package -- it was his decision to import them, just like he
    > imported re and math and httplib. (For example.)


    Exactly. Thus my conclusion, that too many imports (just for commodity)
    in __init__.py might not be such a good idea if one wants to allow
    leaf-only imports.
    >
    > If kitchen is your package, then it is your choice: if you want to have
    > control over when sub-packages get imported, then don't automatically
    > import them in __init__.py.


    Agreed. This is however my problem: Up to my understanding rpc4django
    insists on putting code into __init__.py

    >
    >
    >> One package in question is a huge django application.
    >>
    >> Within the package is one module defining some constants and tiny
    >> functions, which I would like to reuse from some command line tools,
    >> which should start quickly without loading any of the django specific
    >> modules. (the startup penalty and the memory overhead would be noticable)
    >>
    >> I am using rpc4django, which expects, that __init__.py of the django
    >> application exports some all rpc functions
    >> which will basically import 95% of the django application and the entire
    >> django frame work (none of which were required by my command tool,
    >> support utility for this application)
    >>
    >>
    >> I could of course create a separate package just for this tiny sub
    >> module, but somehow it doesn't feel right to me.

    >
    > Why should it be a package? Just make it a stand-alone module.


    True

    > Or do something like this:
    >
    > my_app/
    > +-- __init__.py # lightweight, nearly empty
    > +-- cmd_tools.py
    > +-- module1.py
    > +-- module2.py
    > +-- rpc/
    > +-- __init__.py # huge, imports my_app.module1, my_app.module2, etc.
    >
    > then point rpc4django at my_app.rpc instead of my_app.
    >


    Up to my knowledge rpc4django just imports all __inits__ of all django
    applications to find look for functions with a certain decorator.

    I'm not sure, whether django allows nested appllications, but this might
    be a solution.
     
    Gelonida N, Sep 11, 2011
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. plb
    Replies:
    2
    Views:
    358
  2. Steven Bethard
    Replies:
    2
    Views:
    460
    Steven Bethard
    Feb 16, 2005
  3. Kent Johnson
    Replies:
    7
    Views:
    917
    Jan Niklas Fingerle
    Feb 12, 2006
  4. Li Han
    Replies:
    2
    Views:
    511
    bobicanprogram
    Feb 9, 2009
  5. Ramchandra Apte
    Replies:
    17
    Views:
    343
    Manuel Pégourié-Gonnard
    Sep 30, 2012
Loading...

Share This Page