import packet.module without importing packet.__init__ ?

G

Gelonida N

Hi,

I am little shaky with how exactly python imports packages / modules etc.

Is it possible to import a module from a packet without importing its
__init__.py ?


Full example:
==============

# application.py ---------------------
print "starting application"
import mypacket.module1

# mypacket.__init__.py ---------------------
print "importing __init__"


# mypacket.module1.py ---------------------
print "importing module1"



The output, that I get with python 2.6 is
$ python application.py
starting application
importing __init__
importing module1

Under certain circumstances I'd like to avoid the implicit import
of __init__.py

Due to other constrains of my project I am obliged to have a
non empty __init__.py
 
S

Steven D'Aprano

Gelonida N wrote:

Is it possible to import a module from a packet without importing its
__init__.py ?

Untested, but I think so. But you shouldn't. The solution (if it does work,
as I said I haven't tested it) is a hack.

Suppose you have a library like this:

modules/
+-- spam.py
+-- ham.py
+-- package/
+-- __init__.py
+-- cheese.py

The directory "modules" is in your Python path, so you can "import spam"
or "import package". The trick is to add "modules/package" to the Python
path so that "import cheese" will find cheese.py directly.

You can do that by manipulating sys.path, or by setting the environment
variable PYTHONPATH.

Another hack would be to add a hard link to the top level:

modules/
+-- spam.py
+-- ham.py
+-- cheese.py <------------+
+-- package/ |
+-- __init__.py |
+-- cheese.py <--------+


This is not a copy, it is a hard link: the same file appears in literally
two places. (But note that not all file systems support hard linking.) Soft
links or shortcuts might also work. Now you can "import cheese".

But generally, you should design your package so that it doesn't matter
whether or not __init__.py is imported first. I see three reasnable
situations:

(1) package.cheese should rely on package, in which case it requires
package.__init__ to be imported first. (Sometimes I put code common to the
entire package in __init__.py.)

(2) Just don't worry about the fact that package.__init__ gets imported
first. Do you have any idea how many modules get imported when Python
starts up? What's one more?

(3) If cheese.py truly is independent of package, then it shouldn't be part
of package. Just make it a stand alone module outside the package
directory, and be done. (Your installer can still treat cheese.py as a
dependency of package, and ensure that they are both installed.)
 
G

Gelonida N

Hi Steven,

Thanks for your answer.

Untested, but I think so. But you shouldn't. The solution (if it does work,
as I said I haven't tested it) is a hack.

Suppose you have a library like this:
.. . .
.. . .
The directory "modules" is in your Python path, so you can "import spam"
or "import package". The trick is to add "modules/package" to the Python
path so that "import cheese" will find cheese.py directly.

Yes it's a hack, but good to know it exists. Just in case.
Another hack would be to add a hard link to the top level:

As you said: Hard links would be a little annoying on some file systems.
But generally, you should design your package so that it doesn't matter
whether or not __init__.py is imported first. I see three reasnable
situations:

(1) package.cheese should rely on package, in which case it requires
package.__init__ to be imported first. (Sometimes I put code common to the
entire package in __init__.py.)

(2) Just don't worry about the fact that package.__init__ gets imported
first. Do you have any idea how many modules get imported when Python
starts up? What's one more?

(3) If cheese.py truly is independent of package, then it shouldn't be part
of package. Just make it a stand alone module outside the package
directory, and be done. (Your installer can still treat cheese.py as a
dependency of package, and ensure that they are both installed.)


There's still something, that I am philosophycally missing.

Wy do I have to import the entire tree if I'm just interested in a leave.

this is not always desired.

let's imagine

kitchen.__init__
kitchen.knives
kitchen.pans
kitchen.pots


or alternatively
os
os.path
os.wheteverelse

Let's assume, that kitchen/__init__.py is importing the sub modules
knives, pans and pots

then I understand of course, that the next lines would import the entire
kitchen
import kitchen # will import all activities
kitchen.pots.boil(egg)


But if I know that I just want to boil some eggs I would really
appreciate if I could just say
from kitchen.pot import boil
boil(eggs)
without having the side effect of getting pans and knives also loaded
into memory.

Moving pots out into a separate package doesn't really feel right.
I agree, that a 10 level deep package tree is not really desirable, but
having every package flat on the top level doesn't sound that good either.

So it seems, that developers of huge packages shouldn't really do a lot
in __init__.py.

Otherwise any user would always 'suck in' the entiry package with all
its sub modules.




To give more details about why I asked this question:

The real setup looks a little different and there are some issues, that
I am having:

One package in question is a huge django application.

Within the package is one module defining some constants and tiny
functions, which I would like to reuse from some command line tools,
which should start quickly without loading any of the django specific
modules. (the startup penalty and the memory overhead would be noticable)

I am using rpc4django, which expects, that __init__.py of the django
application exports some all rpc functions
which will basically import 95% of the django application and the entire
django frame work (none of which were required by my command tool,
support utility for this application)


I could of course create a separate package just for this tiny sub
module, but somehow it doesn't feel right to me.


The second issue is a little more complicated. basically I end up having
circular dependencies.

as I have two applications that send a few customs django signals forth
an back.

If I wouldn't import __init__.py or if I moved the declaration of the
signals into a third package then circular dependencies would disappear.


So it seems, that if I can't avoid loading __init__.py in
a nice way I have to create one or two tiny packages outside of my
django application,
or change the python path as you proposed (increases of course name
space clashes)

or to find a way for rpc4django of loading / locating the rpc functions
without putting them in __init__.py change the rpc functions such, that
they will only import the other sub modules when being called the first
time (proxies / delayed imports, . . .)
 
S

Steven D'Aprano

Gelonida N wrote:

There's still something, that I am philosophycally missing.

Wy do I have to import the entire tree if I'm just interested in a leave.

You don't. Python just imports the branches leading to the leaf, not the
entire tree.

[...]
But if I know that I just want to boil some eggs I would really
appreciate if I could just say

without having the side effect of getting pans and knives also loaded
into memory.

In your example, you stated that kitchen explicitly imports kitchen.pans and
kitchen.knives. So in that case, take it up with the designer of the
kitchen package -- it was his decision to import them, just like he
imported re and math and httplib. (For example.)

If kitchen is your package, then it is your choice: if you want to have
control over when sub-packages get imported, then don't automatically
import them in __init__.py. If you want them to be automatically imported,
like os and os.path, then automatically import them. If you don't, don't.

But note that you cannot expect to import kitchen.pot without importing
kitchen first.

One package in question is a huge django application.

Within the package is one module defining some constants and tiny
functions, which I would like to reuse from some command line tools,
which should start quickly without loading any of the django specific
modules. (the startup penalty and the memory overhead would be noticable)

I am using rpc4django, which expects, that __init__.py of the django
application exports some all rpc functions
which will basically import 95% of the django application and the entire
django frame work (none of which were required by my command tool,
support utility for this application)


I could of course create a separate package just for this tiny sub
module, but somehow it doesn't feel right to me.

Why should it be a package? Just make it a stand-alone module.

Or use path manipulation from your command line tool to import it on its
own. It might be somewhat of a hack, but if need to do it, do so.

Or do something like this:

my_app/
+-- __init__.py # lightweight, nearly empty
+-- cmd_tools.py
+-- module1.py
+-- module2.py
+-- rpc/
+-- __init__.py # huge, imports my_app.module1, my_app.module2, etc.


then point rpc4django at my_app.rpc instead of my_app.
 
G

Gelonida N

Hi Steven,

Thanks again for your answer.


Gelonida N wrote:
In your example, you stated that kitchen explicitly imports kitchen.pans and
kitchen.knives. So in that case, take it up with the designer of the
kitchen package -- it was his decision to import them, just like he
imported re and math and httplib. (For example.)

Exactly. Thus my conclusion, that too many imports (just for commodity)
in __init__.py might not be such a good idea if one wants to allow
leaf-only imports.
If kitchen is your package, then it is your choice: if you want to have
control over when sub-packages get imported, then don't automatically
import them in __init__.py.

Agreed. This is however my problem: Up to my understanding rpc4django
insists on putting code into __init__.py
Why should it be a package? Just make it a stand-alone module.
True

Or do something like this:

my_app/
+-- __init__.py # lightweight, nearly empty
+-- cmd_tools.py
+-- module1.py
+-- module2.py
+-- rpc/
+-- __init__.py # huge, imports my_app.module1, my_app.module2, etc.

then point rpc4django at my_app.rpc instead of my_app.

Up to my knowledge rpc4django just imports all __inits__ of all django
applications to find look for functions with a certain decorator.

I'm not sure, whether django allows nested appllications, but this might
be a solution.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,734
Messages
2,569,441
Members
44,832
Latest member
GlennSmall

Latest Threads

Top