Package organization

[Robert Ferrell]

I have a question about how to organize modules in a package. I think
there is something fundamental that I am missing. Say I'm creating a
package,
GreatPackage, which has three sub-packages, myUtilities, GoodStuff,
MoreGoodStuff.

The directories look like:

GreatPackage
|
|-- myUtilities
|
|-- GoodStuff
|
|-- MoreGoodStuff

What is the right way for modules in GoodStuff and MoreGoodStuff to
refer to modules etc... in myUtilities? I'd like to set PYTHONPATH to
just point to the directory containing GreatPackage. But, if put
something like:

from GreatPackage.myUtilities import aUtilityThingy

in a module in GoodStuff, then the whole package gets loaded,
including MoreGoodStuff. That's okay except when I'm developing and
MoreGoodStuff is broken.

What are guidelines for developing/deploying packages which contain
pacakges?

thanks,
-robert

 

[Graham Ashton]

I have a question about how to organize modules in a package. I think
there is something fundamental that I am missing. Say I'm creating a
package,
GreatPackage, which has three sub-packages, myUtilities, GoodStuff,
MoreGoodStuff.

The directories look like:

GreatPackage
|
|-- myUtilities
|
|-- GoodStuff
|
|-- MoreGoodStuff

What is the right way for modules in GoodStuff and MoreGoodStuff to
refer to modules etc... in myUtilities?

There's a PEP that recommends that when importing things you refer to them
explicitly; i.e. inside GoodStuff you'd say

import GreatPackage.myUtilities

rather than just

import myUtilities

I've only benefitted from doing it that way myself in large apps that have
had many packages with sub packages. It keeps you safe from module name
clashes at different levels of the hierarchy (not that I think they're a
sensible thing to have). By name clashes I mean if you have modules
set up like this:

lib
shoe.lib
shoe.fruit

then you can't get at the top level lib module from inside shoe.fruit.

from GreatPackage.myUtilities import aUtilityThingy

in a module in GoodStuff, then the whole package gets loaded,
including MoreGoodStuff.

MoreGoodStuff won't get loaded if you import GreatPackage, unless
GreatPackage/__init__.py causes it to be loaded. Obviously, if something
in myUtilities imports something from MoreGoodStuff then python will
attempt to load MoreGoodStuff when importing myUtilities.

Also, because you're importing something from inside a module using the
"from" style imports you're risking running into circular reference
problems. In my experience from imports are best avoided, unless you're
importing a module from a package.

What are guidelines for developing/deploying packages which contain
pacakges?

For deploying, have you looked at distutils?

-- Graham

 

[Robert Ferrell]

There's a PEP that recommends that when importing things you refer to them
explicitly; i.e. inside GoodStuff you'd say

import GreatPackage.myUtilities

rather than just

import myUtilities

I've only benefitted from doing it that way myself in large apps that have
had many packages with sub packages. It keeps you safe from module name
clashes at different levels of the hierarchy (not that I think they're a
sensible thing to have). By name clashes I mean if you have modules
set up like this:

lib
shoe.lib
shoe.fruit

then you can't get at the top level lib module from inside shoe.fruit.


MoreGoodStuff won't get loaded if you import GreatPackage, unless
GreatPackage/__init__.py causes it to be loaded. Obviously, if something
in myUtilities imports something from MoreGoodStuff then python will
attempt to load MoreGoodStuff when importing myUtilities.

Ah Ha! Somehow I'd convinced myself that GreatPackage/__init__.py had
to import all the submodules. So that's what I was doing. That's
obviously wrong. Now that I've corrected that mistake I see that what
I was having trouble with is not an issue.

Also, because you're importing something from inside a module using the
"from" style imports you're risking running into circular reference
problems. In my experience from imports are best avoided, unless you're
importing a module from a package.

I don't understand this comment. How does "from" style importing
increase the risk of circular imports?

Thanks for the clarification!
-robert

 

[Graham Ashton]

I don't understand this comment. How does "from" style importing
increase the risk of circular imports?

Imagine you've got two modules, spam and eggs.

When you import a module it's top level statements get evaluated. Imagine
that spam imports something from eggs. The interpreter (whilst importing
spam) pauses to import eggs. What if eggs tries to import something from
spam?

You might think, ah, that's a circular import. But it isn't necessarily.
You see, when it starts importing spam the interpreter stores an empty
module object in sys.modules and fills in the details once it's
finished importing spam. Consequently when it's trying to import spam
again from inside eggs it finds a module reference in sys.modules for spam
and can carry on happily.

Things only break on you if eggs is trying to refer to something inside
spam that the original import of spam hasn't filled in yet.

I'll try and give you an example. Here's the spam module:

import eggs

def count_spam():
return 1

And the eggs module:

from spam import count_spam

def count_eggs_and_spam():
return count_spam() + 1

If your main script then tries to import spam it'll blow up:

% python
Python 2.3.2 (#1, Dec 8 2003, 10:49:17)
[GCC 3.3.2 20031022 (Gentoo Linux 3.3.2-r3, propolice)] on linux2
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "spam.py", line 1, in ?
import eggs
File "eggs.py", line 1, in ?
from spam import count_spam
ImportError: cannot import name count_spam

It's because the spam module in sys.modules hasn't been filled in yet, as
the interpreter put off importing the contents of spam until the "import
eggs" statement returned. Replace the from import with a normal import and
the problem goes away. Here's a replacement eggs module:

% cat eggs.py
import spam

def count_eggs_and_spam():
return spam.count_spam() + 1

It's no longer an issue as the spam.count_spam() function doesn't get
evaluated until the count_eggs_and_spam() function is called, which is
long after both the modules have been fully imported.

Stay away from from, it's dangerous. And in my opinion you get less clear
code.

Incidentally, I learnt the nitty gritty of this from Martelli's
Python in a Nutshell. Recommended.

-- Graham

 

[Fredrik Lundh]

I don't understand this comment. How does "from" style importing
increase the risk of circular imports?

python has no problem doing circular (or recursive) imports, but you're likely
to run into problems if you're doing recursive imports and you're referring to
stuff from one module on the *module-level* in another module. from-import
does exactly that.

for a diskussion, see the section on "recursive imports" on this page:

http://www.effbot.org/zone/import-confusion.htm

</F>