PYTHONPATH issue with sibling package names

[Stuart Moffatt]

Environment: Eclipse 3.4.2, Windows XP Pro SP2, Pydev 1.4.4, python
2.6

When I work in eclipse with java, I like to break up my client and
server packages, like this:

client-project/src/org/me/client

server-project/src/org/me/api
server-project/src/org/me/dao
server-project/src/org/me/entity
server-project/src/org/me/<etc>

Then, when I need to call API code from the client, I make sure the
API src is on the path.

I am trying setup pydev projects to do the same thing, but running
into an ImportError because my client code can't see the server src,
even though it is on the path.

Specifically, I am trying to import an entity from the server code
into the client, like this:

from org.me.entity import MyEntity

If I do this from any module in the server project it is fine (because
the src path is in the same eclipse project). But if I do it from
anywhere in the client code I get the ImportError

From what I can tell, python asks for the "closest" module path, which
is the current project. It finds org.me, but there is only the client
sub-package. The org.me.entity sibling is in another eclipse project,
but even though that path is on the PYTHONPATH, python stopped looking
after it found a similarly named parent package.

Is there a trusted way to make sure python looks through all paths for
sibling packages? Can I load 'org.me.client' from one path in
PYTHONPATH and 'org.me.*' from another path in PYTHONPATH? I imagine
if I try to force python to load the second package first that the
same thing will happen in reverse.

 

[Diez B. Roggisch]

Environment: Eclipse 3.4.2, Windows XP Pro SP2, Pydev 1.4.4, python
2.6

When I work in eclipse with java, I like to break up my client and
server packages, like this:

client-project/src/org/me/client

server-project/src/org/me/api
server-project/src/org/me/dao
server-project/src/org/me/entity
server-project/src/org/me/<etc>

Then, when I need to call API code from the client, I make sure the
API src is on the path.

I am trying setup pydev projects to do the same thing, but running
into an ImportError because my client code can't see the server src,
even though it is on the path.

Specifically, I am trying to import an entity from the server code
into the client, like this:

from org.me.entity import MyEntity

If I do this from any module in the server project it is fine (because
the src path is in the same eclipse project). But if I do it from
anywhere in the client code I get the ImportError

From what I can tell, python asks for the "closest" module path, which
is the current project. It finds org.me, but there is only the client
sub-package. The org.me.entity sibling is in another eclipse project,
but even though that path is on the PYTHONPATH, python stopped looking
after it found a similarly named parent package.

Is there a trusted way to make sure python looks through all paths for
sibling packages? Can I load 'org.me.client' from one path in
PYTHONPATH and 'org.me.*' from another path in PYTHONPATH? I imagine
if I try to force python to load the second package first that the
same thing will happen in reverse.

The solution you are searching for is called "namespace packages", and you
can read more about it here:

http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages


Do yourself a favor though, and don't use those several-steps-namespaces.
This is Python, not Java - two levels at *most*, normally a
project-namespace should be enough.

Diez

 

[Stuart Moffatt]

The solution you are searching for is called "namespace packages", and you
can read more about it here:

 http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages

Do yourself a favor though, and don't use those several-steps-namespaces.
This is Python, not Java - two levels at *most*, normally a
project-namespace should be enough.

Diez

Diez,

Thanks for the tips re: namespace packages. Yeah, I know this is
python, but for very large projects (or multiple projects for very
large clients) it is just more flexible to stick to the reverse-dot
java notation of "domain.organization.project.namespace".

All I had to do was make sure that top- and mid-level folders had an
__init__.py with this line:

__import__('pkg_resources').declare_namespace(__name__)

in it (and nothing else).

So, for my example, I had:

client-project/src/org/__init__.py
client-project/src/org/me/__init__.py
server-project/src/org/__init__.py
server-project/src/org/me/__init__.py

....all with the special namespace declaration line above.

Note that the lowest level folders in the package:

client-project/src/org/me/client
server-project/src/org/me/api
server-project/src/org/me/<etc>

....all have __init__.py files with actual module code, and therefore
do NOT have the namespace declaration line.

To expose the server code (org.me.api, org.me.dao, etc) I have a
setup.py in the server-project/src folder with:

from setuptools import setup
setup(name='myserver',
version='1.0',
namespace_packages = ['org','org.me'],
packages=['org.me.api','org.me.dao','org.me.entity'],
)

A similar setup.py can be created for the client library too. Notice
that the setup has every level (except the last) in the
namespace_packages list argument. This is how it walks down the tree
and builds the namespace properly.

With this setup file I can make an .egg or .zip distribution using:

cd server-project/src
python setup.py bdist

or

python setup.py bdist_egg

For my dev environment in eclipse I also wanted the convenience of
editing server code AND have the latest code on the path for my client
project (without rebuilding a dist). To do this I just:

cd server-project/src
python setup.py develop

....which puts a symlink (even does it properly in Windows) in my
python/lib/site-packages folder. Now in my client code I can do a
regular import, even though the code lives in another project, and
though portions of the code base are at different PYTHONPATHs:

from org.me.entity import MyEntity # from server-project/src
from org.me.api import MyAPI # from server-project/src
from org.me.client import MyClient # from client-project/src

When my code is complete, the myserver-1.0-py2.6.egg and myclient-1.0-
py2.6.egg can be included in other python modules by loading compiled
source from inside the egg with:

from pkg_resources import require
require("myserver>=1.0")
require("myclient>=1.0")

and then:

from org.me.entity import MyEntity # from compiled code in
myserver-1.0-py2.6.egg
from org.me.api import MyAPI # from compiled code in myserver-1.0-
py2.6.egg
from org.me.client import MyClient # from compiled code in
myclient-1.0-py2.6.egg


Again, it might seem like a headache to some python developers, but it
replicates fairly well the java way of keeping code organized, but
"merging" it into a virtual namespace (encapsulated in an egg archive)
at run-time.

Hope that helps other java developers in packaging python code.

Stuart