PYTHONPATH issue with sibling package names

Discussion in 'Python' started by Stuart Moffatt, Sep 10, 2009.

  1. Environment: Eclipse 3.4.2, Windows XP Pro SP2, Pydev 1.4.4, python
    2.6

    When I work in eclipse with java, I like to break up my client and
    server packages, like this:

    client-project/src/org/me/client

    server-project/src/org/me/api
    server-project/src/org/me/dao
    server-project/src/org/me/entity
    server-project/src/org/me/<etc>

    Then, when I need to call API code from the client, I make sure the
    API src is on the path.

    I am trying setup pydev projects to do the same thing, but running
    into an ImportError because my client code can't see the server src,
    even though it is on the path.

    Specifically, I am trying to import an entity from the server code
    into the client, like this:

    from org.me.entity import MyEntity

    If I do this from any module in the server project it is fine (because
    the src path is in the same eclipse project). But if I do it from
    anywhere in the client code I get the ImportError

    From what I can tell, python asks for the "closest" module path, which
    is the current project. It finds org.me, but there is only the client
    sub-package. The org.me.entity sibling is in another eclipse project,
    but even though that path is on the PYTHONPATH, python stopped looking
    after it found a similarly named parent package.

    Is there a trusted way to make sure python looks through all paths for
    sibling packages? Can I load 'org.me.client' from one path in
    PYTHONPATH and 'org.me.*' from another path in PYTHONPATH? I imagine
    if I try to force python to load the second package first that the
    same thing will happen in reverse.
     
    Stuart Moffatt, Sep 10, 2009
    #1
    1. Advertising

  2. Stuart Moffatt wrote:

    > Environment: Eclipse 3.4.2, Windows XP Pro SP2, Pydev 1.4.4, python
    > 2.6
    >
    > When I work in eclipse with java, I like to break up my client and
    > server packages, like this:
    >
    > client-project/src/org/me/client
    >
    > server-project/src/org/me/api
    > server-project/src/org/me/dao
    > server-project/src/org/me/entity
    > server-project/src/org/me/<etc>
    >
    > Then, when I need to call API code from the client, I make sure the
    > API src is on the path.
    >
    > I am trying setup pydev projects to do the same thing, but running
    > into an ImportError because my client code can't see the server src,
    > even though it is on the path.
    >
    > Specifically, I am trying to import an entity from the server code
    > into the client, like this:
    >
    > from org.me.entity import MyEntity
    >
    > If I do this from any module in the server project it is fine (because
    > the src path is in the same eclipse project). But if I do it from
    > anywhere in the client code I get the ImportError
    >
    > From what I can tell, python asks for the "closest" module path, which
    > is the current project. It finds org.me, but there is only the client
    > sub-package. The org.me.entity sibling is in another eclipse project,
    > but even though that path is on the PYTHONPATH, python stopped looking
    > after it found a similarly named parent package.
    >
    > Is there a trusted way to make sure python looks through all paths for
    > sibling packages? Can I load 'org.me.client' from one path in
    > PYTHONPATH and 'org.me.*' from another path in PYTHONPATH? I imagine
    > if I try to force python to load the second package first that the
    > same thing will happen in reverse.


    The solution you are searching for is called "namespace packages", and you
    can read more about it here:

    http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages


    Do yourself a favor though, and don't use those several-steps-namespaces.
    This is Python, not Java - two levels at *most*, normally a
    project-namespace should be enough.

    Diez
     
    Diez B. Roggisch, Sep 10, 2009
    #2
    1. Advertising

  3. On Sep 10, 10:12 am, "Diez B. Roggisch" <> wrote:
    > Stuart Moffatt wrote:
    > > Environment: Eclipse 3.4.2, Windows XP Pro SP2, Pydev 1.4.4, python
    > > 2.6

    >
    > > When I work in eclipse with java, I like to break up my client and
    > > server packages, like this:

    >
    > > client-project/src/org/me/client

    >
    > > server-project/src/org/me/api
    > > server-project/src/org/me/dao
    > > server-project/src/org/me/entity
    > > server-project/src/org/me/<etc>

    >
    > > Then, when I need to call API code from the client, I make sure the
    > > API src is on the path.

    >
    > > I am trying setup pydev projects to do the same thing, but running
    > > into an ImportError because my client code can't see the server src,
    > > even though it is on the path.

    >
    > > Specifically, I am trying to import an entity from the server code
    > > into the client, like this:

    >
    > >    from org.me.entity import MyEntity

    >
    > > If I do this from any module in the server project it is fine (because
    > > the src path is in the same eclipse project). But if I do it from
    > > anywhere in the client code I get the ImportError

    >
    > > From what I can tell, python asks for the "closest" module path, which
    > > is the current project. It finds org.me, but there is only the client
    > > sub-package. The org.me.entity sibling is in another eclipse project,
    > > but even though that path is on the PYTHONPATH, python stopped looking
    > > after it found a similarly named parent package.

    >
    > > Is there a trusted way to make sure python looks through all paths for
    > > sibling packages? Can I load 'org.me.client' from one path in
    > > PYTHONPATH and 'org.me.*' from another path in PYTHONPATH? I imagine
    > > if I try to force python to load the second package first that the
    > > same thing will happen in reverse.

    >
    > The solution you are searching for is called "namespace packages", and you
    > can read more about it here:
    >
    >  http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages
    >
    > Do yourself a favor though, and don't use those several-steps-namespaces.
    > This is Python, not Java - two levels at *most*, normally a
    > project-namespace should be enough.
    >
    > Diez



    Diez,

    Thanks for the tips re: namespace packages. Yeah, I know this is
    python, but for very large projects (or multiple projects for very
    large clients) it is just more flexible to stick to the reverse-dot
    java notation of "domain.organization.project.namespace".

    All I had to do was make sure that top- and mid-level folders had an
    __init__.py with this line:

    __import__('pkg_resources').declare_namespace(__name__)

    in it (and nothing else).

    So, for my example, I had:

    client-project/src/org/__init__.py
    client-project/src/org/me/__init__.py
    server-project/src/org/__init__.py
    server-project/src/org/me/__init__.py

    ....all with the special namespace declaration line above.

    Note that the lowest level folders in the package:

    client-project/src/org/me/client
    server-project/src/org/me/api
    server-project/src/org/me/<etc>

    ....all have __init__.py files with actual module code, and therefore
    do NOT have the namespace declaration line.

    To expose the server code (org.me.api, org.me.dao, etc) I have a
    setup.py in the server-project/src folder with:

    from setuptools import setup
    setup(name='myserver',
    version='1.0',
    namespace_packages = ['org','org.me'],
    packages=['org.me.api','org.me.dao','org.me.entity'],
    )

    A similar setup.py can be created for the client library too. Notice
    that the setup has every level (except the last) in the
    namespace_packages list argument. This is how it walks down the tree
    and builds the namespace properly.

    With this setup file I can make an .egg or .zip distribution using:

    cd server-project/src
    python setup.py bdist

    or

    python setup.py bdist_egg

    For my dev environment in eclipse I also wanted the convenience of
    editing server code AND have the latest code on the path for my client
    project (without rebuilding a dist). To do this I just:

    cd server-project/src
    python setup.py develop

    ....which puts a symlink (even does it properly in Windows) in my
    python/lib/site-packages folder. Now in my client code I can do a
    regular import, even though the code lives in another project, and
    though portions of the code base are at different PYTHONPATHs:

    from org.me.entity import MyEntity # from server-project/src
    from org.me.api import MyAPI # from server-project/src
    from org.me.client import MyClient # from client-project/src

    When my code is complete, the myserver-1.0-py2.6.egg and myclient-1.0-
    py2.6.egg can be included in other python modules by loading compiled
    source from inside the egg with:

    from pkg_resources import require
    require("myserver>=1.0")
    require("myclient>=1.0")

    and then:

    from org.me.entity import MyEntity # from compiled code in
    myserver-1.0-py2.6.egg
    from org.me.api import MyAPI # from compiled code in myserver-1.0-
    py2.6.egg
    from org.me.client import MyClient # from compiled code in
    myclient-1.0-py2.6.egg


    Again, it might seem like a headache to some python developers, but it
    replicates fairly well the java way of keeping code organized, but
    "merging" it into a virtual namespace (encapsulated in an egg archive)
    at run-time.

    Hope that helps other java developers in packaging python code.

    Stuart
     
    Stuart Moffatt, Sep 10, 2009
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kathy Burke
    Replies:
    0
    Views:
    710
    Kathy Burke
    Aug 3, 2003
  2. Replies:
    1
    Views:
    2,212
    =?Utf-8?B?YnJpYW5zW01DU0Rd?=
    Feb 16, 2006
  3. Replies:
    11
    Views:
    673
    Fredrik Lundh
    Jan 23, 2006
  4. Replies:
    1
    Views:
    261
    Gabriel Genellina
    Mar 24, 2008
  5. Tobiah

    PYTHONPATH and module names

    Tobiah, Jul 1, 2013, in forum: Python
    Replies:
    6
    Views:
    115
    Fábio Santos
    Jul 1, 2013
Loading...

Share This Page