How should multiple (related) projects be arranged (structured) and configured so that they can shar

Discussion in 'Python' started by ToddLMorgan@gmail.com, Apr 20, 2006.

  1. Guest

    Summary:
    How should multiple (related) projects be arranged (structured) and
    configured so that the following is possible:
    o Sharing common code (one of the projects would be a "common"
    project referenced by all others and likely the others would share at
    least the common project and possibly more as times goes on)
    o Clear separation of "production" code and "test" code (ie to
    readily ship source and test as separate components. Usually I'd do
    that with separate hierarchies (src vs test folders)
    o Enabling the running of individual unittests and aggregated
    testsuites
    o Avoiding namespace collisions - whilst still ensuring commonality
    (or ownership) of the projects (ie common base package)

    The longer version is detailed below which outlines my specific
    problem.

    I was hoping that someone could help me out with my python programming
    dilemma.I am having a huge problem organising my python projects so
    that the imports and from s to work correctly. I'm a Java/J2EE
    developer by trade so I have a reasonable grasp of the fundamentals
    regarding computer programming in general, but obviously not-so-good in
    python. I have noted from my reading on the web that my Java skills may
    actually be more of a hinderance - so feel free to tell me if I am
    making any programming java in python mistakes.

    I'm working on a few projects concurrently so I have tried to
    arranged my projects like this:

    COMMON
    src
    a.b.c.common
    test
    a.b.c.common

    APP1
    src
    a.b.c.app1
    test
    a.b.c.app1

    APP2
    src
    a.b.c.app2
    test
    a.b.c.app2


    But it has not worked due to import/from issues. It appears that using
    a common base package hierarchy (as is standard practice in the java
    world) caused issues. Ie not able to use "a.b.c" as the base
    package for all my projects, in order to avoid namespace collisions. I
    attempted to resolve this by altering the packages to make the project
    name as the base package. Ie the new packages were
    · common.a.b.c.common
    · app1.a.b.c.app1
    · app2.a.b.c.app2

    My final problem is that I am not able to run my tests (unittest) as I
    can't seem to find the right imports or froms to use that will work
    for both from an aggregate (suite) test and the individual tests. I
    have an individual AllTests.py at the root of the test folder and all
    the appropriate __init__.pys in all the folders (to make them package
    loaders). It appears to be that the directory from which the
    AllTests.py is run is subverting the path/package hierarchy as it is
    loaded first and is conflicting with my PYTHONPATH env variables (or I
    am getting the python packaging and path totally wrong - which is
    entirely possible).

    So how should a project to be structured and configured to enable all
    of these requirements? Any special requirements for the __init__.py,
    PYTHONPATH, or other configuration tricks?

    Thanks
    Todd
    , Apr 20, 2006
    #1
    1. Advertising

  2. Re: How should multiple (related) projects be arranged (structured)and configured so that they can share code, have a related package structureand enable proper unittesting, and ensuring no namespace collisions

    wrote:
    > Summary:

    (snip)
    >
    > I'm working on a few projects concurrently so I have tried to
    > arranged my projects like this:
    >
    > COMMON
    > src
    > a.b.c.common
    > test
    > a.b.c.common
    >
    > APP1
    > src
    > a.b.c.app1
    > test
    > a.b.c.app1
    >
    > APP2
    > src
    > a.b.c.app2
    > test
    > a.b.c.app2
    >
    >
    > But it has not worked due to import/from issues. It appears that using
    > a common base package hierarchy (as is standard practice in the java
    > world) caused issues. Ie not able to use "a.b.c" as the base
    > package for all my projects, in order to avoid namespace collisions.


    What you need to understand here is that Python's packages/modules
    system is directly tied to the filesystem. A package is a directory with
    a __init__.py file and possibly modules and sub-packages in it, and the
    package name is the directory name (plus parents packages for
    sub-packages). You *cannot* add arbitrary prefix like 'a.b.c' to the
    package name.

    Regarding import problems, I suggest that you re-read:
    http://www.python.org/doc/2.4.2/tut/node8.html#SECTION008400000000000000000
    with particular attention to 6.4.2.

    Note that you can dynamically modify sys.path - this may solve some
    problems with imports !-)

    Hoping some guru will be more helpfull...
    --
    bruno desthuilliers
    python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
    p in ''.split('@')])"
    bruno at modulix, Apr 20, 2006
    #2
    1. Advertising

  3. Paddy Guest

    Your second point 'clear separation of "production" code and "test"
    code' is best handled by using a revision control system. I know
    Clearcase, because I use it at work, but I have heard good things about
    Subversion.
    Paddy, Apr 20, 2006
    #3
  4. alisonken1 Guest

    I believe that Paddy was referencing his second point about keeping
    "production code" and "test code" clearly delimited, so was
    recommending that a version control system be used rather than the
    local disk structure required by python for building module packages.

    Although, I have found that symlinks work fine for delimiting file
    structure - ex.:

    $BASEDIR/<production files>

    $TESTDIR/<test files>
    $TESTDIR/<link to $BASEDIR>

    Using an example program that I wrote, the following directory
    structure snippet works for me (where 'bin' is a directory to shared
    modules of production code):

    -rw-r--r-- 1 ken users 509 2004-11-30 13:35 label.ini
    lrwxrwxrwx 1 ken users 9 2004-12-29 01:54 bin -> ../../bin/
    drwxr-xr-x 3 ken users 72 2004-12-10 16:01 data/
    drwxr-xr-x 2 ken users 80 2004-12-10 16:46 disp/
    drwxr-xr-x 2 ken users 48 2004-12-10 16:01 etc/
    drwxr-xr-x 2 ken users 80 2004-12-10 16:27 prn/
    drwxr-xr-x 2 ken users 208 2004-12-10 16:14 test/
    drwxr-xr-x 2 ken users 1208 2004-12-10 17:15 label.py
    alisonken1, Apr 20, 2006
    #4
  5. Paddy Guest

    Thanks alisonken1, I did indeed mean that when you have production code
    and test code then its time to bring in the version control systems.
    But having said that ;-) i have been naughty in the past and used the
    -s or --symbolic-link option of the gnu cp (sometimes available as
    gcp), command which copies a whole directory tree of the production
    release to a test area, copying directories but putting links in to the
    original files. I make sure that the production code is all read-only
    thenwork in the test area. If I need to modify a file I:
    * list the link
    * remove the link, replacing it with a copy of the file linked to
    * get write priviliges to the new copy
    * edit the copy.

    But we should all be using a version control system right :)

    - paddy.
    Paddy, Apr 20, 2006
    #5
  6. Re: How should multiple (related) projects be arranged (structured)and configured so that they can share code, have a related package structureand enable proper unittesting, and ensuring no namespace collisions

    Paddy a écrit :
    > Your second point 'clear separation of "production" code and "test"
    > code' is best handled by using a revision control system.


    I'm afraid I don't see the point here ???
    Bruno Desthuilliers, Apr 21, 2006
    #6
  7. alisonken1 Guest

    Not sure about "vcs" - but "cvs" can be more fun if the hard disk dies
    {g}.
    alisonken1, Apr 21, 2006
    #7
  8. alisonken1 Guest

    As to the question "fail to see how version control relates to
    code/test separation", the original poster asked several questions, one
    of which was production/testing code separation.

    Along with the separation (so while you're testing new functionality,
    you don't break production files), a properly setup CVS allows you to
    do this by importing files from a production branch into your testing
    branch so you can re-use vetted code (production) in your trial code
    (testing) without affecting what's already out there (inadvertently
    breaking currently shipping code to customers).
    alisonken1, Apr 21, 2006
    #8
  9. Re: How should multiple (related) projects be arranged (structured)and configured so that they can share code, have a related package structureand enable proper unittesting, and ensuring no namespace collisions

    Paddy a écrit :
    > Thanks alisonken1, I did indeed mean that when you have production code
    > and test code then its time to bring in the version control systems.


    Hopefully all this is under version control right from the start... And
    I still fail to see how version control relates to code/test separation
    (sorry, my only neuron left just refuses to work this late).

    (snip hacks)

    > But we should all be using a version control system right :)


    But we *are* all using a vcs - aren't we ?-)
    Bruno Desthuilliers, Apr 21, 2006
    #9
  10. Re: How should multiple (related) projects be arranged (structured)and configured so that they can share code,have a related package structureand enable proper unittesting, and ensuring no namespace collisions

    alisonken1 wrote:
    > As to the question "fail to see how version control relates to
    > code/test separation", the original poster asked several questions, one
    > of which was production/testing code separation.
    >
    > Along with the separation (so while you're testing new functionality,
    > you don't break production files), a properly setup CVS allows you to
    > do this by importing files from a production branch into your testing
    > branch so you can re-use vetted code (production) in your trial code
    > (testing) without affecting what's already out there (inadvertently
    > breaking currently shipping code to customers).
    >


    since I'm a wandering through the same paths, I would also point out
    that in addition to proper code management, I'm really coming to believe
    in the value of sand boxes or even complete virtual machines as part of
    your testing/production cycle. Unfortunately managing all the source
    code is still a bit of a CF (and I don't mean CompactFlash). I have yet
    to find a distributed configuration management system that works in a
    way that's a comfortable fit with how I work. darcs is the closest so
    far but even that is a bit unwieldy.

    I must admit however that part of my discomfort with development
    techniques and methods comes because I am dependent on speech
    recognition. And thanks to ScanSoft's (now nuance) inability to make a
    speech recognition application that can dictate sanely into
    non-Microsoft applications, I must jump through hoops to write Python on
    a UNIX box (NT Emacs + Webdrive). As a result my perspective is colored
    by my handicap in the same way the TABS experience colors your perception.

    ---eric
    Eric S. Johansson, Apr 21, 2006
    #10
  11. Guest

    Thanks for all the prompt helpful responses :- )

    I have read and reread section 6.4 -> of the python docs and it doesn't
    really help me.

    Just a couple of clarifications.

    I am already using a VCS (Version Control System) - subversion
    actually.

    When I mentioned a clear separation of production and test code I was
    more referring to this sort of arrangement:
    src
    a.b.c.common
    a.b.c.common.test
    which I believe is bad

    versus the clearer physical separation
    test
    a.b.c.commom.test

    The first option has the testing code within the same folder hierarchy
    which makes it harder to separate the two physically at "release" time
    without writing some complicated scripts to selectively exclude items.
    In the Java world the second option is the standard, and my preference
    if it's possible.

    I suspect that some of my import / from problems stem from the fact
    that I want to use a standard base package for all my projects to avoid
    namespace issues. ie all my projects use packages of the form

    a.b.c.projectName

    where a.b.c is my namespace (dns address actually)

    a.b.c.common
    a.b.c.project1
    a.b.c.project2

    Is this not the way python projects should be packaged?

    Is there a problem with having a common base package name like this?

    ie does the first instance of the package a.b.c effectively mask out
    the others so that if a.b.c.common occurs first in the PYTHONPATH then
    a.b.c.project1 will never be found as the "a.b.c" package is only
    loaded from the a.b.c.common folder?

    Does anyone else have any issues with running a master test suite and
    indvidual tests. It seems that if I set up the imports/froms for the
    master testsuite then I can't run the tests individuallly and vice
    versa ... unless I remove all packages and have a single flat
    directory. There has got to be big projects written in python that need
    to separate out their testing code according to some logical grouping.

    thanks
    Todd
    , Apr 21, 2006
    #11
  12. Peter Otten Guest

    wrote:

    > ie does the first instance of the package a.b.c effectively mask out
    > the others so that if a.b.c.common occurs first in the PYTHONPATH then
    > a.b.c.project1 will never be found as the "a.b.c" package is only
    > loaded from the a.b.c.common folder?


    By default, yes. But there's a way around this limitation:

    http://docs.python.org/lib/module-pkgutil.html

    Peter
    Peter Otten, Apr 21, 2006
    #12
  13. Re: How should multiple (related) projects be arranged (structured)and configured so that they can share code, have a related package structureand enable proper unittesting, and ensuring no namespace collisions

    alisonken1 wrote:
    > As to the question "fail to see how version control relates to
    > code/test separation", the original poster asked several questions, one
    > of which was production/testing code separation.
    >
    > Along with the separation (so while you're testing new functionality,
    > you don't break production files), a properly setup CVS allows you to
    > do this by importing files from a production branch into your testing
    > branch


    Ok, get it - you're talking about branching, when the OP talked about
    unit testing, which is something totally different.


    --
    bruno desthuilliers
    python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
    p in ''.split('@')])"
    bruno at modulix, Apr 21, 2006
    #13
  14. ToddLMorgan Guest

    Thanks everyone for their assistance.

    I have managed to achieve all that I set out to do:
    - separation between src and test folders
    - now successfully sharing code between projects
    - running unittest s and suites
    - avoiding namespace collisions

    The solution involved the following (if anyone else is interested)
    - ammending all my __init__.py packages so that they included the
    following:

    from pkgutil import extend_path
    __path__ = extend_path(__path__, __name__)

    as per Peter Otten's suggestion

    - refactoring all the code so that the imports and froms conformed to
    the new package structure (a.b.c.common, a.b.c.app1 etc) and physically
    moving all the required files

    - ammending the PYTHONPATH so that the src and test directories for
    each project is included at the time of running

    ie
    PYTHONPATH=COMMON/src;COMMON/test;APP1/src;APP1/test;APP2/src;APP2/test

    Of course the /test entries are only required for testing and not
    runtime but you get the idea.

    I understand that flatter package structures are apparently the python
    way (http://dirtsimple.org/2004/12/python-is-not-java.html) but I like
    a nice little small little related package of functionality when I am l
    carving up a complex problem so I'm happy to incur any extra
    performance penalty in dict lookups.

    thanks again

    Todd
    ToddLMorgan, Apr 21, 2006
    #14
  15. alisonken1 Guest

    It can be fun when talking several subjects in the same post <g>.

    In this case, it was just a matter of thinking about what the main
    question was about unit testing code that also required other
    production packages, but with the caveats that he didn't want to
    duplicate packages just to test code or hack around during testing,
    then have to change the test code when it was time to add to production
    code (which may introduce more errors that the testing would not find).

    CVS just found it's way from one of the suggestions as a side note in
    reply to a minor point about how to keep track of test code without
    messing with/duplicating production code.
    alisonken1, Apr 21, 2006
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Maciek Zywno
    Replies:
    7
    Views:
    385
    Andrew Thompson
    Nov 16, 2004
  2. Maxwell Hammer
    Replies:
    7
    Views:
    623
    Peter Hansen
    Jun 18, 2005
  3. Mahesh S
    Replies:
    0
    Views:
    292
    Mahesh S
    Jan 15, 2007
  4. sumit chawla
    Replies:
    1
    Views:
    523
    Thomas Matthews
    Jul 13, 2004
  5. botp
    Replies:
    6
    Views:
    184
    Joel VanderWerf
    Oct 5, 2010
Loading...

Share This Page