How should multiple (related) projects be arranged (structured) and configured so that they can shar

[ToddLMorgan]

Summary:
How should multiple (related) projects be arranged (structured) and
configured so that the following is possible:
o Sharing common code (one of the projects would be a "common"
project referenced by all others and likely the others would share at
least the common project and possibly more as times goes on)
o Clear separation of "production" code and "test" code (ie to
readily ship source and test as separate components. Usually I'd do
that with separate hierarchies (src vs test folders)
o Enabling the running of individual unittests and aggregated
testsuites
o Avoiding namespace collisions - whilst still ensuring commonality
(or ownership) of the projects (ie common base package)

The longer version is detailed below which outlines my specific
problem.

I was hoping that someone could help me out with my python programming
dilemma.I am having a huge problem organising my python projects so
that the imports and from s to work correctly. I'm a Java/J2EE
developer by trade so I have a reasonable grasp of the fundamentals
regarding computer programming in general, but obviously not-so-good in
python. I have noted from my reading on the web that my Java skills may
actually be more of a hinderance - so feel free to tell me if I am
making any programming java in python mistakes.

I'm working on a few projects concurrently so I have tried to
arranged my projects like this:

COMMON
src
a.b.c.common
test
a.b.c.common

APP1
src
a.b.c.app1
test
a.b.c.app1

APP2
src
a.b.c.app2
test
a.b.c.app2


But it has not worked due to import/from issues. It appears that using
a common base package hierarchy (as is standard practice in the java
world) caused issues. Ie not able to use "a.b.c" as the base
package for all my projects, in order to avoid namespace collisions. I
attempted to resolve this by altering the packages to make the project
name as the base package. Ie the new packages were
· common.a.b.c.common
· app1.a.b.c.app1
· app2.a.b.c.app2

My final problem is that I am not able to run my tests (unittest) as I
can't seem to find the right imports or froms to use that will work
for both from an aggregate (suite) test and the individual tests. I
have an individual AllTests.py at the root of the test folder and all
the appropriate __init__.pys in all the folders (to make them package
loaders). It appears to be that the directory from which the
AllTests.py is run is subverting the path/package hierarchy as it is
loaded first and is conflicting with my PYTHONPATH env variables (or I
am getting the python packaging and path totally wrong - which is
entirely possible).

So how should a project to be structured and configured to enable all
of these requirements? Any special requirements for the __init__.py,
PYTHONPATH, or other configuration tricks?

Thanks
Todd

 

[bruno at modulix]

Summary: (snip)

I'm working on a few projects concurrently so I have tried to
arranged my projects like this:

COMMON
src
a.b.c.common
test
a.b.c.common

APP1
src
a.b.c.app1
test
a.b.c.app1

APP2
src
a.b.c.app2
test
a.b.c.app2


But it has not worked due to import/from issues. It appears that using
a common base package hierarchy (as is standard practice in the java
world) caused issues. Ie not able to use "a.b.c" as the base
package for all my projects, in order to avoid namespace collisions.

What you need to understand here is that Python's packages/modules
system is directly tied to the filesystem. A package is a directory with
a __init__.py file and possibly modules and sub-packages in it, and the
package name is the directory name (plus parents packages for
sub-packages). You *cannot* add arbitrary prefix like 'a.b.c' to the
package name.

Regarding import problems, I suggest that you re-read:
http://www.python.org/doc/2.4.2/tut/node8.html#SECTION008400000000000000000
with particular attention to 6.4.2.

Note that you can dynamically modify sys.path - this may solve some
problems with imports !-)

Hoping some guru will be more helpfull...

 

[Paddy]

Your second point 'clear separation of "production" code and "test"
code' is best handled by using a revision control system. I know
Clearcase, because I use it at work, but I have heard good things about
Subversion.

 

[alisonken1]

I believe that Paddy was referencing his second point about keeping
"production code" and "test code" clearly delimited, so was
recommending that a version control system be used rather than the
local disk structure required by python for building module packages.

Although, I have found that symlinks work fine for delimiting file
structure - ex.:

$BASEDIR/<production files>

$TESTDIR/<test files>
$TESTDIR/<link to $BASEDIR>

Using an example program that I wrote, the following directory
structure snippet works for me (where 'bin' is a directory to shared
modules of production code):

-rw-r--r-- 1 ken users 509 2004-11-30 13:35 label.ini
lrwxrwxrwx 1 ken users 9 2004-12-29 01:54 bin -> ../../bin/
drwxr-xr-x 3 ken users 72 2004-12-10 16:01 data/
drwxr-xr-x 2 ken users 80 2004-12-10 16:46 disp/
drwxr-xr-x 2 ken users 48 2004-12-10 16:01 etc/
drwxr-xr-x 2 ken users 80 2004-12-10 16:27 prn/
drwxr-xr-x 2 ken users 208 2004-12-10 16:14 test/
drwxr-xr-x 2 ken users 1208 2004-12-10 17:15 label.py

 

[Paddy]

Thanks alisonken1, I did indeed mean that when you have production code
and test code then its time to bring in the version control systems.
But having said that ;-) i have been naughty in the past and used the
-s or --symbolic-link option of the gnu cp (sometimes available as
gcp), command which copies a whole directory tree of the production
release to a test area, copying directories but putting links in to the
original files. I make sure that the production code is all read-only
thenwork in the test area. If I need to modify a file I:
* list the link
* remove the link, replacing it with a copy of the file linked to
* get write priviliges to the new copy
* edit the copy.

But we should all be using a version control system right

- paddy.

 

[Bruno Desthuilliers]

Paddy a écrit :

Your second point 'clear separation of "production" code and "test"
code' is best handled by using a revision control system.

I'm afraid I don't see the point here ???

 

[alisonken1]

Not sure about "vcs" - but "cvs" can be more fun if the hard disk dies
{g}.

 

[alisonken1]

As to the question "fail to see how version control relates to
code/test separation", the original poster asked several questions, one
of which was production/testing code separation.

Along with the separation (so while you're testing new functionality,
you don't break production files), a properly setup CVS allows you to
do this by importing files from a production branch into your testing
branch so you can re-use vetted code (production) in your trial code
(testing) without affecting what's already out there (inadvertently
breaking currently shipping code to customers).

 

[Bruno Desthuilliers]

Paddy a écrit :

Thanks alisonken1, I did indeed mean that when you have production code
and test code then its time to bring in the version control systems.

Hopefully all this is under version control right from the start... And
I still fail to see how version control relates to code/test separation
(sorry, my only neuron left just refuses to work this late).

(snip hacks)

But we should all be using a version control system right

But we *are* all using a vcs - aren't we ?-)

 

[Eric S. Johansson]

As to the question "fail to see how version control relates to
code/test separation", the original poster asked several questions, one
of which was production/testing code separation.

Along with the separation (so while you're testing new functionality,
you don't break production files), a properly setup CVS allows you to
do this by importing files from a production branch into your testing
branch so you can re-use vetted code (production) in your trial code
(testing) without affecting what's already out there (inadvertently
breaking currently shipping code to customers).

since I'm a wandering through the same paths, I would also point out
that in addition to proper code management, I'm really coming to believe
in the value of sand boxes or even complete virtual machines as part of
your testing/production cycle. Unfortunately managing all the source
code is still a bit of a CF (and I don't mean CompactFlash). I have yet
to find a distributed configuration management system that works in a
way that's a comfortable fit with how I work. darcs is the closest so
far but even that is a bit unwieldy.

I must admit however that part of my discomfort with development
techniques and methods comes because I am dependent on speech
recognition. And thanks to ScanSoft's (now nuance) inability to make a
speech recognition application that can dictate sanely into
non-Microsoft applications, I must jump through hoops to write Python on
a UNIX box (NT Emacs + Webdrive). As a result my perspective is colored
by my handicap in the same way the TABS experience colors your perception.

---eric

 

[ToddLMorgan]

Thanks for all the prompt helpful responses :- )

I have read and reread section 6.4 -> of the python docs and it doesn't
really help me.

Just a couple of clarifications.

I am already using a VCS (Version Control System) - subversion
actually.

When I mentioned a clear separation of production and test code I was
more referring to this sort of arrangement:
src
a.b.c.common
a.b.c.common.test
which I believe is bad

versus the clearer physical separation
test
a.b.c.commom.test

The first option has the testing code within the same folder hierarchy
which makes it harder to separate the two physically at "release" time
without writing some complicated scripts to selectively exclude items.
In the Java world the second option is the standard, and my preference
if it's possible.

I suspect that some of my import / from problems stem from the fact
that I want to use a standard base package for all my projects to avoid
namespace issues. ie all my projects use packages of the form

a.b.c.projectName

where a.b.c is my namespace (dns address actually)

a.b.c.common
a.b.c.project1
a.b.c.project2

Is this not the way python projects should be packaged?

Is there a problem with having a common base package name like this?

ie does the first instance of the package a.b.c effectively mask out
the others so that if a.b.c.common occurs first in the PYTHONPATH then
a.b.c.project1 will never be found as the "a.b.c" package is only
loaded from the a.b.c.common folder?

Does anyone else have any issues with running a master test suite and
indvidual tests. It seems that if I set up the imports/froms for the
master testsuite then I can't run the tests individuallly and vice
versa ... unless I remove all packages and have a single flat
directory. There has got to be big projects written in python that need
to separate out their testing code according to some logical grouping.

thanks
Todd

 

[Peter Otten]

ie does the first instance of the package a.b.c effectively mask out
the others so that if a.b.c.common occurs first in the PYTHONPATH then
a.b.c.project1 will never be found as the "a.b.c" package is only
loaded from the a.b.c.common folder?

By default, yes. But there's a way around this limitation:

http://docs.python.org/lib/module-pkgutil.html

Peter

 

[bruno at modulix]

As to the question "fail to see how version control relates to
code/test separation", the original poster asked several questions, one
of which was production/testing code separation.

Along with the separation (so while you're testing new functionality,
you don't break production files), a properly setup CVS allows you to
do this by importing files from a production branch into your testing
branch

Ok, get it - you're talking about branching, when the OP talked about
unit testing, which is something totally different.

 

[ToddLMorgan]

Thanks everyone for their assistance.

I have managed to achieve all that I set out to do:
- separation between src and test folders
- now successfully sharing code between projects
- running unittest s and suites
- avoiding namespace collisions

The solution involved the following (if anyone else is interested)
- ammending all my __init__.py packages so that they included the
following:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

as per Peter Otten's suggestion

- refactoring all the code so that the imports and froms conformed to
the new package structure (a.b.c.common, a.b.c.app1 etc) and physically
moving all the required files

- ammending the PYTHONPATH so that the src and test directories for
each project is included at the time of running

ie
PYTHONPATH=COMMON/src;COMMON/test;APP1/src;APP1/test;APP2/src;APP2/test

Of course the /test entries are only required for testing and not
runtime but you get the idea.

I understand that flatter package structures are apparently the python
way (http://dirtsimple.org/2004/12/python-is-not-java.html) but I like
a nice little small little related package of functionality when I am l
carving up a complex problem so I'm happy to incur any extra
performance penalty in dict lookups.

thanks again

Todd

 

[alisonken1]

It can be fun when talking several subjects in the same post <g>.

In this case, it was just a matter of thinking about what the main
question was about unit testing code that also required other
production packages, but with the caveats that he didn't want to
duplicate packages just to test code or hack around during testing,
then have to change the test code when it was time to add to production
code (which may introduce more errors that the testing would not find).

CVS just found it's way from one of the suggestions as a side note in
reply to a minor point about how to keep track of test code without
messing with/duplicating production code.