how to structure a directory with many scripts and shared code

[Benedict Verheyen]

Hi,

i wanted to ask how you guys structure your code.
I mainly have scripts that automate small tasks.
These scripts use a common set of utility code.
My code is structured like this, for example:

script_1.py
script_2.py
script_3.py
tools\
|----->__init__.py
|----->logutils.py
|----->mailutils.py

I was thinking of putting code in seperate directories because the number
of scripts grows fast and i want to keep it clean.
Something like this:

script_1.py
tools\
|----->__init__.py
|----->logutils.py
|----->mailutils.py
database\
|----->__init__.py
|----->script_2.py
|----->script_3.py

However, when i make a subdirectory, for example database and put a script in there,
how would i import logutils or mailtutils from within the database subdirectory?
This fails:
from tools.logutils import logger

Thanks,
Benedict

 

[Steven D'Aprano]

However, when i make a subdirectory, for example database and put a
script in there, how would i import logutils or mailtutils from within
the database subdirectory? This fails:
from tools.logutils import logger

Sounds like you need to ensure that the top level directory needs to be
added to your PYTHONPATH.

Do you need instructions to do this?

 

[Benedict Verheyen]

Sounds like you need to ensure that the top level directory needs to be
added to your PYTHONPATH.

Do you need instructions to do this?

Hi Steve,


thanks, i think i know how to add the top level directory to the PYTHONPATH.

I'm however looking further into the suggestion of ssteinerX to use a
setup.py file to add the tools utility scripts to the pythonpath.
In the near future, another person will be helping me with programming
and thus, i want to make sure the structure is usable by other people
than myself.

Furthermore, i'm thinking if it wouldn't be cleaner to make such a setup.py
file per script, at least the bigger, more important scripts.
Now my structure is like this:

python_scripts
|
|-->trunk
.......|-----> all scripts
.......|-----> tools

The python_scripts directory and subdirs are versioned via SVN.
I think these steps would help:

1. Make a package of tools.
2. Put the bigger scripts in a seperate directory also using a setup.py file.
3. If possible, put the other scripts that somehow belong together in a seperate
directory. Names need to be more clear but i'm just illustrating a point.

python_scripts
|
|-->trunk
.......|-----> my big script 1
.................|-----> setup.py
.......|-----> my big script 2
.................|-----> setup.py
.......|-----> database
.................|-----> database script 1
.................|-----> database script 2
.......|-----> tools
.................|-----> setup.py

Does that look like a clear structure?

Thanks,
Benedict

 

[ssteinerX]

Hi Steve,


thanks, i think i know how to add the top level directory to the PYTHONPATH.

I'm however looking further into the suggestion of ssteinerX to use a
setup.py file to add the tools utility scripts to the pythonpath.
In the near future, another person will be helping me with programming
and thus, i want to make sure the structure is usable by other people
than myself.

Furthermore, i'm thinking if it wouldn't be cleaner to make such a setup.py
file per script, at least the bigger, more important scripts.
Now my structure is like this:

python_scripts
|
|-->trunk
......|-----> all scripts
......|-----> tools

The python_scripts directory and subdirs are versioned via SVN.
I think these steps would help:

1. Make a package of tools.
2. Put the bigger scripts in a seperate directory also using a setup.py file.
3. If possible, put the other scripts that somehow belong together in a seperate
directory. Names need to be more clear but i'm just illustrating a point.

python_scripts
|
|-->trunk
......|-----> my big script 1
................|-----> setup.py
......|-----> my big script 2
................|-----> setup.py
......|-----> database
................|-----> database script 1
................|-----> database script 2
......|-----> tools
................|-----> setup.py

Does that look like a clear structure?

No.

Make one setup.py at the top, put related scripts (like database) into separate sub-modules, so they can all be imported off a common 'trunk' as you have it above i.e. as trunk.database.xxx.

Then use entry points for any command line scripts.

Slightly harder, to setup initially but much cleaner to use and maintain and, it all installs with one call to setup.py develop.

S

 

[Benedict Verheyen]

On Feb 16, 2010, at 3:28 AM, Benedict Verheyen wrote:

No.

Make one setup.py at the top, put related scripts (like database) into separate sub-modules,
so they can all be imported off a common 'trunk' as you have it above i.e. as trunk.database.xxx.

Then use entry points for any command line scripts.

Slightly harder, to setup initially but much cleaner to use and maintain and, it all installs with one call to setup.py develop.

S

Hhhm, i can see how it makes the maintenance cleaner.
However, in the case of a single setup.py, i will end up installing scripts on servers
that will never use them, only some.
As with making egg files for the bigger scripts, you will only install what's needed.
Then again, the script are small so in that respect it doesn't really matter.

How would i best distribute it? Via the single setup.py or building an egg from the
setup.py or plain copying the files to the server?
The code is company specific so nobody else will benefit from it.

Thanks,
Benedict

 

[Jean-Michel Pichavant]

Hhhm, i can see how it makes the maintenance cleaner.
However, in the case of a single setup.py, i will end up installing scripts on servers
that will never use them, only some.

You should care about this only if your scripts exceeed a faire amount
of Ko.
Are you sure you've written Mo of python code ?

JM

 

[Benedict Verheyen]

Benedict Verheyen wrote:
You should care about this only if your scripts exceeed a faire amount
of Ko.
Are you sure you've written Mo of python code ?

JM

Nope and i indicated that before:

"Then again, the script are small so in that respect it doesn't really matter."

Regards,
Benedict