organizing many python scripts, in a large corporate environment.

[Phat Fly Alanna]

We've been doing a fair amount of Python scripting, and now we have a
directory with almost a hundred loosely related scripts. It's
obviously time to organize this, but there's a problem. These scripts
import freely from each other and although code reuse is generally a
good thing it makes it quite complicated to organize them into
directories.

There's a few things that you should know about our corporate
environment:

1) I don't have access to the users' environment. Editing the
PYTHONPATH is out, unless it happens in the script itself.
2) Users don't install things. Systems are expected to be *already*
installed and working, so setup.py is not a solution.

I'm quite willing to edit my import statements and do some minor
refactoring, but the solutions I see currently require me to divide
all the code strictly between "user runnable scripts" and
"libraries", which isn't feasible, considering the amount of code.

Has anyone out there solved a similar problem? Are you happy with it?

--Buck

 

[Tim Johnson]

We've been doing a fair amount of Python scripting, and now we have a
directory with almost a hundred loosely related scripts. It's
obviously time to organize this, but there's a problem. These scripts
import freely from each other and although code reuse is generally a
good thing it makes it quite complicated to organize them into
directories.

There's a few things that you should know about our corporate
environment:

1) I don't have access to the users' environment. Editing the
PYTHONPATH is out, unless it happens in the script itself.
2) Users don't install things. Systems are expected to be *already*
installed and working, so setup.py is not a solution.

I'm quite willing to edit my import statements and do some minor
refactoring, but the solutions I see currently require me to divide
all the code strictly between "user runnable scripts" and
"libraries", which isn't feasible, considering the amount of code.

Has anyone out there solved a similar problem? Are you happy with it?

Slightly similar - time doesn't permit details, but I used among
other things 4 methods that worked well for me:
1)'Site modules' with controlling constants,including paths
2)Wrappers for the __import__ function that enabled me to fine -
tune where I was importing from.
3)Cut down on the number of executables by using 'loaderers'.
4)I modified legacy code to take lessons from the MVC architecture,
and in fact my architecture following these changes could be
called 'LMVCC' for
loader
model
view
controller
config

I hope I've made some sense with these brief sentences.

 

[Tim Johnson]

* Tim Johnson <[email protected]> [110312 10:41]:
<...> 3)Cut down on the number of executables by using 'loaderers'.
Sheesh! Typo, meant to say 'loaders'..
sorry

 

[bukzor]

* Phat Fly Alanna <[email protected]> [110312 07:22]:

We've been doing a fair amount of Python scripting, and now we have a
directory with almost a hundred loosely related scripts. It's
obviously time to organize this, but there's a problem. These scripts
import freely from each other and although code reuse is  generally a
good thing it makes it quite complicated to organize them into
directories.

There's a few things that you should know about our corporate
environment:

1) I don't have access to the users' environment. Editing the
PYTHONPATH is out, unless it happens in the script itself.
2) Users don't install things. Systems are expected to be *already*
installed and working, so setup.py is not a solution.

I'm quite willing to edit my import statements and do some minor
refactoring, but the solutions I see currently require me to divide
all  the code strictly between "user runnable scripts" and
"libraries", which isn't feasible, considering the amount of code.

Has anyone out there solved a similar problem? Are you happy with it?

  Slightly similar - time doesn't permit details, but I used among
  other things 4 methods that worked well for me:
  1)'Site modules' with controlling constants,including paths
  2)Wrappers for the __import__ function that enabled me to fine -
  tune where I was importing from.
  3)Cut down on the number of executables by using 'loaderers'.
  4)I modified legacy code to take lessons from the MVC architecture,
  and in fact my architecture following these changes could be
  called 'LMVCC' for
  loader
  model
  view
  controller
  config

  I hope I've made some sense with these brief sentences.

Thanks Tim.

I believe I understand it. You create loaders in a flat organization,
in the same directory as your shared library, so that it's found
naturally. These loaders use custom code to find and run the "real"
scripts. This seems to be a combination of solutions d) and e) in my
above post.

This is a solution I hadn't considered.

It seems to have few disadvantages, although it does obfuscate where
to find the "real" code somewhat. It also has the implicit requirement
that all of your scripts can be categorized into a few top-level
categories. I'll have to think about whether this applies in my
situation...

Thanks again,
--Buck

 

[Tim Johnson]

Thanks Tim.

I believe I understand it. You create loaders in a flat organization,
in the same directory as your shared library, so that it's found

Not in the same directory as shared libraries.

naturally. These loaders use custom code to find and run the "real"
scripts. This seems to be a combination of solutions d) and e) in my
above post.

In my case, the loader
1)Executes code that would otherwise be duplicated.
2)Loads modules (usually from a lower-level directory) based on
keywords passed as a URL segment.

It seems to have few disadvantages, although it does obfuscate where
to find the "real" code somewhat. It also has the implicit requirement
that all of your scripts can be categorized into a few top-level
categories.

Correct. In my case that is desirable.

I'll have to think about whether this applies in my
situation...

cheers