How to organize Python files in a (relatively) big project

Discussion in 'Python' started by TokiDoki, Oct 19, 2005.

  1. TokiDoki

    TokiDoki Guest

    Hello there,

    I have been programming python for a little while, now. But as I am
    beginning to do more complex stuff, I am running into small organization
    It is possible that what I want to obtain is not possible, but I would
    like the advice of more experienced python programmers.

    I am writing a relatively complex program in python that has now around
    40 files.
    At first, I had all of my files in one single directory, but now, with
    the increasing number of files, it is becoming hard to browse my directory.
    So, I would want to be able to divide the files between 8 directory,
    according to their purpose. The problem is that it breaks the 'import's
    between my files. And besides,AFAIK, there is no easy way to import a
    file that is not in a subdirectory of the current file (I suppose I
    could adjust the os.path in every file, but that seems not very elegant
    to me).
    I thought about turning the whole into a package. But I have to change
    every 'import module_name' into
    'from package_name.sub_directory_name import module_name'
    which is a little bit time consuming and not very flexible (if I change
    my mind about the subdirectory a file belongs to, I will have to track
    again every import to correct them)

    So, basically, here is the point: is there an 'elegant' way to keep my
    files in separate directory and still be able to import between them
    with simple 'import filename'?

    And if not, what would be the standard way to organize numerous python
    files belonging to the same project?

    Thank you,

    TokiDoki, Oct 19, 2005
    1. Advertisements

  2. TokiDoki

    spinner Guest

    I have this problem myself, and as I am a recent Python convert my
    aproach may not be conventional - but as it is working for me I thought
    I would share.

    First off you need to split your code into logical objects. In my case
    I have an obvious client and server object, But within each I have
    individual stand alone major objects and so my directory structure
    reflects this...

    Prog Dir
    Browser 1
    Browser 2
    Server Object 1
    Server Object 2

    Each file is used to call each downstream file be it as
    a thread or once only run. The beauty of this idea is that if I want
    to completely change say Server Object 2, I can do whatever I like
    within the downstream directory as the only outside call is via an
    external reference to the file.

    If you have a mind, you could have a library file of common apps held
    within the bin directory that you can make available to all downstream

    As I say - it only works if you can logically split your program into
    independent blocks.

    If there is a more Pythonesque way of doing this I be grateful if
    someone could share.

    spinner, Oct 19, 2005
    1. Advertisements

  3. Remember that the directory where you start the toplevel script is always
    included in the sys.path. This means that you can have your structure like
    | - - pkg1
    | - - pkg2
    | - - pkg3

    Files in any package can import other packages. The usual way is to do "import
    pkgN" and then dereference. Within each package, you will have a
    which will define the package API (that is, will define those symbols that you
    can access from outside the package).

    Typically, you only want to import *packages*, not submodules. In other words,
    try to not do stuff like "from pkg1.submodule3 import Foo", because this breaks
    encapsulation (if you reorganize the structure of pkg1, code will break). So
    you'd do "import pgk1" and later "pkg1.Foo", assuming that
    does something like "from submodule3 import Foo". My preferred way is to have just do "from submodules import *", and then each submodule defines
    __all__ to specify which are its public symbols. This allow for more
    encapsulation (each submodule is able to change what it exports in a
    self-contained way, you don't need to modify __init__ as well).

    Moreover, the dependence graph between packages shouldn't have loops. For
    instance, if pkg3 uses pkg1, pkg1 shouldn't use pkg3. It makes sense to think
    of pkg1 as the moral equivalent of a library, which pkg3 uses.
    Giovanni Bajo, Oct 19, 2005
  4. If I understand you right you need a concept in which you can put the files
    of your project where you want, i.e. restructure the nesting of directories
    storing your scripts without the problem of breaking the import statements.
    This will give you not a solution to any problem you maybe have with
    managing a large package but is a problem in itself, so I see that you have
    a huge number of options to choose from:

    1. best is you get a better structure of the project in your mind, so you
    don't need frequent changes by resorting your modules into a various
    directories (flat is better than nested). Set a size limit of a single
    Python script it must reach before you split it into two separate files, so
    that the number of your files decreases to a number you can handle with ease
    (this depends on how good you are at it).

    2. create e.g. an HTML file from which you can view, edit, execute the
    scripts you have. In this file you create your nested structure and have
    this way the overview you need. The Python files remain in one directory
    storing the entire project. This allows you to keep together what belongs
    together, but to work with it as the single files were spread over many
    categories (I mean directories :).

    3. create a Python script which automatically creates an HTML file giving
    you the overview of all your scripts from walking the directories your
    project files are stored below and put in each of your scripts a function
    which on call returns the actual full path of passed name of the file to
    import and updates the Python PATH, so that it can be found (e.g

    4. start to choose from many tools for project management which will keep
    you busy for a while and away from the actual work on your project ...

    None of these above can solve the supposed actual problem, that the project
    grows beyond the capacity to manage it in mind. I know about this limitation
    and I have seen programmers failing on projects with increasing size of
    them. Usually the dependencies between the single modules in large projects
    get more and more complicated. Each next added module making the project
    size larger must work with the entire already existing ones and after a
    given size of project is reached it becomes really hard (and for some just
    impossible) to add a new module to it making the necessary modifications to
    existing ones in order to gain the new functionality. Such modifications
    result usually in breaks in other parts of the project and in my eyes no
    management system can really help in anticipating it at that point. Every
    programmer seems to have a limit of a project size he can manage and in my
    opinion no tool for structuring it or getting a better overview is of real
    help here. The core of the problem is lack of skills required for managing
    large projects and it seems, that not everyone who managed to learn to
    program is able to acquire this skills necessary to increase the size of a
    project over the by his intellectual capacity given limits. That is why
    programmer need to work in teams and have a very good manager able to
    coordinate their efforts in order to succeed in creating large applications.
    In really large projects it is sometimes necessary to work half a year or
    more full time on the existing library of code before becoming capable of
    writing a single line of additional one.

    Publish your code, let others work with it, try to get responses on it and
    you will probably get much more out of this than by asking for ways of
    managing it.

    I am curious if others on this newsgroup agree with what I said above, so I
    would be glad to hear about it.

    Claudio Grondi, Oct 19, 2005
  5. Maybe looking at the todays thread ["dynamical" importing] can be helpful
    also here.

    P.S. Below a copy of one of the responses:
    some alternatives:

    - if you want the modules to remain imported:

    sys.path.insert(0, os.path.join(module_dir, "bin"))
    module = __import__("my_module")
    del sys.path[0]
    object = module.my_object

    - if you're only interested in the object:

    namespace = {}
    execfile(os.path.join(module_dir, "bin", "my_module" + ".py"),
    object = namespace["my_object"]

    Claudio Grondi, Oct 19, 2005
  6. I, too have often come up against the inconvenience of creating
    libraries that I want to reuse but that do not want to incorporate into
    the Python library. I came up with this Python library addition to
    automagically add the directory of where a module resides that I want
    to import to the system path, so that I can simply add the module by
    name without needing to worry about anything else.

    The large advantage of this is that I can now import from adjacent
    directories rather than subordinate ones. For best results in large
    projects with deep tree structure, the paths of the importing script
    and the imported module should have at least 3 levels in common. I do
    not consider typing 3 levels of pathing as a hint to be a great

    It ain't perfect, it ain't Zen, but for me this has worked from Python
    2.2 and up with very few problems for projects up to 1000 files.
    Improvements, anyone?

    The Eternal Squire
    # Place in Python##\Lib
    import sys
    from inspect import getfile
    from os.path import abspath

    backslash = '\\'

    def caller ():

    'name of the calling script'

    frame = sys._getframe(1)
    if not frame: return ''
    frame = frame.f_back
    if not frame: return ''
    result = getfile (frame)
    if result[-4:].lower () in ['.pyc', '.pyo']: result = filename[:-4]
    + '.py'
    return abspath (result)

    def append (path):

    'append relative path to system module search path'

    if not path: return

    calling_script = caller ()

    top_path = path.split (backslash)[0]
    where = calling_script.find (top_path + backslash)

    if where < 0: return
    result = calling_script [:where] + path

    if result not in sys.path: sys.path.append (result)
    return result

    return ''
    #---------------------end of

    #---------------------"A\B\C\more levels\"-----------------#
    #prototypical code, do not try to execute

    import relative
    relative.append ("\A\B\C\subpath") # no file extension!
    from module import whatsit # module resides under subpath
    whatsit ()
    The Eternal Squire, Oct 19, 2005
  7. TokiDoki

    Jarek Zgoda Guest

    Giovanni Bajo napisa³(a):
    How to install this structure eg. on Linux? What layout do you
    recommend? It's tempting to use /opt hierarchy for installation target
    (as it gives relatively much freedom within application directory), but
    many administrators are reluctant to use this hierarchy and prefer more
    standarized targets, such as /usr or /usr/local.
    Jarek Zgoda, Oct 19, 2005
  8. TokiDoki

    Jarek Zgoda Guest

    Micah Elliott napisa³(a):
    I think that installing *application* (not misc. library) modules in
    site-packages is at least stupid idea, as it makes a special filesystem
    inside other, much more standarized filesystem (Windows, FHS, etc.). Why
    not to do this usual way: libraries to $prefix/lib/$appname-$version,
    binary to $prefix/bin, shared files to $prefix/share/$appname, etc --
    and appropriately on Windows.
    Jarek Zgoda, Oct 19, 2005
  9. Read about (and use) the Python-provided distutils, and let it do the
    work for you. In particular,
    discusses installation location. The file name is a misnomer; it's
    equally applicable to linux.

    It allows users to easily specify the installation prefix. The
    default should be the value of "sys.prefix".

    Note that your modules/packages will be installed within your python
    installation under its site-packages/ directory.
    Micah Elliott, Oct 19, 2005

  10. I am not very experienced with Linux, but if you don't use something like
    PyInstaller, you could still install the main tree somewhere like
    /usr/local/myapp/ and then generate a simple script for /usr/local/bin which
    adds /usr/local/myapp to sys.path[0], and "import main" to boot the

    I'm not sure I have answered your question though :)
    Giovanni Bajo, Oct 20, 2005
  11. They're not.
    It is done more the FHS way than you might realize. Assuming
    sys.prefix is '/usr', then using distutils my manpages go to
    '/usr/share/man', my tools/scripts that I want my users to access go
    to '/usr/bin' (and I could even put my config files in '/usr/../etc'
    or maybe just '/etc' but I haven't had to do that). So only your
    modules end up in '/usr/lib/python', but that seems appropriate
    anyway. It is the simplest way for '/usr/bin/python' to locate your
    modules, since it already knows to look in '/usr/lib/python' for
    modules. If you put your modules in say /usr/lib/fooapp then how
    would python know where to look for your modules? You would have
    users messing with something called PYTHONPATH or you would end up
    with boilerplate or custom install logic to set sys.path according to
    special user needs. But why go through the trouble? Just let
    distutils do the work for you.

    I don't want my users to know anything about my *modules* that end up
    in /usr/lib/python or that such a thing as PYTHONPATH even exists.

    If you prefer the /opt route (or /tmp or $HOME or whatever), then
    distutils can create your filesystem structure beneath that
    Micah Elliott, Oct 20, 2005
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.