Python for Large Projects

Discussion in 'Python' started by Ixokai, May 16, 2004.

  1. Ixokai

    Ixokai Guest

    Hello all. :)

    I've been a long time Python fan, and have fairly recently (with the
    support of a coworker who surprised me by mentioning my pet language one
    day) convinced my company to begin the colossal task of basically rewriting
    all of our software in Python. Woohoo.
    Previously we used a few different development environments, mostly
    Borland, for different products in our 'system' of thick clients sort of
    operating with eachother as they dug against a database. The whole thing is
    woefully out of date at this point, and platform-dependant, so we needed to
    switch to something.. and Python won. Yay.
    In the new platform, we'll be building a distributed system with CORBA
    (omniORB + omniORBpy, etc) with Python making up most of our development for
    the new edition of our software. Woohoo. Much fun.

    My question is not 'how suitable is Python for large projects?', but
    instead more, 'Do you have any advice for large projects with Python?'.

    For example, our system will be made up of numerous applications --
    three or four 'servers', multiple clients, all talking to eachother, sharing
    a great deal of code. Any suggestions for organization? Our current plan is
    to put everything under the 'site-packages' tree of a Python installation we
    provide/control, with a fairly deeply nested tree:

    apt
    apt\library
    apt\library\corba.py
    apt\library\database\*
    apt\library\controls\*
    apt\application\globalapp1\
    apt\application\globalapp2\
    apt\system\editorial\application\client\
    apt\system\editorial\application\client2\
    apt\system\editorial\library\elementgrid.py

    .... etc. You get the idea, I hope. We have apt.library where we store all of
    our 'global', 'shared' libraries that everything can make use of, and
    apt.application with some global, shared apps. Then we have 'systems', which
    are a collection of applications and libraries in a single domain. These
    mostly get sold as a single product, really. Etc, etc.

    I end up doing a lot of:

    from apt.library.corba import *
    from apt.library.database import Connection
    from apt.library import machine
    from apt.system.editorial.library.elementgrid import ElementGrid

    ... And such. When PEP328 gets in, this may become more attractive, but
    occasionally I'm alarmed by the depth of the whole thing. But I also want to
    keep stuff logical and separated.

    And how about distribution? Freezing and such won't work, because of the
    shared nature of most of this code. Currently I'm probably planning on
    looking into finding the build scripts for python, win32all, and wxPython
    and seeing if I can combine them into one single install. Don't know if
    that's the best thing to do or not, but expecting my customers to install
    this stuff... is expecting too much of them.

    For example: Traditionally my company has avoided installing software on
    a users local machine. Instead we install to a file server and have everyone
    run the software from there. A bit of a performance hit but the advantage of
    administration is very important, because our customers do not necessarily
    have a very strong IT staff. There is some concern that if we do this same
    thing with Python, the performance hit of loading an interpreted environment
    over the shared filesystem will be prohibitive. Some basic testing has made
    this seem to be the case.

    hmm. That's all I have for now. I think. :)

    Thanks in advance.

    --Stephen
     
    Ixokai, May 16, 2004
    #1
    1. Advertisements

  2. Ixokai

    Peter Hansen Guest

    Just focusing on this one point: it doesn't make sense to me that
    this would be the case, and in my experience it is not. The
    interpreter (python23.dll for example on Win32) is less than 1MB
    and that's basically _tiny_ these days. We always had a single
    shared instance of Python and never noticed more than a small
    fraction of second delay compared to loading it locally. This
    was a network with only about fifteen developers as users, and
    100MBps throughout, but I would think almost any program these
    days is going to be 1MB or more, so I can't see that Python has
    any disadvantage in this area.

    -Peter
     
    Peter Hansen, May 16, 2004
    #2
    1. Advertisements

  3. Ixokai

    john fabiani Guest

    I'm a newbie so take what I say with a grain of salt!
    In the VFP (visual foxpro) world the runtime environment is loaded on
    the local computer (client) and the program is often loaded on to the
    server. I'm wondering if there would be any advanage to placing the
    python runtime on the clients along with the standard modules. Then
    placing the program code on to a server where all the clients/users
    would access the code. It would seem that the advanage would be that
    program load time would about the same but the excutions would be faster
    because of the local runtime. Can this even been done? Would it be faster?

    John
     
    john fabiani, May 16, 2004
    #3
  4. Ixokai

    Jorge Godoy Guest

    If I understood you correctly, this is what we do on our clients. It also
    makes it easier to make bugfixes, updates, etc. in our code.
     
    Jorge Godoy, May 16, 2004
    #4
  5. Ixokai

    john fabiani Guest

    So it can be done. I'll have to learn a little more about path-ing.
    Thanks
    John
     
    john fabiani, May 16, 2004
    #5
  6. Ixokai

    Ixokai Guest

    These are GUI programs and we've decided to use wxPython, so that may be
    adding to it. I will do more testing to see if it was just a fluke the last
    time I tried it out. It might have been :)

    --Stephen
     
    Ixokai, May 16, 2004
    #6
  7. One thing to investigate is file openings. If you combine all of
    python (and in your case wxPython's) .pyc files into a single zip
    wish an actual directory structure matching the packages you want to
    use, you will only have to open the zip file once, rather than pull
    each module in in several file operations.
     
    Scott David Daniels, May 17, 2004
    #7
  8. Ixokai

    David Fraser Guest

    Well you don't neccessarily have to have everything under apt... You
    just need to have unique package names.
    Also the cleaner your system design the less you should have to refer to
    bits inside other packages.
    On the wxPython users list we are planning something called wxPRE which
    would be an environment pre-setup with Python and wxPython. You may want
    to help set this up as it is very similar to your requirements.
    See http://wiki.wxpython.org/index.cgi/wxPRE

    David
     
    David Fraser, May 17, 2004
    #8
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.