code packaging

Discussion in 'Python' started by Paul Rubin, Jul 20, 2007.

  1. Paul Rubin

    Paul Rubin Guest

    I've been through this kind of thing a few times in the past and
    received excellent advice here on clpy about how to deal with specific
    technical aspects (e.g. how to use setuptools, InnoSetup, etc).

    I'm now wondering where this type of thing is best addressed in more
    general terms. What I usually see happening, in projects I've worked
    on and in others, is that developers get the code working on their own
    computers, using source control tools and (if we're lucky) tests
    developed in parallel with the code, but without much attention to
    final packaging until the very end when the code is about to be
    shipped.

    Is this appropriate? Inappropriate? Do your projects start using
    serious packaging and distribution tools very early in development,
    before the code is anywhere near usable? Should they?

    I'm involved in something that I wish was that way, mainly because I
    really don't have any way to put a complete installation on my own
    computers; I have to develop my subsystem more or less independently
    of the overall app, then toss stuff over the wall to the more central
    application guys for integration, which gets slow sometimes because
    we're distributed all over. This is a fairly complex server app that
    depends on a lot of external packages and programs running across
    multiple machines, so I could see the packaging problem being messy.
    It would cause a significant interruption in development to try to
    package things so that the non-central developers could install and
    hack it without a lot of assistance. But I think it would be
    worthwhile given how much easier it would become to test changes. We
    do plan to bundle up and release the code sometime (maybe as a Debian
    package) but for now that's deferred until the code is more complete
    and stable.

    I'm wondering how other projects go about this.
     
    Paul Rubin, Jul 20, 2007
    #1
    1. Advertising

  2. Paul Rubin

    Miki Guest

    Hello Paul,

    > I'm now wondering where this type of thing is best addressed in more
    > general terms. What I usually see happening, in projects I've worked
    > on and in others, is that developers get the code working on their own
    > computers, using source control tools and (if we're lucky) tests
    > developed in parallel with the code, but without much attention to
    > final packaging until the very end when the code is about to be
    > shipped.

    [snipped]

    IMO you should invest the time a build automated packaging and some
    smoke tests for the product.
    After this is done, start using some "continuous automation" tools
    (like http://cruisecontrol.sourceforge.net/, http://buildbot.net/trac
    and others).

    HTH,
    --
    Miki <>
    http://pythonwise.blogspot.com
     
    Miki, Jul 20, 2007
    #2
    1. Advertising

  3. Paul Rubin

    Ben Finney Guest

    Paul Rubin <http://> writes:

    > Is this appropriate? Inappropriate? Do your projects start using
    > serious packaging and distribution tools very early in development,
    > before the code is anywhere near usable? Should they?


    Yes. Test-driven development has taught me that putting off important
    parts of the development process until the end is rarely a good
    idea. So, I try to make a point of writing the first features inside a
    complete unit test environment, and inside a complete
    distribution/build environment. The tests are run continuously while
    developing the code, and the distribution/build process is tested
    manually.

    I'm looking into the "build-bot" approach used to ensure that the
    build process is also run automatically and continually during
    development, just like the unit tests <URL:http://buildbot.net/>; but
    so far I'm running the build manually.

    --
    \ "Read not to contradict and confute, nor to believe and take |
    `\ for granted ... but to weigh and consider." -- Francis Bacon |
    _o__) |
    Ben Finney
     
    Ben Finney, Jul 21, 2007
    #3
  4. > On Behalf Of Paul Rubin
    > I'm wondering how other projects go about this.


    I develop an automated build system from the very beginning. Running the
    build script:
    * Creates the API documentation (epydoc)
    * Creates the help files (extracting some information from the source)
    * Builds a windows executable (py2exe)
    * Builds an installer (Inno Setup)
    * Runs the installer (AutoIt)
    * Runs a smoke test on the installed application (AutoIt)

    The program must at least install and pass the smoke tests at any given
    point in development.

    Another script uploads the installer to my server, and sends an email to the
    testers stating that a new version is ready, with a list of
    additions/improvements.

    Regards,
    Ryan Ginstrom
     
    Ryan Ginstrom, Jul 21, 2007
    #4
  5. Paul Rubin

    John J. Lee Guest

    Paul Rubin <http://> writes:
    [...]
    > I'm now wondering where this type of thing is best addressed in more
    > general terms. What I usually see happening, in projects I've worked
    > on and in others, is that developers get the code working on their own
    > computers, using source control tools and (if we're lucky) tests
    > developed in parallel with the code, but without much attention to
    > final packaging until the very end when the code is about to be
    > shipped.
    >
    > Is this appropriate? Inappropriate? Do your projects start using
    > serious packaging and distribution tools very early in development,
    > before the code is anywhere near usable? Should they?
    >
    > I'm involved in something that I wish was that way, mainly because I
    > really don't have any way to put a complete installation on my own
    > computers; I have to develop my subsystem more or less independently
    > of the overall app, then toss stuff over the wall to the more central
    > application guys for integration, which gets slow sometimes because
    > we're distributed all over. This is a fairly complex server app that
    > depends on a lot of external packages and programs running across
    > multiple machines, so I could see the packaging problem being messy.
    > It would cause a significant interruption in development to try to
    > package things so that the non-central developers could install and
    > hack it without a lot of assistance. But I think it would be
    > worthwhile given how much easier it would become to test changes. We
    > do plan to bundle up and release the code sometime (maybe as a Debian
    > package) but for now that's deferred until the code is more complete
    > and stable.
    >
    > I'm wondering how other projects go about this.


    You can go further: in addition to developing a Debian package (or a
    few of them) for your own project, build your OS image using
    debootstrap (you'll probably want your own local repository(ies), for
    reproducibility). Then you do your development in a virtual machine
    or chroot environment. That way you develop in the same (or very
    similar) environment to the one in which your code will eventually be
    deployed (one difference being the use of an version control checkout
    rather a .deb install for the code you're actually working on, unless
    troubleshooting problems that are somehow tied up with that particular
    aspect of deployment). I've seen this working in a real-world
    project.

    Of course, you can set up a server that does a complete build-and-test
    every night, and a separate continuous integration server that just
    does an "svn up" (rather than a full rebuild) every time somebody
    checks in, so that test failures can be detected quickly and the
    guilty party suitably humiliated :)


    John
     
    John J. Lee, Jul 21, 2007
    #5
  6. Paul Rubin

    Alex Popescu Guest

    "Ryan Ginstrom" <> wrote in
    news:00df01c7cb38$49f83980$030ba8c0@MOUSE:

    >> On Behalf Of Paul Rubin
    >> I'm wondering how other projects go about this.

    >
    > I develop an automated build system from the very beginning. Running
    > the build script:
    > * Creates the API documentation (epydoc)
    > * Creates the help files (extracting some information from the source)
    > * Builds a windows executable (py2exe)
    > * Builds an installer (Inno Setup)
    > * Runs the installer (AutoIt)
    > * Runs a smoke test on the installed application (AutoIt)
    >
    > The program must at least install and pass the smoke tests at any
    > given point in development.
    >
    > Another script uploads the installer to my server, and sends an email
    > to the testers stating that a new version is ready, with a list of
    > additions/improvements.
    >
    > Regards,
    > Ryan Ginstrom
    >


    What "build" tools are available in Python? Any initial hints are highly
    appreciated.

    bests,
    ../alex
    --
    ..w( the_mindstorm )p.
     
    Alex Popescu, Jul 21, 2007
    #6
  7. Paul Rubin

    Alex Popescu Guest

    On 7/22/07, Ryan Ginstrom <> wrote:
    > Hi Alex:
    >
    > Do you develop for Windows? Are you looking to automate a build
    > process?
    >
    > The standard library's build module is distutils:
    > http://docs.python.org/lib/module-distutils.html
    >
    > As I mentioned in my post, I use a variety of third-party modules
    > (epydoc, py2exe, Inno Setup, AutoIt), which I glue together with my
    > own brew of python scripts. Since I develop mostly for Windows (except
    > for Web stuff, which I use Linux for), my build process is tailored to
    > that platform.
    >
    > Regards,
    > Ryan Ginstrom


    Thanks for following up on this. I am mostly used with a world where
    platform dependent builds are not very used (Java).

    My current attempt to learn Python is by creating a Python equivalent of
    the advanced Java testing framework TestNG (http://testng.org) -- but I
    will give more details about this when I'll be starting to have
    something more solid :).

    However, due to my background where builds are required and there are
    some de facto build standards (Ant, Maven, etc.) I am starting to think
    I will be needing something similar while working on the tool I've
    mentioned. At this point I think that what I am interested in are how
    distros are build in Python world (no platform specific distros, but
    rather generic ones) and ways to automate the process of creating the
    distros (and probably running the framework internal tests etc.)

    My research lead me to distutils and/or setuptools, respectively SCon
    and/or buildutils. So I am wondering if I am looking in the right
    direction or do I need to do some more research :).

    tia,
    ../alex
    --
    ..w( the_mindstorm )p.
     
    Alex Popescu, Jul 22, 2007
    #7
  8. Paul Rubin

    Paul Rubin Guest

    (John J. Lee) writes:
    > You can go further: in addition to developing a Debian package (or a
    > few of them) for your own project, build your OS image using
    > debootstrap (you'll probably want your own local repository(ies), for
    > reproducibility). Then you do your development in a virtual machine
    > or chroot environment. That way you develop in the same (or very
    > similar) environment to the one in which your code will eventually be
    > deployed (one difference being the use of an version control checkout
    > rather a .deb install for the code you're actually working on, unless
    > troubleshooting problems that are somehow tied up with that particular
    > aspect of deployment). I've seen this working in a real-world
    > project.


    Thanks for these suggestions. Buildbot (per Ben Finney) looks
    interesting as does the above, and Ryan's notes about experience with
    Windows installers is also appreciated. Whether lot of build
    automation would have been worth it at the beginning of the thing I'm
    working on or not, we probably need to move towards it now that more
    of the system is running.
     
    Paul Rubin, Jul 24, 2007
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. VJ
    Replies:
    1
    Views:
    568
  2. mark jason
    Replies:
    2
    Views:
    363
    Arne Vajhøj
    Dec 10, 2010
  3. mark jason

    packaging python code in zip file

    mark jason, Dec 9, 2010, in forum: Python
    Replies:
    1
    Views:
    441
    Peter Otten
    Dec 9, 2010
  4. R.. Kumar

    Packaging common code with gems

    R.. Kumar, Jun 6, 2010, in forum: Ruby
    Replies:
    2
    Views:
    85
    R.. Kumar
    Jun 6, 2010
  5. Mohan
    Replies:
    3
    Views:
    857
    Arved Sandstrom
    Feb 10, 2012
Loading...

Share This Page