Automating, Building, Testing and Deploying to Production Server

[yoda]

Hi Guys,
I've been used to deploying code to the production server by checking
out of subversion and manually sorting out any kinks. (yes, I know, it
sounds primitive)

I realize I'm losing so much time I could spend more productively. I'd
therefore like to know the different approaches you guys employ to
deploy builds from your staging servers (or laptops to the production
server in an automated repeatable safe manner.

How do you automate the process?
What tools do you use and how?
What documentation is available for the various tools?
What is the best, easiest, most automated, method that provides robust
versioning and easy rollback?

 

[Mike Meyer]

Hi Guys,
I've been used to deploying code to the production server by checking
out of subversion and manually sorting out any kinks. (yes, I know, it
sounds primitive)

Actually, it sounds like your test/development environment is
primitive. There shouldn't be any "kinks" to sort out by the time you
check things out on the production server.

How do you automate the process?

I do what you do - except I use perforce instead of svn, and automate
things. There's a development branch, a test branch, and a production
branch for each project. Developers can check things into the
development branch, and integrate from development to test. QA folks
can integrate from test to production. There are daemon processes
running on the test and production servers that do "p4 sync" ("svn up"
to you) once a minute, thus automatically installing new code on the
appropriate server. The QA folks "sort out the kinks" on the test
server, so that the production server doesn't suffer outages from the
development process.

What tools do you use and how?

Perforce and the python wrapper for same. And of course Python.

What documentation is available for the various tools?

Bundled with the tools, and on their web sites for perforce and the
wrapper. Lots for Python, all over the place.

What is the best, easiest, most automated, method that provides robust
versioning and easy rollback?

The perforce web site has some white papers on "best practices". I
read those - especially the one on web server software - then
implemented what I described above.

<mike

 

[Paul Rubin]

I realize I'm losing so much time I could spend more productively. I'd
therefore like to know the different approaches you guys employ to
deploy builds from your staging servers (or laptops to the production
server in an automated repeatable safe manner.

This is really a wide-ranging question and clpy maybe is not the best
place for it. Basically every installation does it differently
depending on its requirements. You could google some phrases like
"continuous integration" and "configuration management" to see some
approaches. The general outline is:

1) QA dept. has a server configured as identically as possible to the
production server.

2) Engineering releases a sw version by making a source control
branch. QA checks out the branch and tests it against pre-agreed
release criteria. If there's differences of opinion about criteria,
QA manager and engineering manager meet to resolve differences.

3) when QA manager and engineering manager agree to release, code is
pushed out to production server (same process as release to QA:
someone logs onto the production server and does an svn update or
whatever). How that's done depends on how the server works. I
worked at one serious web site which worked as follows. We had a
primary server and hot backup (secondary) server, so we did pushes
by taking the backup server offline, checking out the new build on
the backup server and running some sanity tests, then launching the
service on the backup server and taking down the primary, so there
would be an automatic failover to the backup which was now running
the new code. We'd then upgrade the primary the same way, and fail
over again if desired. Smaller sites with no hot secondary will
typically go briefly offline for version upgrade. Larger sites
with multiple primaries could have a more gradual process where
primaries are upgraded one by one, or whatever.

4) Some high availability services (e.g. telecom) need hot upgrade
(code replacement without stopping the application or interrupting
connections). This needs to be built into the app architecture at
a deep level and I don't think it's what you were asking, but some
languages like Erlang make provisions for it.

Generally, a release cycle for an online service isn't that much
different than that for a shrink-wrapped software package.

 

[Benji York]

I realize I'm losing so much time I could spend more productively. I'd
therefore like to know the different approaches you guys employ to
deploy builds from your staging servers (or laptops to the production
server in an automated repeatable safe manner.

How do you automate the process?

We have a system called "the buildout" that checks out a project's code,
downloads and builds any dependencies, and configures a working system.
(http://svn.zope.org/Sandbox/zc/buildout/trunk/)

We use that for setting up new development environments (each developer
generally has several builds of any project at one time).

What tools do you use and how?

We also have a BuildBot setup that runs a buildout and then runs the
tests and emails a list if the tests fail (http://buildbot.sf.net). You
can see an example of our "public" buildbot here: http://buildbot.zope.org.

What documentation is available for the various tools?

We just publicly released the buildout code a couple of weeks ago, and
it seriously needs docs and tests.

BuildBot doesn't have great docs, but it's configured via Python, so
it's not too bad.

What is the best, easiest, most automated, method that provides robust
versioning and easy rollback?

When we deploy via buildout (as opposed to by using an installer or a
simple archive) we generally either just tell the buildout to update
itself, or if we want the ability to "roll back" make a new buildout and
switch to it (therefore we can switch back to the previous build).

We don't do it this way, but because the buildout for a particular
project is itself versioned, you could just "svn up" to the previous
version and rebuild it and you'd be back to where you started.