Pattern - Continuous Deployment

Continuous Deployment is the notion of having the automated build infrastructure, the Continuous Integration server, go one step further and roll out deployments to the running server.

While automatically rolling out updates to a live server may seem reckless, if your site is providing services for other web sites, such as SOAP or REST Web Services, you do need a way of getting regular updates of that site out to callers. Furthermore, if your web site is undergoing rapid development, you need to push out changes and patches regularly. There are a limited number of ways to do this.

1. Pattern - Manual Deployment, possibly triggering automated tools
2. Scheduled Deployment, with a scheduled task pushing out updates
3. Continuous Integration-triggered deployment, possibly also on a schedule.

The appeal of the latter is that the CI tool can be configured to only push out the deployment if all tests are passing. If a test fails, you need to decide whether or not to disable that test, and explicitly do so (with a defect entered, the build /test configuration files modified to skip the test), before you can release. If you release by hand, you may eliminate the requirement to test that specific build in the staging site (accidentally or due to time pressure), and hence roll out a defective build.

What if the tests pass and yet the live site breaks? Time to write better tests.

Features

• The CI tool pushes out the complete deployment of the staging site, tests it, then, if successful, triggers an update to the live site.
• Fully automated deployment, testing.

Advantages

• Ensures the test are always successful before the deployment goes live.
• Eliminates manual release process.
• Enables frequent releases to be made, such as daily or better.
• With daily releases, you can have remote callers/user test against the latest changes; no bug reports related to old products and defects that have already been fixed.
• Reduces pressure to get last-minute fixes in, as there will be another release the following day.
• Great for performance testing/optimisation -the effect of a change can be measured on the live site within hours of the change being made.

Disadvantages

• Requires a fully automated deployment process
• Requires a fully automated release process, including generation of release notes from the issues tracking database, emailing of announcements.
• If you do a mid-day update, people may notice the loss of service.
• If you do a mid-night update, whoever comes in first in the morning gets to field any support calls that the update triggered. If the development team is a few hours behind the users (e.g. a West Coast team), then users in the East Coast will have been sending emails for some time before anyone comes in to field them, and they may be very upset.
• Some risk of rolling out broken code to customers. You need very good test coverage.
• Risk of rolling out performance problems to customers, unless the staging tests include performance tests.
• Unless developers never break the build, you need a release branch in SCM separate from the development branch, and some means of promoting patches from the development to the release branch.
• Can set up unrealistic expectations.

Because deployment/releasing is fully automated, the cost of rolling out an update is zero. As such, you can start releasing early and often, and so have a very evolutionary development cycle.

Being able to roll out changes to the live site instantaneously can set up unrealistic expectations in management and customers as to how long a defect takes to get fixed. Yes, a spelling mistake can be fixed and the new site updated in 15 minutes, but that does not mean that the transition from IPv4 network addressing to IPv6 addresses can be done in a similar amount of time. Even if you warn them of this, they will still be on the telephone every half hour for a status update. This is where the CI server can help. Once you make it clear that an update will take place overnight, if all tests pass, then they may back off and give you time to write the new tests (optimism!) and fix the code itself. Reducing the rush for pushing out the first fix you can think of in 15 minutes means that good fixes get made, rather than quick and dirty patches.

SmartFrog support

SmartFrog enables automated deployment and testing, so can be used as part of a Continuous Deployment process. We have set up such processes internally, and found it very effective.

SmartFrog itself is released on a two-week cycle. Why?

1. We are a downloaded product, not a web site. We need to give people who use our application time to download and use a release before cutting out a new one. Otherwise we end up fielding bug reports on older versions and have a harder time replicating them.
2. You need to have time to do interesting things, and a two week cycle seems to work well for us there. More often and you end up not being innovative, or breaking the schedule. Yes, there is branching, but long-lived branches are a trouble of their own.
3. We like to complete tangible work every cycle. It makes the release notes more interesting.