Continuous Delivery in the Cloud – Part 5: Configure your Continuous Delivery Pipeline

This article builds upon the previous four articles and explains how to set up fully automated continuous delivery pipeline.

• Part 1: Continuous Delivery Overview
• Part 2: Getting started with Amazon EC2
• Part 3: Configure your Continuous Integration Server
• Part 4: Provisioning your Test, Staging and Production environments
• Part 5: Configure your Continuous Delivery Pipeline
• Part 6: Create a Dashboard of your Systems

Introduction

After we introduced the continuous integration server in Part 3 and learned how to provision development and production environments  in Part 4 , we are ready  to build a continuous delivery pipeline (CDP).

CDP aims to automate software build, deploy, test and release processes.  Every source code commit operation triggers a number of checks,  that give the development team an immediate feedback on the software quality. This part of the CDP functionality covers processes which are typically associated with a continuous integration process. Furthermore, the software is immediately and automatically deployed and tested in development and pre-production environments. Finally, the release of a selected software version is triggered by one mouse click which is safer and quicker compared to custom software releases, where the operations team needs to follow a word document to get the next release into production.

Continuous Delivery Pipeline structure

The structure of a CDP is derived from the project value stream from check-in to release.  Definition of a suitable value stream setup is therefore the most vital task for setting up a CDP structure. Usually one needs a couple of iterations to find an appropriate CDP structure which suits the given project. This process involves not only the development team, but other stackholders as well, such as system integrators, the operations team, and project managers.
Most continuous integration (CI) tools support modeling of continuous delivery processes.  In Jenkins you can define a chain of consecutive jobs using the Build Pipeline Plugin as described in Part 3 of the this article series.

In this demo we identified seven consecutive stages in the CDP, executed in the following order:  Commit Stage, Acceptance Environment Provisioning Stage, Acceptance Test Stage, Performance Test Stage, User Acceptance Test (UAT) Provisioning Stage, Green Production Environment Provisioning Stage, and Blue Production Environment Provisioning Stage.

Jenkins automatically starts the successor job after successful completion of the predecessor stage including the UAT Provisioning Stage.  The deployment to the production environments is triggered manually.

I continue with a detailed description of each stage.

Commit Stage

The first stage evaluates the latest commit operation to the version control system. It includes the following jobs:

1. Code compilation. If this job fails, a notification containing the compilation error is sent to the development team.
2. Execution of commit tests, which are the part of existing unit tests, designed to check basic application functionality on a low API level.
3. Versioning of the artifacts. After successful compilation and unit testting, the software version is counted up. The source code is tagged in the version control system and deployed into Artifactory, which is used for managing versioned and tested software binaries.
4. Static source code quality check using Sonar.  In this step, test coverage, cyclomatic complexity, amount of duplicated code, and the number of coding rule violations are verified (plugins used are findbugs, checkstyle, pmd)

An important requirement for the commit stage is to keep its execution time less than 10 min. This stage results in creation of a deployable release candidate with a version number.

Provisioning Acceptance Test Stage

Unit tests which are run during the commit stage verify only the developer’s view on the newly made application changes. On the contrary, the acceptance tests stage has a user-oriented perspective and verifies his requirements on the system functionality. The best practice is to deploy the software for acceptance testing into a close-to-production environment. Such approach helps to verify configuration parameters and deployment scripts (integration tests) in the course of the acceptance tests.

During this stage the environment for acceptence tests is prepared using Puppet as described in Part 4 .

Automated Acceptance Test Stage

The main purpose of this stage is to run automated acceptance tests, which ensure that the user acceptance criteria on the software are met. Furthermore, with a growing from release to release  acceptance test base, the application will be regression tested as well. Regression tests check that new changes to the software do not break the old functionality.

There are a lot of test frameworks which are suitable for automated acceptance tests. The most widely used are JBehave, Robot Framework, Canoo WebTests. We have integrated JBehave into the Jenkins Build pipeline as described in Part 3 .

If there are errors in the test execution flow, the build pipeline stops and the responsible developer team is notified. In this case the software cannot be released.

Performance Test Stage

After successful execution of automated acceptance tests, CDP delivers a ready-to-deploy software. For many applications, however, non-functional parameters play a vital role and must be tested additionally. That is the reason why we include the performance test stage in the delivery pipeline. This stage runs automated performance tests and checks whether the latest software change influences any application capacity thresholds.

To accomplish this goal we implemented performance tests using JMeter Test Framework. Using maven integration, these tests can be executed on any CI Server. We integrated JMeter tests using the Chronos-plugin, which gives us a graphical representation of the performance reports.

User Acceptance Test (UAT) Provisioning Stage

UAT provisioning stage prepares the environment, on which the test team can test those aspects of the application functionality, which cannot be checked by automated tests. This usually includes complex functional work-flows or operations with dynamic data usage, which cannot be mocked for testing purposes. The tests in this stage are done manually. It is always good to have a test environment setup that you can give to “real users” for sanity testing.   

Provision Production Blue/Green Stages

For deployment into production environments, we have created a web site where the user can choose the software version and the target environment for the new release. The software version and binaries are retrieved from Artifactory. Any application version, which was previously deployed into Artifactory, can be released by a single mouse click. This manual step in the continuous delivery pipeline can be omitted, and in such case the software  can be automatically released with every comited code change.

Summary

To sum up there are three major advantages of  continuous delivery pipeline introduction into the software development process. First, with the full automated delivery process, all software changes, which successfully pass through all pipeline stages, will be released immediately. Secondly, not only development teams but also all stakeholders (release team, testers, project managers etc.) have access to the artifacts they need without additional communication overhead. On the other hand  the software release process is visible to all stakeholders. Due to immediate feedback bottlenecks can be identified, optimized, and removed faster.