Generated Jenkins Jobs and automatic Branch Merging for Feature Branches

In my current project we are using feature branches to keep the master-branch clean and stable. Development, peer code reviews and also the first pre-integration acceptance tests by the product owner team takes place in the feature branches.

Many projects are using a set of jenkins-jobs to execute builds, run tests and provide code-metrics. Most of these projects are only running these jobs for the master-branch.

Our goal was to reduce the integration risk at the end of a feature branch`s lifecycle. The same Jenkins-Jobs as used for the master-branch should be used for all feature branches. Developers for example should fix their test-problems and ui-test-failures as long as they are working in the branch. As a nice side-effect, the product owner team will have an overview with all branches and their current state. They will not start any acceptance test, until the feature-branch-jobs are green.

Bamboo for example has a feature called plan branches, but there is nothing comparable in Jenkins. If you search the community, a lot of solutions for Jenkins are based on job cloning and mostly depend on a set of jenkins-plugins. There is a nice blog post by zeroturnaround including a thesis which describes a possible solution, but we missed an easy branch-overview for example.

So we implemented another solution based on the Job-DSL-Plugin , which fits our needs:

• Branch-Dashboard with a clear status for each branch
• Automatic creation/deletion of jobs, no manual work
• version-controlled job-definitions
• reduced fix-time in master-branch after integration
• reduced effort for master-to-feature-branch-merges (see tipps & tricks: automatic merges)

What we did

1. Express your jobs in DSL-Scripts
   We always wanted to have our job configurations in our SCM. Changes should be comprehensible and version controlled. So we used the Job-DSL-Plugin and started to express our Job-Definitions in DSL-Scripts.

   1// basic example-job, which checks out sources from mercurial,
   2// runs a maven build and sends mails
   3job {
   4    name('build.job')
   5    logRotator(-1,3)
   6    scm {
   7    hg('http://mercurial.example.com/project123/','default') 
   8    }
   9    triggers { scm('* 7-20 * * 1-5') }
   10    steps {
   11    maven {
   12        rootPOM('parent/pom.xml')
   13        goals('clean install -T 1C')
   14        property('skipITs','true')
   15        property('maven.test.failure.ignore','true')
   16    }
   17    }
   18    publishers {
   19    mailer('devs@example.com',true,false)
   20    archiveJunit('**/target/surefire-reports/*.xml')
   21    }
   22}

2. Set up the seed job
   The seed-job will create the jobs based on your DSL-Scripts. You can use this Tutorial , which describes the creation of the seed-job. But instead of using a provided script, we used the “Look on Filesystem”-option with a regular expression to use our scripts from our SCM.
   
   

   Build-Step-Definition for the seed-job

   
   The seed-job is triggered by SCM-changes in our master branch and additionally one time per day. That’s needed if nobody is working in the master branch for example, but a new branch is added.
3. Get a new-line-separated file containing your branches
   To create jobs for all branches, we need to know, which branches exist. We are using a small Shell-Script to create a new-line-separated text-file containing the branch names. It runs as an additional build-step in the seed-job. We are using Mercurial, and Mercurial is running on the same server. So we can just change to the mercurial-repo and ask mercurial.

   1WORKING_DIR=$PWD
   2cd /opt/mercurial/hg-repo
   3hg branches | column -t | awk '{printf "%s\n",$1}' | sort > ${WORKING_DIR}/branches.txt
   4cd $WORKING_DIR

   

   Shell-Step in seed-job

   
   All our scripts are also stored in our SCM, thats why the script has it’s own file. You could also write the commands directly in the shell step box.

4. Set up your jobs to be generated for every branch
   Now we have a basic build job and a file containing all branches. Now we extend the build-job-DSL to run for every branch.

   1// read the text-file containing the branch list
   2def branches = []
   3readFileFromWorkspace("seed-job","branches.txt").eachLine { branches << it }
   4 
   5// for every branch...
   6branches.each {
   7    def branchName = it
   8 
   9    job {
   10        // use the branchName-variable for the job-name and SCM-configuration
   11        name("branch.${branchName}.build.job")
   12        // ...
   13        scm {
   14        hg('http://mercurial.example.com/project123/',"${branchName}") 
   15        }
   16        // ...
   17    }
   18}

   The job-console should look like this:

   1Processing DSL script build.groovy
   2Adding items:
   3    GeneratedJob{name='branch.featureFoo.build.job'}
   4    GeneratedJob{name='branch.featureBar.build.job'}

That’s it. Now a build-job for every branch is created automatically. The Jobs are also deleted automatically, after the branch is merged back to the master branch. There is no manually work needed.

Some tipps & tricks, if you like to implement something like this:

• Use a repository cache
  We are using Mercurial and every job would create an own repository copy for each branch. But you can set up the Mercurial-Jenkins-Plugin to “Use Repository Caches” and “Use Repository Sharing”. So make sure to enable this in the global Jenkins settings.
• Create views
  If you have many branches and/or create multiple jobs for each branch, it might be a good idea to create some views to sort your jobs. The Job-DSL-Plugin can also generate views.

  1// example-view containing all jobs starting with "branch"
  2view(type: ListView) {
  3    name 'Builds per Branch'
  4    jobs { regex("branch.*") }
  5    columns {
  6    status()
  7    weather()
  8    name()
  9    lastSuccess()
  10    lastFailure()
  11    lastDuration()
  12    buildButton()
  13    }
  14}

• Maybe use choice parameters
  We have a job, which is used by our product owner team to deploy the branch they want to test on one of the test servers. We don’t generate a deploy job for each branch, instead we are using a job parameter for the branch name. We can easily reuse our text-file containing the branch-names to provide a Select-Box.

  1def branches = []
  2readFileFromWorkspace("seed-job","branches.txt").eachLine { branches << it }
  3 
  4job {
  5    name('deploy.test')
  6    parameters {
  7        // create a parameter "BRANCH", containing all entries from "branches"
  8        // default-value is the first entry from "branches"
  9        choiceParam('BRANCH',branches,'Which branch do you want to deploy?')
  10    }
  11    scm {
  12        hg('http://mercurial.example.com/project123/',"$BRANCH") 
  13    }
  14    //...
  15}

• Think about automatic merges
  After we implemented the first jobs, we thought about automatic merges from our master branch to the feature branches. In our project, the Interfaces to some needed subsystem are changed quite often and the project-code to access this subsystems is always fixed in the master branch. So the product owner team was always frustrated, when they decided to start a test of a feature branch which was currently not working against the changed interfaces. We spent a lot of time with no-brainer-merges into feature-branches. Based on the techniques described above, we started to implement an automatic merge. Now every change in our master branch will get merged into all feature-branches if there is no merge-conflict. The merge is done by a small shell script. The script is called by a job, which is generated by the Job-DSL-Plugin for every branch. If there is a merge-conflict, a developer has to do the merge manually. The goal of every project should be, that there are not so many open feature branches. But if this goal is not reachable for some reasons, you might think about automatic merges.

  1# Job-DSL Automerge
  2 
  3def branches = []
  4// we use another file here, to filter some branches which should not get automerged
  5readFileFromWorkspace("seed-job","branchesAutomerge.txt").eachLine { branches << it }
  6 
  7branches.each {
  8    def branchName = it
  9 
  10    job {
  11    name("branch.${branchName}.automerge")
  12    triggers { cron('H 5 * * 1-5') }
  13    wrappers {
  14        environmentVariables {
  15        env('BRANCH', "${branchName}")
  16        }
  17 
  18        // we are using a single repository for the automerge-jobs. So we have to be sure, that only one job is using the repository
  19        exclusionResources('AUTOMERGE_REPO')
  20    }
  21    steps {
  22        criticalBlock {
  23        shell(readFileFromWorkspace('parent/jenkinsJobs/scripts/automerge.sh'))
  24        }
  25    }
  26    }
  27}

  1# automerge.sh
  2 
  3# Jenkins uses "-e" parameter, but we want to handle the exit-code manually
  4set +e
  5 
  6WORKING_DIR=$PWD
  7cd /var/lib/jenkins/repoAutomerge
  8 
  9# reset local changes
  10hg update -C .
  11# get changes from repository
  12hg pull
  13# update to branch
  14hg update -C ${BRANCH}
  15 
  16# try the merge
  17hg merge develop --config "ui.merge=internal:merge"
  18mergereturn=$?
  19 
  20case $mergereturn in
  21    0) 
  22        echo '##################################'
  23        echo '#####   Merge successfully   #####'
  24        echo '##################################'
  25 
  26        # commit and push
  27        hg commit -m 'Automerge' -u 'AutoMerger'
  28        hg push
  29 
  30        rc=0
  31        ;;
  32    1) 
  33        echo '####################################################'
  34        echo '#####   Merge-Conflict, manual merge needed!   #####'
  35        echo '####################################################'
  36        rc=1
  37        ;;
  38    255) 
  39        echo '############################################'
  40        echo '#####   No Changes (Return-Code 255)   #####'
  41        echo '############################################'
  42        rc=0
  43        ;;
  44    *) 
  45        echo '###############################################'
  46        echo "#####   Merge-Returncode : $mergereturn   #####"
  47        echo '###############################################'
  48        rc=1
  49        ;;
  50esac
  51 
  52# reset local changes
  53hg update -C .
  54 
  55exit $rc