Overview

Generated Jenkins Jobs and automatic Branch Merging for Feature Branches

No Comments

In my current project we are using feature branches to keep the master-branch clean and stable. Development, peer code reviews and also the first pre-integration acceptance tests by the product owner team takes place in the feature branches.

Many projects are using a set of jenkins-jobs to execute builds, run tests and provide code-metrics. Most of these projects are only running these jobs for the master-branch.

Our goal was to reduce the integration risk at the end of a feature branch`s lifecycle. The same Jenkins-Jobs as used for the master-branch should be used for all feature branches. Developers for example should fix their test-problems and ui-test-failures as long as they are working in the branch. As a nice side-effect, the product owner team will have an overview with all branches and their current state. They will not start any acceptance test, until the feature-branch-jobs are green.

Bamboo for example has a feature called plan branches, but there is nothing comparable in Jenkins. If you search the community, a lot of solutions for Jenkins are based on job cloning and mostly depend on a set of jenkins-plugins. There is a nice blog post by zeroturnaround including a thesis which describes a possible solution, but we missed an easy branch-overview for example.

So we implemented another solution based on the Job-DSL-Plugin, which fits our needs:

  • Branch-Dashboard with a clear status for each branch
  • Automatic creation/deletion of jobs, no manual work
  • version-controlled job-definitions
  • reduced fix-time in master-branch after integration
  • reduced effort for master-to-feature-branch-merges (see tipps & tricks: automatic merges)

What we did

  1. Express your jobs in DSL-Scripts
    We always wanted to have our job configurations in our SCM. Changes should be comprehensible and version controlled. So we used the Job-DSL-Plugin and started to express our Job-Definitions in DSL-Scripts.

    // basic example-job, which checks out sources from mercurial,
    // runs a maven build and sends mails
    job {
        name('build.job')
        logRotator(-1,3)
        scm {
    	hg('http://mercurial.example.com/project123/','default') 
        }
        triggers { scm('* 7-20 * * 1-5') }
        steps {
    	maven {
    	    rootPOM('parent/pom.xml')
    	    goals('clean install -T 1C')
    	    property('skipITs','true')
    	    property('maven.test.failure.ignore','true')
    	}
        }
        publishers {
    	mailer('devs@example.com',true,false)
    	archiveJunit('**/target/surefire-reports/*.xml')
        }
    }
  2. Set up the seed job
    The seed-job will create the jobs based on your DSL-Scripts. You can use this Tutorial, which describes the creation of the seed-job. But instead of using a provided script, we used the “Look on Filesystem”-option with a regular expression to use our scripts from our SCM.
    Build-Step-Definition for the seed-job

    Build-Step-Definition for the seed-job


    The seed-job is triggered by SCM-changes in our master branch and additionally one time per day. That’s needed if nobody is working in the master branch for example, but a new branch is added.
  3. Get a new-line-separated file containing your branches
    To create jobs for all branches, we need to know, which branches exist. We are using a small Shell-Script to create a new-line-separated text-file containing the branch names. It runs as an additional build-step in the seed-job. We are using Mercurial, and Mercurial is running on the same server. So we can just change to the mercurial-repo and ask mercurial.

    WORKING_DIR=$PWD
    cd /opt/mercurial/hg-repo
    hg branches | column -t | awk '{printf "%s\n",$1}' | sort > ${WORKING_DIR}/branches.txt
    cd $WORKING_DIR

    Shell-Step in seed-job

    Shell-Step in seed-job


    All our scripts are also stored in our SCM, thats why the script has it’s own file. You could also write the commands directly in the shell step box.
  4. Set up your jobs to be generated for every branch
    Now we have a basic build job and a file containing all branches. Now we extend the build-job-DSL to run for every branch.

    // read the text-file containing the branch list
    def branches = []
    readFileFromWorkspace("seed-job","branches.txt").eachLine { branches << it }
     
    // for every branch...
    branches.each {
        def branchName = it
     
        job {
            // use the branchName-variable for the job-name and SCM-configuration
            name("branch.${branchName}.build.job")
            // ...
            scm {
    	    hg('http://mercurial.example.com/project123/',"${branchName}") 
            }
            // ...
        }
    }

    The job-console should look like this:

    Processing DSL script build.groovy
    Adding items:
        GeneratedJob{name='branch.featureFoo.build.job'}
        GeneratedJob{name='branch.featureBar.build.job'}

That’s it. Now a build-job for every branch is created automatically. The Jobs are also deleted automatically, after the branch is merged back to the master branch. There is no manually work needed.

Some tipps & tricks, if you like to implement something like this:

  • Use a repository cache
    We are using Mercurial and every job would create an own repository copy for each branch. But you can set up the Mercurial-Jenkins-Plugin to “Use Repository Caches” and “Use Repository Sharing”. So make sure to enable this in the global Jenkins settings.
  • Create views
    If you have many branches and/or create multiple jobs for each branch, it might be a good idea to create some views to sort your jobs. The Job-DSL-Plugin can also generate views.

    // example-view containing all jobs starting with "branch"
    view(type: ListView) {
        name 'Builds per Branch'
        jobs { regex("branch.*") }
        columns {
    	status()
    	weather()
    	name()
    	lastSuccess()
    	lastFailure()
    	lastDuration()
    	buildButton()
        }
    }
  • Maybe use choice parameters
    We have a job, which is used by our product owner team to deploy the branch they want to test on one of the test servers. We don’t generate a deploy job for each branch, instead we are using a job parameter for the branch name. We can easily reuse our text-file containing the branch-names to provide a Select-Box.

    def branches = []
    readFileFromWorkspace("seed-job","branches.txt").eachLine { branches << it }
     
    job {
        name('deploy.test')
        parameters {
            // create a parameter "BRANCH", containing all entries from "branches"
            // default-value is the first entry from "branches"
            choiceParam('BRANCH',branches,'Which branch do you want to deploy?')
        }
        scm {
            hg('http://mercurial.example.com/project123/',"$BRANCH") 
        }
        //...
    }
  • Think about automatic merges
    After we implemented the first jobs, we thought about automatic merges from our master branch to the feature branches. In our project, the Interfaces to some needed subsystem are changed quite often and the project-code to access this subsystems is always fixed in the master branch. So the product owner team was always frustrated, when they decided to start a test of a feature branch which was currently not working against the changed interfaces. We spent a lot of time with no-brainer-merges into feature-branches. Based on the techniques described above, we started to implement an automatic merge. Now every change in our master branch will get merged into all feature-branches if there is no merge-conflict. The merge is done by a small shell script. The script is called by a job, which is generated by the Job-DSL-Plugin for every branch. If there is a merge-conflict, a developer has to do the merge manually. The goal of every project should be, that there are not so many open feature branches. But if this goal is not reachable for some reasons, you might think about automatic merges.

    # Job-DSL Automerge
     
    def branches = []
    // we use another file here, to filter some branches which should not get automerged
    readFileFromWorkspace("seed-job","branchesAutomerge.txt").eachLine { branches << it }
     
    branches.each {
        def branchName = it
     
        job {
    	name("branch.${branchName}.automerge")
    	triggers { cron('H 5 * * 1-5') }
    	wrappers {
    	    environmentVariables {
    		env('BRANCH', "${branchName}")
    	    }
     
    	    // we are using a single repository for the automerge-jobs. So we have to be sure, that only one job is using the repository
    	    exclusionResources('AUTOMERGE_REPO')
    	}
    	steps {
    	    criticalBlock {
    		shell(readFileFromWorkspace('parent/jenkinsJobs/scripts/automerge.sh'))
    	    }
    	}
        }
    }
    # automerge.sh
     
    # Jenkins uses "-e" parameter, but we want to handle the exit-code manually
    set +e
     
    WORKING_DIR=$PWD
    cd /var/lib/jenkins/repoAutomerge
     
    # reset local changes
    hg update -C .
    # get changes from repository
    hg pull
    # update to branch
    hg update -C ${BRANCH}
     
    # try the merge
    hg merge develop --config "ui.merge=internal:merge"
    mergereturn=$?
     
    case $mergereturn in
    	0) 
    		echo '##################################'
    		echo '#####   Merge successfully   #####'
    		echo '##################################'
     
    		# commit and push
    		hg commit -m 'Automerge' -u 'AutoMerger'
    		hg push
     
    		rc=0
    		;;
    	1) 
    		echo '####################################################'
    		echo '#####   Merge-Conflict, manual merge needed!   #####'
    		echo '####################################################'
    		rc=1
    		;;
    	255) 
    		echo '############################################'
    		echo '#####   No Changes (Return-Code 255)   #####'
    		echo '############################################'
    		rc=0
    		;;
    	*) 
    		echo '###############################################'
    		echo "#####   Merge-Returncode : $mergereturn   #####"
    		echo '###############################################'
    		rc=1
    		;;
    esac
     
    # reset local changes
    hg update -C .
     
    exit $rc

Comment

Your email address will not be published. Required fields are marked *