Using Jenkins Job DSL for Job Lifecycle Management

No Comments

This is a short story how the Jenkins Job DSL made our and the customers life a whole lot easier.

Any software developing company that takes pride in software craftsmanship has some sort of build automation tool in place. In most cases this is Jenkins CI. Figure 1 shows a typical lifecycle of a Jenkins Job. When a new software project is started, new build, test and deploy jobs are created. Then developers start coding and commit their code to a version control system (DVCS). This triggers the execution of the build job. After a while developers might decide to switch from Java 7 to Java 8, so they need to migrate their code and update the CI job to build the code with JDK 8 instead of JDK 7. After a couple years in production the project becomes obsolete and gets deleted or archived from the DVCS. Build, test and deploy jobs have to be deleted in order to keep the CI infrastructure clean.

Figure 1: Jenkins Job Lifecycle

Jenkins is a great automation tool. It is easy to get started and once you have your first jobs automated, you do not want to go back. After a while you will have several Jenkins instances and slaves running and your Continuous Integration (CI) infrastructure might look like the following diagram.

Figure 2: Typical Software Development Infrastructure & Tools

Figure 2: Typical Software Development Infrastructure & Tools

You are running at least one Jenkins master with a dozen or more slaves and between 100 and 1000 different jobs. Each job is either manually created, copied from a template, created using the REST API or client JAR. There are various ways and they all work just fine.

You typically have jobs for building, testing and releasing software. It does not matter which programming language you prefer. With Jenkins only the sky seems to be the limit. There are more than 1000 plugins for integrating Jenkins with typical developer tools.

And then things start to change. Here is just a short list of things that I came across over the last couple of years:

  • Java Version upgrade
    • Jobs need to use a different JDK for compiling the source code
    • Since all build artifacts (WAR/EAR files) are automatically deployed to application servers, these servers needed to be upgraded as well, to run with the correct Java version
    • Generated server and app configurations needed to be upgraded
  • Build tools get replaced
    • Ant Jobs need to be migrated to Maven Jobs
    • Deployment and Test scripts and Jobs need to be updated as well
  • Version Control Systems change
    • After a migration from CVS or Subversion to Git, all Jobs need to be updated to pull code from a new server
  • Application Server upgrade
    • Automated deployment scripts need to deploy to new servers
    • Job parameters change
    • Configuration templates need to be upgraded
  • Projects get refactored and renamed in Git
    • Update Job name and SCM settings
  • Projects want to use different branches for CI
    • The default Branch for the CI Job needs to be updated
  • Framework Versions change
    • Spring Framework v4 and Hibernate v5 require JDK6+
    • Hadoop requires JDK7+
    • Your company framework might require JDK7 for newer versions
  • New Projects
    • Create new Build and Release Jobs
  • Projects are deprecated
    • Cleanup Jobs of deprecated projects on all Jenkins Server

Now comes the fun part. Imagine you have to take care of 140 Java projects. About 1000 different builds run on your Continuous Integration (CI) infrastructure each day and all of these projects are migrating at different times from Ant to Maven, from Java 6 to Java 7, from Tomcat 6 to Tomcat 7 and so on.

On top of that, we have a full copy of the CI infrastructure for testing purposes (http://jenkins-test, http://sonar-test, http://artifactory-test, http://git-test, …). We use these test instances to try out new tool versions, test data migrations and to validate infrastructure changes. In order for the test environment to be useful, we need to keep those jobs up-to-date as well.


Your email address will not be published. Required fields are marked *