In the last three blog posts we – Thomas Strecker and I – focused on writing tests and also covered how to run them. But running tests on your local development box is only the first step. The more important one is to run them on a CI server automatically. What has to be considered for this?
The CI Server and the Atlassian SDK
There are a lot of CI servers to choose from. According to two reports, the most common CI servers are Jenkins (+Hudson), Bamboo and TeamCity. While Jenkins and Hudson are open source tools the other two (Bamboo and TeamCity) are commercial products. These solutions are web applications, which run on your system. Differences exist in multiple areas, e.g. supported SCM and build systems. Another big diffentiator is of course the user interface. Fortunately all of them support Maven which is used in the JIRA ecosystem.
The version 4 of the Atlassian SDK uses Maven 2.1. This Maven version is outdated and certain plugins require a newer Maven version. Using a newer Maven version with this version of the Atlassian SDK is supported but complicated. Fortunately, the new version 5 of the Atlassian SDK (released June 2014) ships with Maven 3.2.1. This greatly eases the setup of your project in the CI server. Still, the configuration of the Maven is the main task.
There are multiple ways to create a working configuration.
- Create a config for a Maven executable and specify the path to the Maven installation embedded in the Atlassian SDK (e.g. …/atlassian-plugin-sdk-5.0.3/apache-maven-3.2.1). In addition you have to pass in the ATLAS_HOME environment variable (in the task config) and set it to the Atlassian SDK (…/atlassian-plugin-sdk-5.0.3/)
- Since Bamboo executes the mvn executable and this is a shell script, it is also possible to modify the shell script by adding a line “export ATLAS_HOME=…/atlassian-plugin-sdk-5.0.3/” and remove the setting of this environment variable in the first variant.
- It is also possible to use the existing atlas-mvn as the mvn executable. For this you have to create an empty directory (lets call it …/wrapper/) with a bin directory and inside the bin a symbolic link called mvn, which links to …/atlassian-plugin-sdk-5.0.3/bin/atlas-mvn. Inside Bamboo create a Maven executable with the path …/wrapper
Jenkins does not use the mvn executable to run the Maven build step. So only the first variant is possible. For this you have to specify the environment variable ATLAS_HOME in the global configuration with the value …/atlassian-plugin-sdk-5.0.3/. In addition you have to create a Maven installation with MAVEN_HOME set to …/atlassian-plugin-sdk-5.0.3/apache-maven-3.2.1/. Sadly it is not possible to use the EnvInject Plugin to move the global ATLAS_HOME config into an individual pre-build step. See https://issues.jenkins-ci.org/browse/JENKINS-17620 and https://issues.jenkins-ci.org/browse/JENKINS-18234.
TeamCity like Jenkins does not use the mvn executable either. You can specify the ATLAS_HOME environment variable either at the project level or for an individual build configuration. In the Maven build step you have to choose a “Custom” Maven selection and pass the path …/atlassian-plugin-sdk-5.0.3
A better solution?!
There are a few general ways to use the Maven from the Atlassian SDK in the CI server:
- You can set the ATLAS_HOME variable before starting the CI server process. This has the disadvantage that you have to take extra steps when you use another SDK version. You basically lose the ability to run your tests with different SDK versions.
- The ATLAS_HOME variable is only needed for referencing the …/atlassian-plugin-sdk-5.0.3/repository/ directory in the …/atlassian-plugin-sdk-5.0.3/apache-maven-3.2.1/conf/settings.xml. So one solution is to expand the path inside the settings.xml file and forget about the ATLAS_HOME environment variable.
In general, we recommend that last solution (expanding the path in the settings.xml file). Creating a copy of the SDK directory (62mb here) and sed-ing one file is a small effort, which reduces other problems down the road.
There are some other problems you need to be aware of.
Weak CI Server
On a less capable machine (like for example an AWS instance) it is possible that the startup of JIRA takes too long and a timeout condition is raised. Setting the atlassian.plugins.enable.wait to a high value helps here:
Note: setting the wait to 1000 seconds will produce warning messages in the logs, claiming the plugins have not been compiled to provide a 1000 seconds wait. This warning can be ignored since it only applies to the test cycle and not the production code.
Running WebUI Tests
WebUI tests are more complex than other test types. For one, they need to start and communicate with a browser. And on a Linux CI server such a graphical program requires some X11 display. On such headless cases the virtual framebuffer X server (Xvfb) is used usually. While there is an option in AMPS to let the WebUI test start the Xvfb we were not able to make this work. Starting the Xvfb once as a long running server (for example in a startup script of the operation system) is the alternative solution. Related to frontend tests you want to set the DISPLAY environment variable to something like “:1.0”.
What about Code Coverage?
For us Sonar is a central dashboard for the source code quality. So especially if you inherited some legacy code you want to know which classes are tested in what way. Code coverage can act as an indicator for this. You still have to take a look at the tests to see if they make sensible assertions, which reflect the functional requirements.
Bundled with the SDK is Clover which is a commercial code coverage tool from Atlassian. A good start for this is the atlas-clover command which you can run from the command line. It will instrument the code, run the tests and output a short summary in addition to the HTML files generated in target/site/clover. The atlas-clover command does execute under the hood a normal maven command with the two goals clover2:setup clover2:clover wrapping the test execution. Sonar also has such a two step execution so this is a good match. We ended up with two build goals which are executed after each other:
clean clover2:setup verify
With this setup it is possible to measure the coverage of both unit tests and integration tests. Using other coverage tools like JaCoCo and Cobertura we were unable to collect code coverage for integration tests as well.
Armed with these information there is no longer an excuse of not using a CI server to test the JIRA plugin you are developing. As you can see we like tests and think that tests are an important part of the software development activity. But we also see that writing tests for a JIRA plugin is sometimes more cumbersome than desired.
Currently we have no further open topics regarding testing of JIRA plugins. So this is the last post in this series.