How to automate OpenCms module import

In one of our projects, we have been using OpenCms for a long time as the system for editorial work regarding website content. Additionally, we continuously develop and extend a backend application that takes care of business logic like web service integration. Being an agile development team, we do of course want to have an agile testing environment as well. Thus, the first task in sprint 1 of the current project was to set up a continuous integration environment. The related user story was called “As a tester, I want to test on a daily basis in order to find defects as early as possible.” I’m now going to tell you how we realized the “daily basis” part through automated deployments.

In the “real” target environment, both OpenCms and the underlying database are clustered. So for example, as soon as you deploy a module, the OpenCms cluster logic kicks in and replicates resources to the cluster nodes. In order to reduce the technical complexity for the CI environment, we decided to use a single-node setup. At the same time, we keep in mind that cluster-related bugs can only be found after the modules/applications have been deployed to the clustered target environment.

In order to be able to test on a daily basis, and in order to test the latest versions of all components, a daily deployment of all Cms modules and the backend applications is inevitable. We have been using Hudson for several months now, which provides us with automated build and packaging jobs. Essentialy, we were only missing a link between Hudson builds and deployments on the test server.

Here’s what needs to be done:

  1. After a successful build, the build artifact must be uploaded to the test server.
  2. The uploaded artifacts must be deployed daily at a given time.
  3. After the deployment, an email with log messages has to be sent out.

The required artifact upload to the test server was pretty easy to implement: Hudson tests and builds the artifacts using Maven, so we could simply plug into the “deploy” phase in order to call an Ant SCP target. Upload done.

Now for the actual deployment – we have been using the OpenCms Shell features for years now in order to synchronize modules between Eclipse and OpenCms, or for example to import entire sites with test content. So in order to import a module to OpenCms, we use the Cms Shell commands “loginUser”, “deleteModule” and “importModule”. Here’s an example:

loginUser Admin admin
deleteModule de.codecentric.example
importModule /opt/somewhere/autodeploy/de.codecentric.example.zip
exit

You would of course put real login information, a real module name and a real path to a module ZIP in there.

Because we develop many modules in parallel, we also have to deploy many modules at once. We decided to write a Bash script that would generate a temporary Cms Shell script which can then be executed by the Cms Shell. Take a look at the commented script:

First, set a couple of variables to configure the script:

#!/bin/bash
HOME_DIR=/opt/somewhere
PACKAGE_DIR=$HOME_DIR/autodeploy
JBOSS_BIN_DIR=$HOME_DIR/jboss/bin
JBOSS_SERVER_DIR=$HOME_DIR/jboss/server/opencms
OPENCMS_BASE=$JBOSS_SERVER_DIR/deploy/cms.war/WEB-INF
 
LOG_FILE=$PACKAGE_DIR/log
LOG_MAILTO=receiver@example.net
LOG_MAILFROM=autodeploy@yourhost
 
CMS_USER=Admin
CMS_PASSWORD=admin
 
JAVA=/opt/jdk1.6.0_12/bin/java

Here’s a little convenience function that prints out which modules were found in the artifact directory:

function list {
    if [ 0 != $# ]; then
        echo "The following modules were found:"
        i=1
 
        for package in $@
        do
                echo "  $i. `basename "$package"`"
                ((i=$i+1))
        done
    else
	echo -e "No modules were found.\n"
    fi
}

What follows is essentialy “the heart” of the script: the “deploy” function. It takes a list of Cms module ZIP file paths as a parameter. Only if this parameter is not empty, a Cms Shell script will be generated. Everything following the “exit” statement has been copied from cmsshell.sh (see WEB-INF in the OpenCms WAR) and has of course been adapted to the specific situation on our test server. Take it at as an example ;) As soon as the classpath is filled with the required entries, the Cms Shell is started. It executes the script we have generated, which means that the modules will now be deployed. One last remark about this: we have added a “purgeJspRepository” to the Cms Shell script in order to make absolutely sure that no JSP fragments will remain in the Flex Cache.

function deploy {
    if [ 0 != $# ]; then
	cd $PACKAGE_DIR
 
	# Add login statement to the CMS script
	echo "loginUser $CMS_USER $CMS_PASSWORD" > script
 
	# Add import statements to the CMS script
	for zipfile in $@
	do
		echo "deleteModule \"`basename $zipfile .zip`\"" >> script
		echo "importModule \"$zipfile\"" >> script
	done
 
	# Purge JSP Repo
	echo "purgeJSPRepository" >> script
 
	# Add exit statement to the CMS script
	echo "exit" >> script
 
	for JAR in $JBOSS_SERVER_DIR/lib/*.jar; do
	    TOMCAT_CLASSPATH="$TOMCAT_CLASSPATH:$JAR"
	done
 
	OPENCMS_CLASSPATH=""
	for JAR in $JBOSS_SERVER_DIR/deploy/web.war/WEB-INF/lib/*.jar; do
	    OPENCMS_CLASSPATH="$OPENCMS_CLASSPATH:$JAR"
	done
 
	$JAVA -classpath "$OPENCMS_CLASSPATH:$TOMCAT_CLASSPATH:classes" org.opencms.main.CmsShell -base="$OPENCMS_BASE" -script=script >> $LOG_FILE
    fi
}

The rest of the bash script consists of a couple of helper functions: they start and stop (in our case) JBoss, send out the logfile mail, and clean up after the deployment:

function stopJBOSS {
    $JBOSS_BIN_DIR/shutdown.sh
    tail -f $JBOSS_SERVER_DIR/log/server.log > $PACKAGE_DIR/jboss_down.log &
    until tail -n 1 $JBOSS_SERVER_DIR/log/server.log | grep "Shutdown complete"; do sleep 0.01; done
    killall tail
}
 
function startJBOSS {
    rm $JBOSS_SERVER_DIR/tmp/* -r
    rm $JBOSS_SERVER_DIR/work/* -r
    $JBOSS_BIN_DIR/startup.sh
    tail -f $JBOSS_SERVER_DIR/log/server.log > $PACKAGE_DIR/jboss_up.log &
    until tail -n 1 $JBOSS_SERVER_DIR/log/server.log | grep "Started"; do sleep 0.01; done
    killall tail
}
 
function mailLog {
	mailsubject="Auto-Deploy-Log `date "+%d.%m.%Y, %H:%M"`" 
	cat $LOG_FILE $PACKAGE_DIR/jboss_down.log $PACKAGE_DIR/jboss_up.log | mail -s "$mailsubject" $LOG_MAILTO -a "From: $LOG_MAILFROM"
}
 
function cleanup {
	rm $PACKAGE_DIR/*
}

In here, please note that startup.sh and shutdown.sh are not usual parts of the JBoss distribution – they are selfmade wrapper scripts.

The only part missing from the bash script now is the couple of lines calling the functions:

echo -e "Looking for modules in: $PACKAGE_DIR"
 
packages=`find $PACKAGE_DIR -name *.zip`
 
# Do some work! :)
stopJBOSS
list $packages
deploy $packages
startJBOSS
mailLog
cleanup

As you can see, all ZIP files inside PACKAGE_DIR are regarded as OpenCms modules. The version of the script currently being used by us does some more work: WAR files are deployed, rulesets are updated, and soon we will also run SQL scripts along with each deployment. Feel free to change or extend the script at will!

This script enables us to precisely fulfill the story “As a tester, I want to test on a daily basis in order to find defects as early as possible.”. We are even able to run the script manually during the day in order to quickly deploy a bugfix and retest it. Of course (needless to say this) after every deployment we automatically run our Robot tests. All in all, we have created a very valuable and helpful continuous integration environment that saves us a lot of manual work and that optimally supports our job of developing high-quality software.

A quick note towards the end of this article: originally, our deployment script was only used with OpenCms 7.5.2 and JBoss 4.2.3. In the meantime, another codecentric team has successfully adapted the script and happily uses it with OpenCms on BEA WebLogic.

For the future, we are thinking about – at least partially – automating deployments on the first of three stages in the “real” target environment. We will see if OCEE (information about OCEE available here) and the OpenCms Shell like each other. If anyone has experience with this: you are very welcome to leave comments :-)

  • Facebook
  • Delicious
  • Digg
  • StumbleUpon
  • Reddit
  • Blogger
  • LinkedIn
Robert Spielmann

4 Responses to How to automate OpenCms module import

  1. James says:

    Thanks for this post … i am in the middle of creating a autodeploy script.

  2. ben says:

    Hi.

    > Wir werden sehen, ob OCEE (Infos zu OCEE hier) sich mit der Cms Shell verträgt.

    Konntet ihr das in der Zwischenzeit testen? Funktioniert das?

    Besten Dank für den Eintrag! Super Sache… google hilft also doch… ;)

    ben

  3. ben says:

    Ok.

    Wir schauen uns gerade auch ein paar Möglichkeiten an. Mal sehen.

    Besten Dank.

    ben

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>