Overview

Automated Acceptance-Testing using Concordion

1 Comment

There are plenty of open-source tools available that are supporting the automation of acceptance tests. This also results in a lot of different concepts to do this. For example the Robot Framework is using a keyword-driven approach and offers a lot of tooling and ready-made test libraries. FitNesse on the other hand is organising the tests in an own Wiki and offers the possibility to start tests from this Wiki right away. Other tools like JBehave are going into a different direction and focus very much on a lean approach and are using BDD-format very consequently for describing the tests. The test implementation is then done entirely in Java. We have taken a closer look to this approach already here.

When discussing agile testing tools Concordion is almost always mentioned sooner or later. And often it is mentioned in a very positive manner. On the other hand I have not yet found anyone using it productively in any project. Thus I thought it is time to have a closer look to know what people are talking about :-).

The test implementation for Concordion happens typically in Java. But functionality has already been ported to .NET, Ruby, und Python. To make my life a little bit easier we will take a look at the Java version in the following.

As this article is quite long here comes a short overview on the topics presented in the following:

  • Download and “Hello World”-Example: Explains the installation and how to execute the sample project provided with the download.
  • Test description and test implementation: After executing the example let’s try to really understand Concordion and implement some tests of our own.
  • Reporting: How well are the testing results presented by Concordion?
  • Integration into a CI environment: How to integrate tests written with Concordion into some environment for Continuous Integration.
  • Conclusion: How suitable is Concordion for automating Acceptance Tests? And for whom?

 

Download and “Hello World”-Example

Obviously the first step is to download Concordion :-). this can be either done from the download page or from the Getting Started page. Concordion comes along as a ZIP-file that contains an Eclipse-Project. All required libraries are included as well as some small sample project to demonstrate the basic concepts. This article is based on concordion-kickstart-1.4.2.zip.

To import the project in Eclipse I create a new workspace into which I then import the Concordion directory from the ZIP-file as a new project. Afterwards you have the following result in Eclipse:

On first sight I notice the Ant and Gradle build scripts. I am for sure not a big Maven-fan, but somehow it is a standard, isn’t it? But let’s put that aside for the time being, as the first test can be simply executed from Eclipse as a Junit test. To do this just excecute the “HelloWorldTest.java” located under “com.example.specs.greeting” in the “specs”-directory as a JUnit test. That works without any problems and it is printed to the console where to find the report with the testing results.

C:\DOKUME~1\tja\LOKALE~1\Temp\concordion\com\example\specs\greeting\HelloWorld.html
Successes: 1, Failures: 0

The generated report that looks as follows:

This looks very good so far. But how about implementing some tests of our own?

 

Test description and test implementation

All agile testing tools are seperating the test description from the test implementation. Concordion is using XHTML as the input format for the test description. In writing this test description the user has a lot of freedom and is not bound to any one format like tabular or BDD-style. So let’s try to write a test for the Java Stack implementation. This way we gain very good comparability with our article on JBehave, where the same example was used.

The syntax for describing the tests is easy to learn and the following test description is quickly created. There are some commands that are embedded into the HTML-code to set variables, execute methods and assert expected results. In Concordion this is called instrumentation. As the class we want to test here is a Java API class and thus already existing we only have to create a new package “com.example.specs.stack” below “specs”. Inside this package we now create our test description called “Stack.html” as well as the corresponding test implementation in the class “StackTest.java”. Please notice the coupling of these two files based on some naming conventions. The HTML file must have the same name as the Java class, but without “Test” and of course with “.html” instead of “.java”.

To be able to compare this example very well with the one used in the corresponding JBehave article, the test description is written in BDD-syntax. Please note that this is not required and that the syntax to be used offers a lot of freedom. In the end what is relevant for the test execution are the commands that are embedded. This results in the following test description:

<html xmlns:concordion="http://www.concordion.org/2007/concordion">
<link href="../concordion.css" rel="stylesheet" type="text/css" />
<body>
 
    <h1>Stack Test</h1>
 
    <p>Testing the basic functionality of the Java Stack implementation.</p>
 
    <div class="example">
		<h3>Basic functionality of a Stack</h3>
        <p>
            Given an empty Stack<br/>
            <span concordion:execute="initializeStack()" />
            When the string <b concordion:set="#firstString">Java</b> is added<br/>
            <span concordion:execute="addElement(#firstString)" />
            And the string <b concordion:set="#secondString">C++</b> is added<br/>
            <span concordion:execute="addElement(#secondString)" />
            And the last element is removed again<br/>
            <span concordion:execute="removeLastElement()" />
            Then the resulting element should be 
            <b concordion:assertEquals="stackResult()">Java</b>
        </p>
    </div>
 
    <div class="example">
		<h3>Stack search</h3>
        <p>
            Given an empty Stack<br/>
            <span concordion:execute="initializeStack()" />
            When the string <b concordion:set="#firstString">Java</b> is added<br/>
            <span concordion:execute="addElement(#firstString)" />
            And the string <b concordion:set="#secondString">C++</b> is added<br/>
            <span concordion:execute="addElement(#secondString)" />
            And the string <b concordion:set="#thirdString">PHP</b> is added<br/>
            <span concordion:execute="addElement(#secondString)" />
            And the element Java is searched for<br/>
            Then the resulting element should be 
            <b concordion:assertEquals="stackSearch(#firstString)">3</b>
        </p>
    </div>
 
</body>
</html>

The corresponding Java class is trivial. It has to implement the instrumentalisation done in the HTML file, which are defined by the expressions: concordion:execute and concordion:assertEquals. This results in the following class:

package com.example.specs.stack;
 
import java.util.Stack;
import org.concordion.integration.junit3.ConcordionTestCase;
 
public class StackTest extends ConcordionTestCase {
 
    private Stack<String> testStack;
 
    public void initializeStack() {
        testStack = new Stack<String>();
    }
 
    public void addElement(String element) {
        testStack.push(element);
    }
 
    public void removeLastElement() {
        testStack.pop();
    }
 
    public String stackResult() {
    	return testStack.pop();
    }
 
     public int stackSearch(String element) {
    	return testStack.search(element);
    }
}

It is very nice that no further classes are required here to glue things together. All required functionality to run the tests is inherited from the class “ConcordionTestCase” and is thus transparent for the developer.

 

Reporting

The generated report for a test run looks very much like the test description itself. Thus it is up to the developer of the test description how well this report looks like and how easy it is to read the tests. On the other hand there is really not much additional functionality or information offered. Our example results in the following report:

Those values that are checked using assertions are marked green in case of success. Otherwise the – not expected – values are marked red.

 

Integration into a CI environment

Maybe I have overlooked something here, but on the Concordion web pages I was not really able to find any explanation how to execute the tests from the command line. I guess that this can be done using the Ant or Gradle scripts mentioned above.

As my experience using Gradle is even smaller than my experiences using Ant I try the latter one. And luckily everything works out without any problems. After opening the build.xml build file you can use the Eclipse Outline-View to select the “run tests” task. This one is then executing the tests and writes the results to the workspace under “build/concordion-output”.

This is the output that is generated by Ant. To implement a corresponding Maven configuration is something to keep in mind for a lter blog post :-).

Buildfile: C:\workspace\concordion\concordion-kickstart\build.xml
compile:
    [mkdir] Created dir: C:\workspace\concordion\concordion-kickstart\build\classes
    [javac] Compiling 3 source files to C:\workspace\concordion\concordion-kickstart\build\classes
     [copy] Copying 6 files to C:\workspace\concordion\concordion-kickstart\build\classes
run.tests:
    [mkdir] Created dir: C:\workspace\concordion\concordion-kickstart\build\test-output
    [mkdir] Created dir: C:\workspace\concordion\concordion-kickstart\build\concordion-output
    [junit] Running com.example.specs.greeting.HelloWorldTest
    [junit] C:\workspace\concordion\concordion-kickstart\build\concordion-output\com\example\specs\greeting\HelloWorld.html
    [junit] Successes: 1, Failures: 0
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0,234 sec
    [junit] Running com.example.specs.stack.StackTest
    [junit] C:\workspace\concordion\concordion-kickstart\build\concordion-output\com\example\specs\stack\Stack.html
    [junit] Successes: 2, Failures: 0
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0,031 sec
BUILD SUCCESSFUL
Total time: 3 seconds

 

Conclusion

Getting started with Concordion is quite easy for a new user. The example from the download packages works out-of-the-box and the underlying concept is easy to understand. Of course it helps here knowing already some other agile testing tools :-). It will be more difficult to really follow the concept of Concordion to write specifications and not scripts. The possibility to do this is definitly a strength of this tool.

When writing the specification the user has a lot of freedom and instrumenting the specification is straightforward. Overall Concordion is a lean tool and to some extend quite comparable with JBehave. Executing tests in some Continuous Integration environment is also possible without problems. The reporting on the other hand is minimalistic. But as usual everything depends on ones requirements here.

Of course there is more to explore, but hopefully this article offers a good starting point to start test automation using Concordion. Getting started is easy and there are hardly any frustrating moments working with it. Everything works as expected. Thus if you are evaluating agile testing tool Concordion is worth a glance.

Thomas Jaspers

Long-term experience in agile software projects using
Java enterprise technologies. Interested in test automation tools and concepts.

Share on FacebookGoogle+Share on LinkedInTweet about this on TwitterShare on RedditDigg thisShare on StumbleUpon

Kommentare

  • Prem Kumar

    19. May 2015 von Prem Kumar

    Hi Sir,

    Thanks for this post. I is helpfull.
    It will be really great if provide a project with 4-5 classes and there acceptance test using Concordion.

    Many Thanks šŸ™‚

Comment

Your email address will not be published. Required fields are marked *