FitNesse vs. Robot Framework – Agile Testing Tools

Selecting the “Opponents”

From the rather huge amount of – commercially and freely available – tools supporting agile testing I would like to have a closer look at FitNesse and the Robot Framework. Both tools are available for free and by looking at the corresponding project pages it can be seen that both frameworks are constantly enhanced. Another positive aspect is that there are mailing-lists available that can be used to get support. This is definitly helpful when using one of those tools in some real-life projects. Of course selecting these tools for a review might feel a bit random, especially because there are lot of other tools in this area. For example Concordion as a freely available alternative or Liquid as a commercial one to name only two. But as time is – unfortunetly – a limited resource I had to do some preselection here. Hopefully there will be the possibility to have a look at some of the alternatives in some later article still.

The Installation

At first glance the installation of FitNesse seems to be extremely simple. As the tool is entirely implemented in Java no additional prerequisites are needed. But the installation does not run as smoothly as expected as the proposed installation command java -jar fitnesse.jar was throwing a lot of errors (at least on my machine):

Unpacking new version of FitNesse resources. Please be patient.
java.io.FileNotFoundException: .FitNesseRootfilesimagesexecutionStatusThumbs.db (Das System kann den angegebenen Pfad nicht finden)

Thus I decided to use an older release for the evaluation of the tool. Here the installation comes as a ZIP-file and after unpacking this one FitNesse could be taken into use without any problems.

The Robot Framework seems to be more demanding here as it requires the installation of Python and Jython first. But due to the fact that all the installers are working very well there are also no problems in installing locally under Windows XP nor in installation on a Linux server.

Attention: To be able to use the graphical Editor of the Robot Framework Jython version 2.5.0 (or higher) needs to be installed. Jython versions are installed into parallel directories and if you have installed an older version first – like I have done – you must edit the file C:Python26Scriptsjybot.bat and set the corresponding line to set jython=”C:jython2.5.0jython.bat. The paths used here are the ones of the default installation under Windows.

Thus the effort for installation – from the download to the first successful startup – is very low for both tool. It is in the range of 15 to 30 minutes. And even though there have been some problems in the installation of FitNesse first I would rate both tools as very mature with respect to this topic.

The Philosophy

On first sight both tools are following quite different approaches. FitNesse is getting started as a Web-Server running its own Wiki.

FitNesse_1

New testcases can be added by editing the main page. Those pages can then be worked on as usual in a Wiki. It is possible to execute individual tests directly from the web frontend. Of course tests can also be executed using a scripting approach.

In comparison the Robot Framework looks a bit more frugal. Testcases are written using tables in HTML-format and there is also no possibility to execute individual tests from some kind of GUI. The test execution is always started using the command line.
But with the Robot IDE there is a very good graphical editor (even though still Alpha) that eases the creation and editing of the HTML-pages representing the tests extremely.

Robot_1

But there exists also a very fundamental commonness of both tools. Tests are implemented as a collection of so called keywords. Both frameworks support Java as a language to implement such keywords. The Robot Framework is supporting in addition Python “natively”, but the focus of this article is writing keywords in Java. Thus the common aspects are:

  • Keywords are written in Java and included as libraries by creating JAR-files for example.
  • Several keywords can be combined to create an individual testcase.
  • Parameters can be given to those keywords to test different scenarios.
  • Tests can be executed by a script and some kind of report is generated based on the execution.

Of course both tools have different focus areas as well when it comes to the definition of testcases. For example FitNesse has its own concept to pass testdata in tabular form to the testing class. In addition it is here to some extend possible to define keywords in a more “natural language'”, by mixing method names with parameters. The Robot Framework has here a more classical approach but due to the fact that method names can have spaces in the HTML definition this is also very well readable. It also supports the easy definition of new keywords by combining already existing ones. Another focus area is the possibility to extensively document testcases and to tag those, which is then used in the generated report later on.

Testing with Selenium

To evaluate both tools using a simple example that requires no additional source code I decided to use a simple Selenium-Test on some public web pages. Thus I will search for a specific site on Google and then navigate to a specific page on that site and check for the existence of some text element there. Both frameworks bring their own libraries with them to support Selenium. (FitNesse Selenium and Robot Selenium). But those have not really convinced me as there are too many problems and limitations and in the end it is simply much more complicated to use those libraries in comparison to the excelent Selenium IDE, which can be installed as a plug-in to Firefox.

Thus in this test I will utilize the Selenium IDE to generate a Java class with the relevant keywords (=testing methods). This is at the same time a good test to see how easy (or complicated) it is to use such Java libraries with these frameworks. The following class has been generated out of the Selenium IDE (as Java export). First the term “codecentric gmbh” is searched on Google. The codecentric homepage must then be among the results on the first page, as the corresponding link is clicked. Then we navigate to the imprint and check for the proper value of the trade register. (Of course this is a rather simple test, but should be enough for demonstrating purposes.)

package de.codecentric.agiletesting.demo;
 
import com.thoughtworks.selenium.*;
import java.util.regex.Pattern;
 
public class MyDemo extends SeleneseTestCase {
	public void setUp() throws Exception {
		setUp("http://www.google.de/", "*chrome");
	}
	public void testKeyword() throws Exception {
		selenium.open("/");
		selenium.type("q", "codecentric gmbh");
		selenium.click("btnG");
		selenium.waitForPageToLoad("30000");
		selenium.click("//div[@id='res']/div/ol/li[1]/h3/a/em");
		selenium.waitForPageToLoad("30000");
		selenium.click("//div[@id='footer_innen']/p/a[1]/span");
		selenium.waitForPageToLoad("30000");
		verifyTrue(selenium.isTextPresent("HRB 19214"));
	}
}

After some small adjustments to some names and the package name this class can be used for our tests right away. This has the additional advantage that those tests can be in priciple still be executed as a JUnit test from within Eclipse. This is a bit hidden by deriving from the class SeleneseTestCase.

Preparation
This class will now be included to a small Eclipse project and then exported as SeleniumKeywordDemo.jar. To do so we need in addition the selenium-java-client-driver.jar, which can be downloaded from the Selenium homepage. Then we need the junit.jar, which is part of every Eclipse installation. I am collecting all those JARs into one directory C:projectscodecentricSeleniumKeywordDemolib and the tools can utilize these JARs from there. Of course we need to start a local Selenium-Server in addition to be able to execute those tests.

Robot Framework – Integrating and executing the tests
A good starting point writing tests with the Robot Framework is to launch the graphical editor. Using a default Windows installation this can be done as:

C:>cd C:Python26Scripts
C:Python26Scripts>ride.py

Afterwards we are creating a new testsuite using Suite -> New Suite. Then the required libraries must be added under the top-level folder using -> New Library. It must be noted that here the full qualified class names (de.codecentric.agiletesting.demo.MyDemo) are given and not the name of any JAR-files. Those must be included to the classpath later on in the script used for the execution of the tests. In the Robot IDE one can then simply add a new test where the keywords from our Java class are getting called.
As the setUp()-method does not really implement a testcase it should be defined as the Setup-method for the testsuite instead. A click onto the following screenshot will lead to the HTML sources for the final Robot test.

Robot_2

Now the only thing left is the batch-file that is required to start the tests. The following script will do this job for a Windows system:

@echo off
set CLASSPATH=C:projectscodecentricSeleniumKeywordDemolibjunit.jar;C:projectscodecentricSeleniumKeywordDemolibselenium-java-client-driver.jar;C:projectscodecentricSeleniumKeywordDemolibSeleniumKeywordDemo.jar
C:Python26Scriptsjybot C:projectscodecentricSeleniumKeywordDemoSeleniumDemoTest.html

The test is then started by executing this batch-file and is running through smoothly. The Firefix- and the SeleniumRC-windows are staying open when the test is finished and must be closed manually. In a real test closing those windows could be easily added to a teardown-method in the Java class.

The Robot Framework is creating a XML output file from which a very nice and complete report can be generated in HTML format. An additional HTML file is generated that contains corresponding log-information that can be used for troubleshooting.

FitNesse – Integrating and executing the tests
The usual way to add a new test in FitNesse is using the web frontend. Thus it is required that the FitNesse server is running. Choosing Edit from the homepage it is possible to add a new test there called SeleniumDemoTest and save the page again. Now this page is displayed with a “?” as a link. By clicking on that link the new page is opened for editing. Also here the testcase is implemented quite quickly even though there is one trap. The Java class must be derived from the class DoFixture that comes with FitNesse as otherwise the following error is shown “Class de.codecentric.agiletesting.demo.MyDemo is not a fixture.”. To find this out really required a bit of research work. Thus we have the following source code for the test:

!path C:fitnessefitnesse.jar
!path C:fitnesselibfitlibrary.jar

!path C:projectscodecentricSeleniumKeywordDemolibjunit.jar
!path C:projectscodecentricSeleniumKeywordDemolibSeleniumKeywordDemo.jar
!path C:projectscodecentricSeleniumKeywordDemolibselenium-java-client-driver.jar

!|de.codecentric.agiletesting.demo.MyDemo|

|setUp|
|testKeyword|

In the Java class the fitlibrary.DoFixture must be imported and the class must be derived from DoFixture. Afterwards the test can be executed directly from the web frontend and results in the following output:

FitNesse_2

Usually tests will be executed as a part of a continuous integration environment and thus it must be possible to start them using a scripting approach. For FitNesse the following command can be used for this:

C:fitnesse>java -cp C:fitnessefitnesse.jar fitnesse.runner.TestRunner -xml report.xml localhost 8070 SeleniumDemoTest

This will also generate – a rather short – report in XML format.

Forgetting to include the fitnesse.jar JAR file into the test definition the execution of the test is hanging without any further error message. This happens when starting the test from the web and when starting it from a command line and is not too nice.

Summary

In my opinion it is not really possible to assess these tools in a “short” blog article in such a way that this results in a general recommendation. Due to the complexitiy of the tools and the whole topic “Agile Testing” it would be easily possible to write an own book for each of the frameworks. In addition I have been gained some experience with the Robot Framework in earlier projects before this evaluation which would make such a judgement a bit unfair in addition. But nevertheless I would like to name a few pros and cons I found from evaluating the tools.

One strength of FitNesse is for sure the specific format to enter testdata in a tabular format and the possibility to describe tests to some extend using “colloquial phrases”. The Robot Framework is not having that many possibilities here but is on the other hand not too far away from this. Starting tests from the web frontend is on first sight another advantage for FitNesse. But one has to check in detail here if this is really meaningful and possible in the concrete use case. Potential problems here are dependencies between tests and also the problem that some parameters must be passed to the testcase to execute them locally or in the ci-environment. Looking at the collaborative work on testcases the underlying SCM-system is probably more important than the question whether to use a Wiki or a graphical editor. In both cases the test documents must be stored under version control.

Wirting tests with the Robot Framework is very easy thanks to the graphical editor. This is especially true, because the underlying concepts of testsuites, testcases and keywords are nicely reflected here. When it comes to sharing testplans with a customer I also see some advantages in using the HTML-format as this can be viewed everywhere using some browser. Exchanging test definitions written with FitNesse requires that the Wiki-server is available as the source code of those is not too readable. Also the very strong reporting and the possibility to use any Java class (even JUnit tests) as libraries and keywords are on the positive side for the Robot Framework. But especially die-hard Java developers will have to overcome some mistrust here that Java testcases can really be executed without problems using Jython.

FitNesse, Robot or a completely different testing framework? It remains a questions of concrete project needs, available experiences within the team and personal preferences. Both tools are defintily worth a closer look when evaluating frameworks for agile testing.

  • Facebook
  • Delicious
  • Digg
  • StumbleUpon
  • Reddit
  • Blogger
  • LinkedIn
Thomas Jaspers

13 Responses to FitNesse vs. Robot Framework – Agile Testing Tools

  1. Rainer Foetzki says:

    Die Verwendung von Fit/Fitnesse fand ich in der Vergangenheit eher umständlich. Verwendet man statt Fitnesse allerdings Slim zusammen mit Trinidad fallen die genannten Nachteile weg.
    – Tests können ins VCS gelegt werden
    – Eigene Fixtures sind nicht mehr abgeleitet
    – Neue Fixtures erleichtern den Einsatz
    – Es wird kein wiki Server benötigt
    – Tests können als JUnit Test in der IDE gestartet werden

    Robot Framework verwende ich derzeit für den Test von Swing Applikationen und bin bisher sehr angetan.

    Für Kunden erstelle ich eine “ausführbare Dokumentation” mit Concordion – sehr empfehlenswert!

  2. Thomas Jaspers says:

    “Ausführbare Dokumentationen” sind definitiv ein interessantes Thema und danke für den Tip mit Concordion, werde ich mir auf jeden Fall anschauen.

    Das Robot Framework hat laut Forum im neuesten Release auch besseren Support in diese Richtung bekommen: “… ease creating requirement-like test cases with sentence-like keywords such as `Given user selects book from list`. …”

    Das wäre ja dann auch noch mal einen Vergleich mit Concordion wert :).

  3. Petri Haapio says:

    Hi Thomas,

    Nice review, thanks.
    And good to see that you’ve found a good company to work with!

    Cheers,
    Petri

  4. Thomas Jaspers says:

    Hi Petri,

    cool you found this one :) … and thanks for the positive feedback. In October I will be at the Agile Testing Days in Berlin and hope to get hold of Pekka to get latest news from Robot Framework :-). Have a good time.

    Cheers
    – Thomas

  5. Tuomas Kiviaho says:

    Fitnesse reminded me of a cool product called GreenPepper. The GUI is basically either JIRA, Confluence or Maven you like CLI. If I remember correctly selenium dialect was also supported.

  6. Susana says:

    I tried to do the integration of selenium IDe and RIDE but I have a problem with the library import
    “ImportError:No module named ”

    Do you have some idea about which is the problem?

    Thanks in advance

  7. Thomas Jaspers says:

    Sorry for the late reply, but I saw your comment only now. If the problem still exists you can try importing the Selenium Library as follows:

    Library | SeleniumLibrary | 10.0 | localhost | 4445

    The “10.0” is a timeout value that can be adjusted and “localhost” and “4445” must be replaced with your server name and port.

    As Selenium Library is an external Robot Library you have to download and install it before taking it into use. You can download it from here: http://code.google.com/p/robotframework-seleniumlibrary/downloads/list

    If you have more problems please let me know and I can try to help :-).

  8. John says:

    You may also like to try out Tellurium for structured tests. For example,

    ui.Container(uid: "GoogleSearchModule", clocator: [tag: "td"], group: "true"){
       InputBox(uid: "Input", clocator: [title: "Google Search"])
       SubmitButton(uid: "Search", clocator: [name: "btnG", value: "Google Search"])
       SubmitButton(uid: "ImFeelingLucky", clocator: [value: "I'm Feeling Lucky"])
    }
    
    type "GoogleSearchModule.Input", "Tellurium test"
    click "GoogleSearchModule.Search"
    waitForPageToLoad 30000
    
  9. Uncle Bob says:

    In the 8 months since this review was posted, a lot of work has been done on FitNesse to make it even better.

  10. Jaya Chakravarthi says:

    Hi Thomas,

    That was quite helpful.

    But the ‘selenium’ object in testKeyword method threw a null pointer exception. May be the object dint exist over in that method. I might have done something wrong here, though.

    What i tried is create DefaultSelenium’s private object @ global level and make it static. This time around it dint throw any nullPointerException.

    Thanks again !

  11. Pingback: QuickLinks for March 2011 | (Agile) Testing

  12. Thank you Thomas for mentioning LiquidTest (you refer to it as “Liquid”). It might be noteworthy for anyone reading that LiquidTest 3.0 now supports the Selenium format. Thomas I hope you get a chance to come back and check out LiquidTest again. – Anthony Scotney (Founder of JadeLiquid – LiquidTest creator)

  13. Sébastien B. says:

    So many test automation tools available out there and so many different requirements it’s impossible to tell which is “better”.

    If you evaluate Fitnesse, you might want to give a try at Xebium. It’s built on top of Fitnesse but allows a better integration with Selenium IDE.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>