Overview

Style Tests using Selenium and Robotframework

4 Comments

In projects facing end customers style matters, often more than less. While in internal apps it doesn’t matter if the UI changes after each release, there might be customers that want their app to follow a very strict style guide to integrate with their corporate identity.
If you ever worked with HTML and pixel perfect CSS you know that things can easily screw up, even with very strict layout guidelines.
Last Friday we brainstormed in our team on how to make this part of our nightly build.
We found a solution which was surprisingly simple..

The idea

When the developer created HTML and CSS for a component (like a login box) and made sure that it was looking fine in all required browsers, the developer would take a screenshot, and crop it to contain only the box. Like depicted here on the right, we would want to make sure that the logo inside the red box looks exactly like that.
Next the automatic build needs to be told on which page this box is expected to be (e.g. a page with a user that is not logged in). The test takes then a screenshot and searches it for the expected box.
We considered also telling at which position we expect it, but that would cause multiple issues: The dev needs to find the pixel position. It needs to be absolute on different screen sizes, changes are hard to track.
When thinking about this, we believe that finding the expected sub-image in the screenshot is sufficient. If you would want to make sure it is at a specific position you just include neighboring objects in your expectation image.

The Implementation

We wrote a very simple keyword that takes a screenshot of a given URL (and possibly after performing steps like putting stuff to a shopping cart) and checks the occurrences of template screenshots. The keyword is implemented in Robot BDD using Selenium.

*** Keywords ***	
 
Match Styles ${Styles}
    Execute Javascript  window.resizeTo(100,100)
    ${ID} =  Get Unique ID
    Capture Page Screenshot  ${OUTPUT DIR}${/}StyleTest-${ID}.png  css=
	:FOR  ${style}  IN  @{Styles}
    \	Match Style  ${CURDIR}${/}styles${/}${style}.png  ${OUTPUT DIR}${/}StyleTest-${ID}.png
    Remove File  ${OUTPUT DIR}${/}StyleTest-${ID}.png

The Javascript helps in bringing down the file size of the screenshot and makes searching faster, as it compresses unused whitespace which occurs in many liquid layouts. Get Unique ID generates us an ID using Java UUID.

The keyword is simple:

public static void matchStyle(String expectedFilename, String actualFilename) throws IOException {
	boolean check = StyleChecker.check(expectedFilename, actualFilename);
	if (!check) {
		throw new IllegalStateException("could not find expected image in actual screenshot");
	}
}

The actual picture searching is done in Java. The code is not superclever, but after a few rounds of optimization pretty fast (worst case: up to 300ms when expected part is not found, in real use cases averages about 200ms for existing sub-images). In fact the screenshot generation is the bottleneck.
Java code can be downloaded and is excluded here for brevity.

A test can look like this:

*** Keyword ***
Behaviour
    [Arguments]  ${Url}  @{Styles}
 
    Go To  ${Url}
    Match Styles @{Styles}
 
| *Test Case* | | *Url*                        | *style*          |
| 1 | Behaviour | http://${BASE_URL}${CONTEXT} | Login | ShoppingCart-Empty|

Caveats

  1. The screenshots need to be lossless.
  2. Our code is pixel perfect. Which means that every tiny detail is important. Because of that Developers cannot take screenshots of their own to crop, but need to take the screenshot that is produced by selenium. There are differences, for example clear type, or available fonts.
  3. When implementing picture comparison in Java you can make it horribly wrong. I hope my code is not that bad. At least I benchmarked it and for the simplicity of the algorithm it performs fine.

Benefits

With those simple steps we are able to do regression testing on our user interfaces that have strict standards. It helps even more, because CSS is sometimes a bit fragile. And because of the simplicity of the testsetup it is also very easy to add new test withing seconds.

Debugging

What happens, if a style test fails? More often than not, the analysis is trivial, because style bugs that should be found with these kind of tests (wrong borders and margins, colors don’t match, wrong image, etc.) can be quickly detected by looking at the screenshot. But right now we had the situation, where this was not the case, and immediately suspected a refactoring gone bad. We were wrong! In order to detect differences in two images, you can use GIMP’s layer mode “difference”;

  • Load screenshot (1) in GIMP
  • Load image (2), that should be found in the screenshot, in GIMP
  • Lay image (2) as new, half-transparent layer over the screenshot
  • Apply mode “difference” to the new layer
  • Merge the visible layers
  • Brighten up the darker image parts, to see the differences.

So far for the short tutorial for debugging. In case you run into troubles, please comment.

Kommentare

  • Trimper

    12. July 2010 von Trimper

    Will this work in more than just IE and Firefox where Selenium screenshot capabilities function?

  • Fabian Lange

    As this invokes the “Capture Page Screenshot” Keyword, it does require it. It is restricted to browsers where it is supported.

  • Andreas Ebbert-Karroum

    @Trimper,

    another thing to watch out for, when doing style tests for different browsers is that there will be differences between the screenshots, so you probably need different screenshots to compare, one for each browser.

    You could do it, by putting all style templates for one browser in a subdirectory, identified by the variable ${BROWSER_PROFILE}. You probably already have a similar variable to configure Selenium accordingly.

    *** Keywords ***	
     
    Match Styles ${Styles}
        Execute Javascript  window.resizeTo(100,100)
        ${ID} =  Get Unique ID
        Capture Page Screenshot  ${OUTPUT DIR}${/}StyleTest-${ID}.png  css=
    	:FOR  ${style}  IN  @{Styles}
        	Match Style  ${CURDIR}${/}styles${/}${BROWSER_PROFILE}${/}${style}.png  ${OUTPUT DIR}${/}StyleTest-${ID}.png
        Remove File  ${OUTPUT DIR}${/}StyleTest-${ID}.png
  • eleg

    you could use “perceptual hash”:

    http://www.phash.org/

    if you want it to be less fragile.

Comment

Your email address will not be published. Required fields are marked *