Overview

Cucumber: Testing Web Applications With Capybara, Poltergeist and PhantomJS

13 Comments

This tutorial shows how to write acceptance tests for web applications using Cucumber, Capybara, Poltergeist and PhantomJS. Along the way we will also briefly touch some other interesting technologies like Node.js and AngularJS. This is actually the second part of a two part tutorial about Cucumber. If you have never used Cucumber before you might want to start with part one of this tutorial.

The Stack in Detail – Capybara, Poltergeist and PhantomJS

Before we dive head first into the action we should take a moment to have a look at the tools that we will be using in addition to Cucumber (which has been introduced in part one):

  • Capybara calls itself an “acceptance test framework for web applications”. Wait. Isn’t Cucumber already an acceptance test framework? Why do we need another? First, Cucumber is all about Behaviour Driven Development and not per se an acceptance test framework. You can work with Cucumber on the unit test level, or on the acceptance test level or anywhere in between. Second, Cucumber knows nothing about web applications. Cucumber only gives you the ability to write your tests (or specs or scenarios, whatever you call them) in a very readable, non-technical form (Gherkin) and to execute them. Capybara now integrates nicely with Cucumber (see section Using Capybara with Cucumber in the Capybara docs) and hides the details of controlling a browser (via Selenium or Poltergeist) behind a nice API. It is just the right level of abstraction (for my taste) to make writing web application tests fun. Of course you could use something like Selenium directly in your Cucumber step definitions, but in my opinion that is too low-level and too verbose.
  • Poltergeist is labeled “a PhantomJS driver for Capybara” and that’s just what it is. It is the connection between Capybara and PhantomJS, see below. After registering Poltergeist as the driver in Capybara you do not interact directly with Poltergeist nor PhantomJS, only with the Capybara API.
  • PhantomJS is headless, full featured browser. It is based on WebKit and supports all important web standards (CSS, JavaScript, …). It is ideally suited for automated website tests because running it on a CI server (even without X installed) is a no-brainer.

Setup

Setting up the Application Under Test

As this tutorial is about testing web applications we obviously need a web application to test. Let me introduce you to the Audiobook Collection Manger. If you are a regular at this blog you might have met the Movie Database application in one of its several incarnations. Since I’m really not that much into movie but have a serious audio book fetish, this time it’s the Audiobook Collection Manager, a simple AngularJS CRUD application to, well, manage your collection of audio books. Head over to https://github.com/basti1302/audiobook-collection-manager-ui and follow the instructions to set it up. In the process you will also clone Storra, which is a very simple persistence REST service written in Node.js, used by the Audiobook Collection Manager front end to store its data. Hint: In part one I mentioned that Cucumber has been ported to many languages. If you want to see a bit of cucumber.js, the JavaScript port of Cucumber, you can take a look at the directory storra/features.

If you have the Audiobook Collection Manager up and running, you might want to explore the application manually for a minute, to see what functionality it offers (not much, really).

Setting up Capybara and Poltergeist

If you have followed the first part of this tutorial, you should already have checked out the example project. If so, do

git checkout 03_setup_capybara

now, to get the new Gemfile with the required gems for Capybara and Poltergeist. (In this branch, the first feature and the corresponding step file have been removed because we no longer need them.)

If you have not yet cloned the repository, you might want to do that now:

git clone -b 03_setup_capybara https://github.com/basti1302/audiobook-collection-manager-acceptance.git

The Gemfile has changed, there are a few new gems in it so you should do bundle install now. This will install the Capybara and Poltergeist gems and a few other gems.

If you followed the first part of the tutorial, you should also have PhantomJS installed. If not, head over to http://phantomjs.org/download.html and do that now.

The branch of audiobook-collection-manager-acceptance that you just checked out comes with all necessary configuration code for Capybara and Poltergeist, so you don’t need to do anything there. We can quickly review the configuration code. All setup code is contained in features/support/env.rb and features/support/hooks.rb. env.rb starts with a few requires:

require 'rspec/expectations'
require 'capybara/cucumber'
require 'capybara/poltergeist'

We already have discussed require 'rspec/expectations' in part one of this tutorial – it makes the RSpec object expectations available in all step files. require 'capybara/cucumber' does the same for the methods from the Capybara API. require 'capybara/poltergeist' is needed to register Poltergeist as the browser driver for Capybara.

Following the require statements comes a relatively large if-else block:

if ENV['IN_BROWSER']
  # On demand: non-headless tests via Selenium/WebDriver
  # To run the scenarios in browser (default: Firefox), use the following command line:
  # IN_BROWSER=true bundle exec cucumber
  # or (to have a pause of 1 second between each step):
  # IN_BROWSER=true PAUSE=1 bundle exec cucumber
  Capybara.default_driver = :selenium
  AfterStep do
    sleep (ENV['PAUSE'] || 0).to_i
  end
else
  # DEFAULT: headless tests with poltergeist/PhantomJS
  Capybara.register_driver :poltergeist do |app|
    Capybara::Poltergeist::Driver.new(
      app,
      window_size: [1280, 1024]#,
      #debug:       true
    )
  end
  Capybara.default_driver    = :poltergeist
  Capybara.javascript_driver = :poltergeist
end

You don’t need to analyze this in detail, but here are the important bits: The else part is what we will be using most of the time. It registers Poltergeist as the driver for Capybara, and Poltergeist in turn uses PhantomJS. PhantomJS, being headless, is very convenient for running in a continuous integration environment (that is, on your build server).

If, however, once in a while you want to see what happens in a real browser, you can start tests like this

IN_BROWSER=true bundle exec cucumber

or, if you are on Windows:

SET IN_BROWSER=true
bundle exec cucumber

With this environment variable present, Selenium WebDriver is used instead of Poltergeist and Firefox instead of PhantomJS, so you can watch your scenarios executing. If stuff happens too quickly for your taste you can even do

IN_BROWSER=true PAUSE=1 bundle exec cucumber

or on Windows

SET IN_BROWSER=true
SET PAUSE=1
bundle exec cucumber

to have Cucucumber wait one second after each step.

The rest of env.rb defines some constants and helper methods for accessing the application and test data fixtures.

hooks.rb currently contains only this:

After do |scenario|
  if scenario.failed?
    save_page
  end
end

which registers a hook that is run after each scenario. If the scenarios has been marked as failed, Capybara is instructed to write a snapshot of the html to disk for later inspection. Take a look at the Cucumber documentation for more information on hooks.

Testing the Web Application

Your First Feature With Capybara

Now that we have the setup down, we can start with the first scenario, shall we?

The first feature we would like to test is to list all audio books which are already in the collection. So without further ado, let’s write down how we want the application to behave when the user goes to the page that contains the list of all entries. That’s how the file features/list_audiobooks.feature could look like:

Feature: Display the list of audio books
  In order to know which audio books the collection contains
  As an audio book enthusiast
  I want to see a list of all audio books
 
  Scenario: Display the list of all audio books in the collection
    Given some audio books in the collection
    When I visit the list of audio books
    Then I see all audio books

Remark: All code shown in this section (the Cucumber feature, the step definition plus the helper file storage.rb) is available in the branch “04_list_audiobooks”. You can fetch it with git checkout 04_list_audiobooks.

Remember that in your scenario you can write anything you like as long as each line begins with Given, When, Then or And. Thus you do not need to think about how you will implement the step later, instead you can focus on the requirement you are trying to express. This is why it is often better to write the scenario first and implement the step definitions later. The scenario above should be fairly self explaining.

You can execute bundle exec cucmber to get some suggestions for the step definitions or write them from scratch. Let’s put the step definitions in features/step_definitions/audiobooks.rb.

#encoding: utf-8
 
Given /^some audio books in the collection$/ do
  upload_fixtures backend_url('audiobooks'), $fixtures
end
 
When /^I visit the list of audio books$/ do
  visit ui_url '/index.html'
end
 
Then /^I see all audio books$/ do
  page.should have_content 'Coraline'
  page.should have_content 'Man In The Dark'
  page.should have_content 'Siddhartha'
end

This step definitions use Capybara’s DSL to implement the steps from the scenario. You can read more about the Capybara DSL on its Github page or on rubydoc.info. Let’s take a closer look.

The Given step calls upload_fixtures, a method that is implemented in features/support/storage.rb and uses Storra (the persistence service used by the AngularJS UI) directly to insert a number of audio books from a file into the database. The code is not specific to Cucumber or Capybara, so I’m not going into detail here but you are invited to have a look at the code in storage.rb if you are interested.

The When step: Parentheses for method calls are optional in ruby, that is why visit ui_url '/index.html' is the same as visit(ui_url('/index.html')) (I personally think the version without parentheses reads more fluently but that is a matter of taste). ui_url is a little helper method from env.rb that translates a path into the full URL of the application under test. Finally, visit is a method from Capybara that navigates to the given URL. Thus, visit ui_url '/index.html' just points the browser (PhantomJS in this case) to http://localhost:8000/app/index.html, the page which displays the list of audio books.

The Then step uses Capybara’s page object (a representation of the current page, that is, the current DOM as presented to the browser) to verify that three pieces of text are there. Let’s assume the step Given some audio books in the collection would have inserted three audio books, namely “Coraline” by Neil Gaiman, “Man in the Dark” by Paul Auster and “Siddhartha” by Herman Hesse. Reading Capybara’s documentation you would probably expect the check to read as page.has_content?('Coraline'). To spice things up a bit, we use RSpec’s ability to create a custom matcher for any predicate (any method ending with a “?”) on the fly, see RSpec Matcher docs. The RSpec library exploits Ruby’s meta-programming capabilities to create these matchers and as a result, we can write page.should(have_content('Coraline')), which is what we did (except for the parentheses).

A Word About Test Data

If you execute the feature multiple times and then open http://localhost:8000/app/index.html in your browser, you will see that the list of audio books is littered with multiple copies of the test fixtures that we insert in the Given step. This can turn into a real problem later when we try to test features like adding or deleting audio books: The tests are not isolated because they operate on the same database. Imagine a scenario for adding an audio book, let’s say with the title “Foobar”. To see if that operation was successful we would probably check page.should have_content 'Foobar' in the Then step after navigating to the list of audio books. Now this test might run fine a number of times. But what if the feature to add an audio book breaks in the application under test? Our Cucumber scenario might still be green as a cuke because there are a number of copies of “Foobar” in the database already. That’s the worst case for an automated test: a test that is green although the production code is broken. So we need to do something about that.

This problem is relevant for almost every application on the level of acceptance testing and there are at least two different solutions to this problem:

  1. Delete everything before or after each test. With everything I really mean the whole database. This is often the easiest option. Of course, this makes it necessary to have a dedicated environment to run your acceptance tests in CI (separate from a stage for manual testing, that is). But any non-trivial project should have that anyway.
  2. Separate tests by using some kind of unique data space for each test run. It depends on your domain model if this option is suitable. Quick example: Let’s say you have a customer object, which in turn can have one or more associated order objects and each order might have a few items. A user always only sees and interacts with her own orders. In such a setting you could just create a new user for each test (maybe with a uuid as her id) and you have your test isolation covered.

We will go with option 1 and ask Storra to delete the entire collection before each test by adding the following lines to features/support/hooks.rb:

Before do
  delete_database backend_url('audiobooks')
end

and this to features/support/storage.rb:

def delete_database(url)
  RestClient.delete url
end

Note: backend_url is defined in features/support/env.rb and returns the URL for the collection resource managed by Storra.

More Tests for Showing the List of Audio Books

Note: The code for the following scenario is available per git checkout 05_filter. This branch also contains the hooks to delete the database as discussed in the previous section.

If you have explored the audio book collection manager manually you might have noticed the text input with the label “Filter” next to it. If you start to type something there, only matching audio books will be shown and titles which do not match will be hidden. So, if you type “Cor”, “Coraline” will be shown but other titles will be omitted from the list.

To express this as a Cucumber scenario, we can add the following to features/list_audiobooks.feature:

  Scenario: Filter the list
    Given some audiobooks in the collection
    When I visit the list of audiobooks
    And I search for "Cor"
    Then I only see titles matching the search term
    When I remove the filter
    Then I see all audiobooks again

To implement the steps we can add this to features/step_definitions/audiobooks.rb:

When /^I search for "(.*?)"$/ do |search_term|
  fill_in('filter', :with => search_term)
  @matching_titles = ['Coraline']
  @not_matching_titles = ['Man In The Dark', 'Siddharta']
end
 
When /^I remove the filter$/ do
  # funny, '' (empty string) does not work?
  fill_in('filter', :with => ' ')
  @matching_titles = @not_matching_titles = nil
end
 
Then /^I see all audiobooks(?: again)?$/ do
  page.should have_content 'Coraline'
  page.should have_content 'Man In The Dark'
  page.should have_content 'Siddhartha'
end
 
Then /^I only see titles matching the search term$/ do
  @matching_titles.each do |title|
    page.should have_content title
  end
 
  @not_matching_titles.each do |title|
    page.should have_no_content title
  end
end

Let’s go through the step definitions one by one:

  • When I search for...: The first line (fill_in('filter', :with => search_term)) uses Capybara’s API to type a value into the text box. The value is provided by a capturing group from the regex, so if we call this step with When I search for "Foobar" “Foobar” would be entered into the text box. The next two lines set the instance variables @matching_titles and @not_matching_titles. This is a technique I use from time to time with Cucumber, that is, setting up expectations (or rather: expected values) in When steps which will be checked later in a Then step. The expected values set here are checked immediately in the next step. This might feel a bit like a hack because expectations are not the domain of the When step but of the Then step. However, it often provides a pragmatic way to use a Then step with different expectations (set up in different When steps). In this particular case there is definitely room for improvement because there is a mismatch between using a variable text for the the fill_in call (the variable search_term, taken from the capturing group) and the hard coded values for @matching_titles and @not_matching_titles.
  • Then I only see titles matching the search term: This step uses the expected values that have been set up in the When step (see above). For all titles matching the search term, we verify that the content is on the page. For the non matching titles, we verify that the content is not there.
  • When I remove the filter: This justs removes the input from the text box which should result in all audio books being shown.
  • The step definition Then /^I see all audiobooks(?: again)?$/ do is our existing step definition (Then /^I see all audiobooks/ do), only the regex has been expanded with the optional non-capturing group (?: again)? to make the Cucumber scenario more readable. If you copied the above into your existing audiobooks.rb, make sure to delete the original step definition without the expanded regex to avoid ambigious steps.

Testing AJAX functionality with Capybara

Let’s pause for a moment and think about the test we just implemented. The application under test is implemented with AngularJS and is a true Single Page Application. There are is no navigation that takes the user from one page to another, everything happens on the same page by asynchronous JavaScript, DOM manipulation and asynchronous data exchange with a backing service (Storra). For instance, the filter functionality tested in the last section happens completely on the client side, that is, when the search term changes, some list items are removed from the DOM and others are shown again.

Did this make our testing harder? More complicated? Did we need special code for testing the AJAX stuff? No! We did not even think about that until now. The reason is: Capybara is based on the assumption that in a modern web application, potentially everything might happen asynchronously. Whenever you verify that some content is there or some condition is met, Capybara by default waits for the content to appear or the condition to become true. The timeout for this is configurable, of course.

This is what I expect from a decent browser test abstraction nowadays: When writing acceptance tests, I don’t care if the application under test produces the expected content by navigating to a whole new page, does a partial DOM update via AJAX or a tiny imp inside my screen paints the content with a miniature brush. I just want to test if the expected content is there. And I don’t want to change my test code when the application under test changes the way it produces the content.

Capybara fulfills this expectation. A number of other web application test frameworks do not, especially the ones operating on a lower level, like Selenium. If your test code is littered with stuff like waitForXyz or even the dreaded, arbitrary sleep 2s, chances are you are using the wrong test framework.

Homework: More Features and Scenarios

We could now continue to write more features and scenarios for the audio book collection manager. The master branch indeed contains a few more, for example for adding an audio book or for displaying detailed information about a particular audio book. You can take a look at them with git checkout master. If you have worked your way through this tutorial up to this point, probably neither the additional features nor their corresponding step definitions contain something fundamentally new to you.

You are invited to add more Cucumber features of your own, for example for deleting an audio book or for showing the cover picture of an audio book (the cover picture is automatically shown in the details page if the ASIN of the audio book has been provided and amazon.com has an image for this ASIN).

Conclusion

This completes our little excursion into the world of Cucumber and Capybara. I hope you enjoyed the ride. This tutorial only provides a little glimpse of what Cucumber and Capybara can do, of course it does not cover every aspect of the mentioned frameworks. I think it nonetheless shows that this combination is a quite powerful one and allows you to write your acceptance tests in an elegant and readable manner.

If you have questions or comments, please drop a line below.

Credits

A part of the setup code (especially the stuff with using either Poltergeist and PhantomJS or Selenium WebDriver and Firefox, controlled by an environment variable) was created by Michael Schuerig with whom I had the pleasure to work together in a former project. He also introduced me to Capybara and Poltergeist.

Kommentare

  • nik

    I get the following error with phantomjs:

    When I visit the list of audio books # features/ste
    p_definitions/audiobooks.rb:7
    The detector # failed to detect theversion of the executable at ‘C:/Program Files/phantomjs-
    1.9.2-windows/phantomjs.EXE’ (ArgumentError)
    ./features/step_definitions/audiobooks.rb:8:in `/^I visit the list of audi
    o books$/’
    features\list_audiobooks.feature:8:in `When I visit the list of audio book
    s’

  • Bastian Krol

    Hi Nik,

    I can not reproduce this problem. I just tested it successfully with PhantomJS 1.9.2 on a Windows machine. What else are you using? Ruby 1.9 or 2.0? Windows 7, 8 or XP? 64-bit? 32-bit? …

    What happens if you just do phantomjs –version on the command line?

    I justed tested it on a fairly old 32-bit WinXP box with Ruby 2.0 and PhantomJS 1.9.2. In the process, I also updated the Gemfile.lock in the repository to work with Ruby 2.0. Maybe you could try to do git pull and bundle install again on your machine.

    Not sure if that helps, though.

    • nik

      Ruby 1.93, windows 7 64bit

      phantomjs –version returns ‘1.8.1’

      Might be related to

      https://github.com/jonleighton/poltergeist/pull/366

      or

      https://github.com/jonleighton/poltergeist/issues/389

      • Bastian Krol

        So the error message is about ‘C:/Program Files/phantomjs-
        1.9.2-windows/phantomjs.EXE’ and phantomjs –version returns 1.8.1. Sounds weird, no? Looks like you have 1.8.1 installed somewhere in the PATH but Poltergeist tries to find it in a path that sounds like version 1.9.2. My guess is that your PhantomJS installation or the PATH environment variable is somehow messed up. I think you should find out where this 1.8.1 PhantomJS executable hides and clean up your PATH, so that it only includes the 1.9.2 executable’s directory. Maybe you also need to reinstall PhantomJS.

        • Miguel Angel Pozos

          28. October 2013 von Miguel Angel Pozos

          I can also reproduce this problem.

          I’m running Ruby 1.9.3 p 448 and my command prompt returns “1.9.2” for phantomjs –version

          I’m on Windows 8.1 (64-bit)

          As far as reinstalling PhantomJS, it’s just a zip file to extract and add to PATH environment variable, right? If the console returns a version, the system can find the executable though Cliver cannot

          • Miguel Angel Pozos

            28. October 2013 von Miguel Angel Pozos

            Update: I have resolved my issue by moving my PhantomJS install away from Program Files (a directory with a space in the name) to the root C: (and updated the PATH environment).

            I had tried updating the cliver gem to the latest version and received the same error.

  • Bastian Krol

    @Miguel: Good to see you could solve your issue. As for the reinstall, quite possible it’s just a zip, it’s been a while since I used it on Windows.

    @nik: I think the PhantomJS or the Poltergeist mailing list might be better able to help you. Sorry.

  • Rob

    I had the same issues as above, and came to the conclusion that the Cliver gem doesn’t like relative paths, so using something like the following should resolve it:


    Capybara.register_driver :poltergeist do |app|
    Capybara::Poltergeist::Driver.new(
    app,
    window_size: [1280, 1024],
    phantomjs: 'c:/phantomjs-1.9.2-windows/phantomjs.exe'
    )
    end

  • Bhagi Raj Limbu

    11. December 2013 von Bhagi Raj Limbu

    It is so useful, readable and wonderful article.

    Thank you.

    Regards and Metta,
    Bhagi Raj Limbu

  • Andrew Camilleri

    9. April 2014 von Andrew Camilleri

    This is such a hassle… why couldn’t they keep things simple and documented?

  • Thu Nguyen

    21. April 2014 von Thu Nguyen

    I could not download the source code from Git, please help how to checkout source code from git. I’m studying Cucumber + Capybara, I really need the sourcecode for practise!

    Thanks in advance!
    Thu Nguyen

Comment

Your email address will not be published. Required fields are marked *