This tutorial shows how to write acceptance tests for web applications using Cucumber, Capybara, Poltergeist and PhantomJS. Along the way we will also briefly touch some other interesting technologies like Node.js and AngularJS. This is actually the second part of a two part tutorial about Cucumber. If you have never used Cucumber before you might want to start with part one of this tutorial.
The Stack in Detail – Capybara, Poltergeist and PhantomJS
Before we dive head first into the action we should take a moment to have a look at the tools that we will be using in addition to Cucumber (which has been introduced in part one):
- Capybara calls itself an “acceptance test framework for web applications”. Wait. Isn’t Cucumber already an acceptance test framework? Why do we need another? First, Cucumber is all about Behaviour Driven Development and not per se an acceptance test framework. You can work with Cucumber on the unit test level, or on the acceptance test level or anywhere in between. Second, Cucumber knows nothing about web applications. Cucumber only gives you the ability to write your tests (or specs or scenarios, whatever you call them) in a very readable, non-technical form (Gherkin) and to execute them. Capybara now integrates nicely with Cucumber (see section Using Capybara with Cucumber in the Capybara docs) and hides the details of controlling a browser (via Selenium or Poltergeist) behind a nice API. It is just the right level of abstraction (for my taste) to make writing web application tests fun. Of course you could use something like Selenium directly in your Cucumber step definitions, but in my opinion that is too low-level and too verbose.
- Poltergeist is labeled “a PhantomJS driver for Capybara” and that’s just what it is. It is the connection between Capybara and PhantomJS, see below. After registering Poltergeist as the driver in Capybara you do not interact directly with Poltergeist nor PhantomJS, only with the Capybara API.
Setting up the Application Under Test
If you have the Audiobook Collection Manager up and running, you might want to explore the application manually for a minute, to see what functionality it offers (not much, really).
Setting up Capybara and Poltergeist
If you have followed the first part of this tutorial, you should already have checked out the example project. If so, do
git checkout 03_setup_capybara
now, to get the new Gemfile with the required gems for Capybara and Poltergeist. (In this branch, the first feature and the corresponding step file have been removed because we no longer need them.)
If you have not yet cloned the repository, you might want to do that now:
git clone -b 03_setup_capybara https://github.com/basti1302/audiobook-collection-manager-acceptance.git
The Gemfile has changed, there are a few new gems in it so you should do
bundle install now. This will install the Capybara and Poltergeist gems and a few other gems.
If you followed the first part of the tutorial, you should also have PhantomJS installed. If not, head over to http://phantomjs.org/download.html and do that now.
The branch of
audiobook-collection-manager-acceptance that you just checked out comes with all necessary configuration code for Capybara and Poltergeist, so you don’t need to do anything there. We can quickly review the configuration code. All setup code is contained in
env.rb starts with a few requires:
require 'rspec/expectations' require 'capybara/cucumber' require 'capybara/poltergeist'
We already have discussed
require 'rspec/expectations' in part one of this tutorial – it makes the RSpec object expectations available in all step files.
require 'capybara/cucumber' does the same for the methods from the Capybara API.
require 'capybara/poltergeist' is needed to register Poltergeist as the browser driver for Capybara.
require statements comes a relatively large if-else block:
You don’t need to analyze this in detail, but here are the important bits: The
else part is what we will be using most of the time. It registers Poltergeist as the driver for Capybara, and Poltergeist in turn uses PhantomJS. PhantomJS, being headless, is very convenient for running in a continuous integration environment (that is, on your build server).
If, however, once in a while you want to see what happens in a real browser, you can start tests like this
IN_BROWSER=true bundle exec cucumber
or, if you are on Windows:
SET IN_BROWSER=true bundle exec cucumber
With this environment variable present, Selenium WebDriver is used instead of Poltergeist and Firefox instead of PhantomJS, so you can watch your scenarios executing. If stuff happens too quickly for your taste you can even do
IN_BROWSER=true PAUSE=1 bundle exec cucumber
or on Windows
SET IN_BROWSER=true SET PAUSE=1 bundle exec cucumber
to have Cucucumber wait one second after each step.
The rest of
env.rb defines some constants and helper methods for accessing the application and test data fixtures.
hooks.rb currently contains only this:
After do |scenario| if scenario.failed? save_page end end
which registers a hook that is run after each scenario. If the scenarios has been marked as failed, Capybara is instructed to write a snapshot of the html to disk for later inspection. Take a look at the Cucumber documentation for more information on hooks.
Testing the Web Application
Your First Feature With Capybara
Now that we have the setup down, we can start with the first scenario, shall we?
The first feature we would like to test is to list all audio books which are already in the collection. So without further ado, let’s write down how we want the application to behave when the user goes to the page that contains the list of all entries. That’s how the file
features/list_audiobooks.feature could look like:
Feature: Display the list of audio books In order to know which audio books the collection contains As an audio book enthusiast I want to see a list of all audio books Scenario: Display the list of all audio books in the collection Given some audio books in the collection When I visit the list of audio books Then I see all audio books
Remark: All code shown in this section (the Cucumber feature, the step definition plus the helper file storage.rb) is available in the branch “04_list_audiobooks”. You can fetch it with
git checkout 04_list_audiobooks.
Remember that in your scenario you can write anything you like as long as each line begins with Given, When, Then or And. Thus you do not need to think about how you will implement the step later, instead you can focus on the requirement you are trying to express. This is why it is often better to write the scenario first and implement the step definitions later. The scenario above should be fairly self explaining.
You can execute
bundle exec cucmber to get some suggestions for the step definitions or write them from scratch. Let’s put the step definitions in
#encoding: utf-8 Given /^some audio books in the collection$/ do upload_fixtures backend_url('audiobooks'), $fixtures end When /^I visit the list of audio books$/ do visit ui_url '/index.html' end Then /^I see all audio books$/ do page.should have_content 'Coraline' page.should have_content 'Man In The Dark' page.should have_content 'Siddhartha' end
The Given step calls
upload_fixtures, a method that is implemented in
features/support/storage.rb and uses Storra (the persistence service used by the AngularJS UI) directly to insert a number of audio books from a file into the database. The code is not specific to Cucumber or Capybara, so I’m not going into detail here but you are invited to have a look at the code in
storage.rb if you are interested.
The When step: Parentheses for method calls are optional in ruby, that is why
visit ui_url '/index.html' is the same as
visit(ui_url('/index.html')) (I personally think the version without parentheses reads more fluently but that is a matter of taste).
ui_url is a little helper method from
env.rb that translates a path into the full URL of the application under test. Finally,
visit is a method from Capybara that navigates to the given URL. Thus,
visit ui_url '/index.html' just points the browser (PhantomJS in this case) to
http://localhost:8000/app/index.html, the page which displays the list of audio books.
The Then step uses Capybara’s
page object (a representation of the current page, that is, the current DOM as presented to the browser) to verify that three pieces of text are there. Let’s assume the step
Given some audio books in the collection would have inserted three audio books, namely “Coraline” by Neil Gaiman, “Man in the Dark” by Paul Auster and “Siddhartha” by Herman Hesse. Reading Capybara’s documentation you would probably expect the check to read as
page.has_content?('Coraline'). To spice things up a bit, we use RSpec’s ability to create a custom matcher for any predicate (any method ending with a “?”) on the fly, see RSpec Matcher docs. The RSpec library exploits Ruby’s meta-programming capabilities to create these matchers and as a result, we can write
page.should(have_content('Coraline')), which is what we did (except for the parentheses).
A Word About Test Data
If you execute the feature multiple times and then open http://localhost:8000/app/index.html in your browser, you will see that the list of audio books is littered with multiple copies of the test fixtures that we insert in the Given step. This can turn into a real problem later when we try to test features like adding or deleting audio books: The tests are not isolated because they operate on the same database. Imagine a scenario for adding an audio book, let’s say with the title “Foobar”. To see if that operation was successful we would probably check
page.should have_content 'Foobar' in the Then step after navigating to the list of audio books. Now this test might run fine a number of times. But what if the feature to add an audio book breaks in the application under test? Our Cucumber scenario might still be green as a cuke because there are a number of copies of “Foobar” in the database already. That’s the worst case for an automated test: a test that is green although the production code is broken. So we need to do something about that.
This problem is relevant for almost every application on the level of acceptance testing and there are at least two different solutions to this problem:
- Delete everything before or after each test. With everything I really mean the whole database. This is often the easiest option. Of course, this makes it necessary to have a dedicated environment to run your acceptance tests in CI (separate from a stage for manual testing, that is). But any non-trivial project should have that anyway.
- Separate tests by using some kind of unique data space for each test run. It depends on your domain model if this option is suitable. Quick example: Let’s say you have a customer object, which in turn can have one or more associated order objects and each order might have a few items. A user always only sees and interacts with her own orders. In such a setting you could just create a new user for each test (maybe with a uuid as her id) and you have your test isolation covered.
We will go with option 1 and ask Storra to delete the entire collection before each test by adding the following lines to
Before do delete_database backend_url('audiobooks') end
and this to
def delete_database(url) RestClient.delete url end
backend_urlis defined in
features/support/env.rband returns the URL for the collection resource managed by Storra.
More Tests for Showing the List of Audio Books
Note: The code for the following scenario is available per
git checkout 05_filter. This branch also contains the hooks to delete the database as discussed in the previous section.
If you have explored the audio book collection manager manually you might have noticed the text input with the label “Filter” next to it. If you start to type something there, only matching audio books will be shown and titles which do not match will be hidden. So, if you type “Cor”, “Coraline” will be shown but other titles will be omitted from the list.
To express this as a Cucumber scenario, we can add the following to
Scenario: Filter the list Given some audiobooks in the collection When I visit the list of audiobooks And I search for "Cor" Then I only see titles matching the search term When I remove the filter Then I see all audiobooks again
To implement the steps we can add this to
When /^I search for "(.*?)"$/ do |search_term| fill_in('filter', :with => search_term) @matching_titles = ['Coraline'] @not_matching_titles = ['Man In The Dark', 'Siddharta'] end When /^I remove the filter$/ do # funny, '' (empty string) does not work? fill_in('filter', :with => ' ') @matching_titles = @not_matching_titles = nil end Then /^I see all audiobooks(?: again)?$/ do page.should have_content 'Coraline' page.should have_content 'Man In The Dark' page.should have_content 'Siddhartha' end Then /^I only see titles matching the search term$/ do @matching_titles.each do |title| page.should have_content title end @not_matching_titles.each do |title| page.should have_no_content title end end
Let’s go through the step definitions one by one:
When I search for...: The first line (
fill_in('filter', :with => search_term)) uses Capybara’s API to type a value into the text box. The value is provided by a capturing group from the regex, so if we call this step with
When I search for "Foobar"“Foobar” would be entered into the text box. The next two lines set the instance variables
@not_matching_titles. This is a technique I use from time to time with Cucumber, that is, setting up expectations (or rather: expected values) in When steps which will be checked later in a Then step. The expected values set here are checked immediately in the next step. This might feel a bit like a hack because expectations are not the domain of the When step but of the Then step. However, it often provides a pragmatic way to use a Then step with different expectations (set up in different When steps). In this particular case there is definitely room for improvement because there is a mismatch between using a variable text for the the
fill_incall (the variable
search_term, taken from the capturing group) and the hard coded values for
Then I only see titles matching the search term: This step uses the expected values that have been set up in the When step (see above). For all titles matching the search term, we verify that the content is on the page. For the non matching titles, we verify that the content is not there.
When I remove the filter: This justs removes the input from the text box which should result in all audio books being shown.
- The step definition
Then /^I see all audiobooks(?: again)?$/ dois our existing step definition (
Then /^I see all audiobooks/ do), only the regex has been expanded with the optional non-capturing group
(?: again)?to make the Cucumber scenario more readable. If you copied the above into your existing
audiobooks.rb, make sure to delete the original step definition without the expanded regex to avoid ambigious steps.
Testing AJAX functionality with Capybara
Did this make our testing harder? More complicated? Did we need special code for testing the AJAX stuff? No! We did not even think about that until now. The reason is: Capybara is based on the assumption that in a modern web application, potentially everything might happen asynchronously. Whenever you verify that some content is there or some condition is met, Capybara by default waits for the content to appear or the condition to become true. The timeout for this is configurable, of course.
This is what I expect from a decent browser test abstraction nowadays: When writing acceptance tests, I don’t care if the application under test produces the expected content by navigating to a whole new page, does a partial DOM update via AJAX or a tiny imp inside my screen paints the content with a miniature brush. I just want to test if the expected content is there. And I don’t want to change my test code when the application under test changes the way it produces the content.
Capybara fulfills this expectation. A number of other web application test frameworks do not, especially the ones operating on a lower level, like Selenium. If your test code is littered with stuff like
waitForXyz or even the dreaded, arbitrary
sleep 2s, chances are you are using the wrong test framework.
Homework: More Features and Scenarios
We could now continue to write more features and scenarios for the audio book collection manager. The master branch indeed contains a few more, for example for adding an audio book or for displaying detailed information about a particular audio book. You can take a look at them with
git checkout master. If you have worked your way through this tutorial up to this point, probably neither the additional features nor their corresponding step definitions contain something fundamentally new to you.
You are invited to add more Cucumber features of your own, for example for deleting an audio book or for showing the cover picture of an audio book (the cover picture is automatically shown in the details page if the ASIN of the audio book has been provided and amazon.com has an image for this ASIN).
This completes our little excursion into the world of Cucumber and Capybara. I hope you enjoyed the ride. This tutorial only provides a little glimpse of what Cucumber and Capybara can do, of course it does not cover every aspect of the mentioned frameworks. I think it nonetheless shows that this combination is a quite powerful one and allows you to write your acceptance tests in an elegant and readable manner.
If you have questions or comments, please drop a line below.
A part of the setup code (especially the stuff with using either Poltergeist and PhantomJS or Selenium WebDriver and Firefox, controlled by an environment variable) was created by Michael Schuerig with whom I had the pleasure to work together in a former project. He also introduced me to Capybara and Poltergeist.