So in a average IT project something like acceptance test comes up sooner or later. Which is a good thing because we want to be sure about the functionality that is provided by the software actually works. So we make acceptance tests and show the results on a dashboard. Most people agree that acceptance tests are critical in providing resilient software. But people also tend to agree that acceptance tests are expensive. They take some time to run (10+ min. in bigger projects) and they take extra time to create. This is time not spent on the actual creation of functionality. So we need them, but we need to be careful.
A whole different problem is software not providing the expected behavior or functionality. This is something Acceptance Test Driven Developement(ATDD) tries to solve. ATDD is something that originated from the test driven movement, although Kent Beck in 2003 thought it was impractical. ATDD still gained momentum, and it has its benefits. By defining the tests before actually building the software ATDD provides more clarity about what needs to be created.
Other benefits of ATDD are:
- You know when and if functionality is provided without manual testing.
- Forces careful thinking about the functionality.
And ofcourse there is a drawback:
- You need to invest more time before creating the functionality.
Maybe there are more drawbacks of ATDD. I know that Acceptance tests themselves have some drawbacks. It makes sense to write your acceptance tests first before starting the actual coding. Maybe not for small and simple things, but definitely for the large and complex.
Implementing the code for running the test descriptions should take as little time as possible. We want to implement this before the functionality so we see a red bar. For this we use tools that translate these descriptions. The descriptions should be readable for the tool, but we would like to be as free as possible. Often the syntax used for these descriptions are sentences starting with Given, When and Then which orginates from Behavior Driven Development (BDD) approach invented by Dan North and Chris Matts.
Next to being free in our way we write our tests a test framework should support us as much as possible in writing tests quickly. Which means the following according to me:
- Not a lot of coding needed before a test runs.
- IDE should support my preferred test description.
- I can generate some code based on the test description.
- The tool should run the tests in a convenient way.
- Not a lot of boilerplate code needed to setup.
- I can get support from a community.
- I can see the internals and improve on it (Open source).
- I can integrate the tool in a build pipeline.
- The tool provides libaries or integration with libraries that can test a certain UI , API or data
This is quite a list of capabilites for a tool. A small team, including me, of codecentric wanted know if there are any frameworks available that allow us to write tests faster and thus prevent headaches. The following acceptance test frameworks score highly on the capabilities I mentioned.
- Robot Framework
Although we tried to look at all the acceptance test frameworks briefly we probably did miss some. Cucumber is part of the list and I already use it a lot. I am more curious about the other frameworks which maybe allow me to write tests faster.
Robot Framework looked very promising and I studied it in more detail. Concordion, Gauge and Jbehave are very nice frameworks but we looked at them only briefly because of time constraints.