Detox vs. Appium – a comparison of React Native testing frameworks

Currently, there are especially two end-to-end testing frameworks which are interesting for React Native developers: Detox and Appium. During my internship at codecentric, I analyzed and compared both frameworks in detail, writing tests with both frameworks. In this article, I summarize the most important results. At the end of the article, you find links to the complete comparison and to my Github repository with the two React Native projects that have been used for the comparison, one containing the Detox and the other one the Appium framework. The linked document also contains developer guidelines on how to include the frameworks into your app and write your first test.

1. Facts about Detox

Detox uses grey-box-testing by monitoring the state of the app to tackle the problem of flaky tests. Flaky tests sometimes fail or pass without a change in the code. Grey-box-testing means that the tester has a partial understanding of the internal structure of the system under test. Monitoring the status of the app implies that Detox monitors the app in order to know when it’s idle. A test cannot continue to the next line if the app is busy executing the preceding test code. Detox is test-runner independent, meaning you can use any test-runner you want. The test suite is being run outside of the app and communicates with the app via a network using a websocket, while the Detox server acts as a proxy, as shown in the figure below. After the invocation of a method by the test runner, the Detox server invokes the corresponding native testing method from the EarlGrey Testing framework of iOS or from the Espresso Testing framework of Android.

2. Facts about Appium

Under the hood, Appium uses automation frameworks like for example XCUITest from Apple and UIAutomator from Google, which are wrapped by the Webdriver API. Appium is a webserver that exposes a REST API, as shown in the figure below. It receives connections from a client, listens for commands, executes those commands on a mobile device, and responds with an HTTP response representing the result of the command execution. The main difference between Appium and Detox is that Appium uses black-box-testing, meaning that it doesn’t monitor the internal state of the app. Black-box-testing is defined as a testing technique in which the functionality of the application is tested without looking at the internal code structure. The lack of monitoring of the state of the app might lead to flakiness.

3. Functional requirements

The tests that have been written included these functional requirements:

4. Comparison of the frameworks

Following you see the table with the results of the comparison, the definitions of the criteria can be found afterward. The weighting depends on the importance of the comparison criterion, 1 is low importance, 2 medium, and 3 high importance. In the calculation, the points given for the criterion will then be calculated *1, *2, or *3. As an example, the first criterion for Detox (costs) got 5 points and a weight of 1, so the calculation will be 5(points)*1(weight)=5.

Here are the definitions of the comparison criteria:

1. 1. Costs contain how much you have to pay to be able to use the framework.
   2. Setup time means, how long it takes to integrate the testing framework into an app and get a first example test running.
   3. Time to implement means, how long it takes to write runnable tests.
   4. The learning curve implies, how long it takes to be able to use the framework for writing and changing tests, which depends on the quality of the documentation, necessary preknowledge, if there is additional research required, and if there is support found, like e.g. in forums.
   5. Adaptability defines how quickly tests can be changed by the programmer.
   6. Extendibility defines how quickly tests can be expanded by the programmer.
   7. Flakiness means that the same tests may sometimes pass and sometimes fail even if there has not been a change of the code. This can happen for example due to network requests being made which have different response times. The less flaky the test is, the higher the grade will be.
   8. Overhead implies, how much code is needed to implement a certain test.
   9. The speed of the test runs concerns the time for a test run from start to finish.
   10. Coverage implies how many functionalities are included in the testing framework, like e.g. methods for finding UI elements and methods for changing a certain UI element. It also takes into account if it was possible to implement the tests completely or if parts of it couldn’t be implemented.
   11. Android and iOS compatibility means if the tests are able to run on Android and iOS.

For the detailed explanation about in which case are given how many points, have a look at the following table.

5. Conclusion

The comparison of the frameworks shows that Detox scores more points than Appium. Nevertheless, it has to be considered that the resulting ratings for the frameworks are very close to each other (112 vs. 99). The Webdriver.io driver and the Jasmine framework which have been used for writing the tests, are not the only driver and framework you can couple with Appium. There are other drivers and frameworks that can be used, too. This means that in the comparison between Detox and Appium you have to keep in mind that by using other drivers or frameworks, Appium could have been seen as better or worse in the final result.

In general, it can be said that both testing frameworks have their strengths and weaknesses. While Appium requires more time to get an overview of and might result in flakiness, everything was at least runnable on iOS and Android. Detox is easy to learn and not flaky, but sometimes only works on iOS or Android. Both frameworks make it possible to easily adapt and extend your tests and don’t cost anything. The complete comparison with the developer guidelines (the Appium guideline is in the appendix .6) can be found here:

Detox and Appium complete comparison

The two React Native projects with Detox and Appium can be downloaded here:

Detox and Appium projects