//

Behaviour Driven Development with Elasticsearch

24.2.2014 | 5 minutes of reading time

Elasticsearch has been riding on top of the hype for a while now, and I expect it to hit even harder with the release of 1.0 – We will continue to see a massive growth in various fields throughout the tech world, and even more use cases will be discovered and put to production in stunning speed.

While it’s all hot and fresh I want to urge every Developer to try to include proper craftmanship techniques in his daily work with Elasticsearch. We all strive to ensure great results without regression – In this post I want to talk about a behaviour driven approach when it comes to Elasticsearch, something we at codecentric have had tremendous success so far.

Let’s imagine you’re having an Elasticsearch cluster up and running, and you’re trying to improve your search results for a specific use case, maybe by using the fantastic function score query – would you feel save pushing that change into production? Are you sure all the queries your customers throw at you will be answered sufficiently? If your answer is “HELL NO!” then you know you have a problem.

It’s not unsolvable though: This can be approached by having a decent set of tests that will provide accurate safety against regressions and support agile development of new features: Acceptance Tests! We are able to execute our tasks fast by starting up a whole ES node with the NodeBuilder in the java API and with a JUnit Rule ( as described by Florian Hopf here ) :

1public class ElasticsearchTestNode extends ExternalResource {
2 
3    private Node node;
4    private Path dataDirectory;
5 
6    @Override
7    protected void before() throws Throwable {
8        try {
9            dataDirectory = Files.createTempDirectory("es-test", new FileAttribute[]{});
10        } catch (IOException ex) {
11            throw new IllegalStateException(ex);
12        }
13        ImmutableSettings.Builder elasticsearchSettings = ImmutableSettings.settingsBuilder()
14                .put("http.enabled", "false")
15                .put("path.data", dataDirectory.toString());
16 
17        node = NodeBuilder.nodeBuilder()
18                .local(true)
19                .settings(elasticsearchSettings.build())
20                .node();
21    }
22 
23    @Override
24    protected void after() {
25        node.close();
26        try {
27            FileUtils.deleteDirectory(dataDirectory.toFile());
28        } catch (IOException ex) {
29            throw new IllegalStateException(ex);
30        }
31    }
32 
33    public Client getClient() {
34        return node.client();
35    }
36}
37

So let’s write our first test for it! Let’s create an Index, index a document and retrieve it again – in only a couple of clean lines!

1public class NodeCreationTest {
2 
3    @Rule
4    public ElasticsearchTestNode testNode = new ElasticsearchTestNode();
5 
6    @Test
7    public void indexAndGet() throws IOException {
8        testNode.getClient().prepareIndex("myindex", "document", "1")
9                .setSource(jsonBuilder().startObject().field("test", "123").endObject())
10                .execute()
11                .actionGet();
12 
13        GetResponse response = testNode.getClient().prepareGet("myindex", "document", "1").execute().actionGet();
14        assertThat((String) response.getSource().get("test"),equalTo("123"));
15    }
16}
17

Run the test and we’ll see in the console log that the node boots up, actually handles the request and shuts down gracefully, awesome!

So we could be done right here and commence happy TDD – but let’s crank it up a notch and

  1. add JBehave to our stack
  2. create a custom mapping within our code that we want to test

Let’s imagine we are building the next Twitter application and after careful consideration we come up with the follwing story:

 Scenario:  Basic Tweet retrieval

Given A user Chris submitted a tweet I luv tweeting

When We list all tweets for the user Chris

Then A tweet with the text I luv tweeting will be found

To introduce JBehave I can really recommend the fantastic JUnitReportingRunner from my workmates, grab it from Maven Central and create a Story Class that wires our story with some sane defaults. For further explanation check out Andreas’ post here .

1@RunWith(JUnitReportingRunner.class)
2public class TwitterStories extends JUnitStories {
3 
4    private final CrossReference xref = new CrossReference();
5 
6    public TwitterStories() {
7        super();
8    }
9 
10    @Override
11    protected List storyPaths() {
12        String codeLocation = codeLocationFromClass(this.getClass()).getFile();
13        List paths =  new StoryFinder().findPaths(codeLocation, asList("Tweet.story"
14        ), asList(""),"");
15        return paths;
16    }
17 
18    @Override
19    public InjectableStepsFactory stepsFactory() {
20        return new InstanceStepsFactory(configuration(), new TweetRetrievalTest());
21    }
22 
23    @Override
24    public Configuration configuration() {
25        Class<? extends Embeddable> embeddableClass = this.getClass();
26        Properties viewResources = new Properties();
27        viewResources.put("decorateNonHtml", "true");
28        viewResources.put("reports", "ftl/jbehave-reports-with-totals.ftl");
29        // Start from default ParameterConverters instance
30        ParameterConverters parameterConverters = new ParameterConverters();
31        // factory to allow parameter conversion and loading from external resources (used by StoryParser too)
32        ExamplesTableFactory examplesTableFactory = new ExamplesTableFactory(new LocalizedKeywords(), new LoadFromClasspath(embeddableClass), parameterConverters);
33        // add custom converters
34        parameterConverters.addConverters(new ParameterConverters.DateConverter(new SimpleDateFormat("yyyy-MM-dd")),
35                new ParameterConverters.ExamplesTableConverter(examplesTableFactory));
36        return new MostUsefulConfiguration()
37                .useStoryLoader(new LoadFromClasspath(embeddableClass))
38                .useStoryParser(new RegexStoryParser(examplesTableFactory))
39                .useStoryReporterBuilder(new StoryReporterBuilder()
40                        .withCodeLocation(CodeLocations.codeLocationFromClass(embeddableClass))
41                        .withViewResources(viewResources)
42                        .withFormats(STATS)
43                        .withFailureTrace(true)
44                        .withFailureTraceCompression(true)
45                        .withCrossReference(xref))
46                .useParameterConverters(parameterConverters)
47                        // use '%' instead of '$' to identify parameters
48                .useStepPatternParser(new RegexPrefixCapturingPatternParser(
49                        "$"))
50                .useStepMonitor(xref.getStepMonitor());
51    }
52

Here you can see we’re loading our previous story called “Tweet.story” and a test called “TweetRetrievalTest”. This test maps our story to actual executable code and takes care of the Elasticsearch node bootup:

1public class TweetRetrievalTest{
2 
3    public ElasticsearchTestNode testNode = new ElasticsearchTestNode();
4 
5    @BeforeStory
6    public void setUp() throws Throwable {
7        testNode.before();
8 
9        testNode.getClient().admin().indices().create(new CreateIndexRequest("twitter")).actionGet();
10        testNode.getClient().admin().indices()
11                .preparePutMapping("twitter")
12                .setType("tweets")
13                .setSource(mapping())
14                .execute().actionGet();
15    }
16 
17    @AfterStory
18    public void after(){
19        testNode.getClient().admin().indices().prepareGetFieldMappings("twitter").execute().actionGet();
20        testNode.after();
21    }
22 
23    private SearchResponse response;
24 
25    @Given("A user $user submitted a tweet $tweet")
26    public void userTweets(@Named("tweet") String tweet , @Named("user") String user) throws IOException {
27        testNode.getClient().prepareIndex("twitter", "tweets", "1")
28                .setSource(jsonBuilder()
29                        .startObject()
30                        .field("user", user)
31                        .field("message", tweet)
32                        .endObject())
33                .execute()
34                .actionGet();
35    }
36 
37    @When("We list all tweets for the user $user")
38    public void retreiveTweetsForUser(@Named("user") String user) {
39        response = testNode.getClient().prepareSearch("twitter").
40                setTypes("tweets")
41                .setQuery(QueryBuilders.termQuery("user", user))
42                .setFrom(0).setSize(60).setExplain(true)
43                .execute()
44                .actionGet();
45 
46    }
47 
48    @Then("A tweet with the text $text will be found")
49    public void expectTweet(@Named("tweet") String tweet)  {
50        for (SearchHit hitFields : response.getHits().getHits()) {
51            if(hitFields.field("tweet").getValue().equals(tweet)) {
52                return;
53            }
54        }
55        fail("expected Tweet " + tweet + "not found");
56    }
57 
58    /**
59     * Overriding mapping
60     */
61    public XContentBuilder mapping() throws Exception {
62        XContentBuilder xbMapping =
63                jsonBuilder()
64                        .startObject()
65                        .startObject("tweet")
66                        .startObject("properties")
67                        .startObject("source")
68                        .field("type", "string")
69                        .endObject()
70                        .startObject("user")
71                        .field("type", "string")
72                        .endObject()
73                        .startObject("message")
74                        .field("type", "string")
75                        .endObject()
76                        .endObject()
77                        .endObject()
78                        .endObject();
79        return xbMapping;
80    }
81 
82}
83

As a side-note, see how easy it is to inject a custom mapping into the whole setup! Feel free to experiment with it:

  • provision your Elasticsearch production nodes with a custom mapping from a .yml file
  • make use of the API: boost values, give it a custom scoring, try out different filters or analyzers
  • run the test and know that it’s going to work. Awesome!

Happy testing folks! You can grab the code for this small example on our company github account here .

Google

share post

Likes

0

//

More articles in this subject area\n

Discover exciting further topics and let the codecentric world inspire you.

//

Gemeinsam bessere Projekte umsetzen

Wir helfen Deinem Unternehmen

Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.

Hilf uns, noch besser zu werden.

Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.