Log Management for Spring Boot Applications with Logstash, Elasticsearch and Kibana

5 Comments

In this blog post you will get a brief overview on how to quickly setup a Log Management Solution with the ELK Stack (Elasticsearch-Logstash-Kibana) for Spring Boot based Microservices. I will show you two ways how you can parse your application logs and transport it to the Elasticsearch instance. Basically you can replace Spring Boot with any other application framework which uses Logback, Log4J or any other known Java logging framework. So this is also interesting for people who are not using Spring Boot.

This post doesn’t contain detailed insights about the used technologies, but you will find a lot of informations in the web about it. So, before we start have a short look at Elasticsearch, Logstash and Kibana. A good starting point is the website of elasticsearch.org with a lot of resources and interesting webinars. Also my codecentric colleagues have already blogged about some topics in this area. The reason why I have selected Spring Boot for this Demo is, that we actually using it in some projects and I believe it will help to make the next big step in the area of Enterprise Java Architectures. With this Micrservice based approach there will be a lot more logfiles you have to monitor, so a solution is definitely needed here.

First of all, clone the example repository into your workspace and go into the root of this directory.

git clone http://github.com/denschu/elk-example
cd elk-example

The Spring Boot example application is a small batch job which is located in the directory “loggging-example-batch”. Start the JVM with the following commands:

cd loggging-example-batch/
mvn spring-boot:run

Take a look inside “/tmp/server.log”. There you will find some log statements like those:

2014-10-10 17:21:10.358  INFO 11871 --- [           main] .t.TomcatEmbeddedServletContainerFactory : Server initialized with port: 8090
2014-10-10 17:21:10.591  INFO 11871 --- [           main] o.apache.catalina.core.StandardService   : Starting service Tomcat
2014-10-10 17:21:10.592  INFO 11871 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/7.0.55
2014-10-10 17:21:10.766  INFO 11871 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2014-10-10 17:21:10.766  INFO 11871 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 2901 ms
2014-10-10 17:21:11.089  INFO 11322 [main] --- s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8090/http

The question is now: How can we transport and parse those log statements? So let’s setup the ELK Stack and try out two methods on how to parse and transport these logfiles with Logstash.

Preparation

Elasticsearch

Open a new shell and download the Elasticsearch archive. Afterwards you can directly start the instance.

curl -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.tar.gz
tar zxvf elasticsearch-1.1.1.tar.gz
./elasticsearch-1.1.1/bin/elasticsearch

Kibana

In another shell download Kibana and extract the contents of the archive. It contains the JavaScript-based Dashboard which you can simply serve with every HTTP Server. In this example we use a lightweight Python-based HTTP server.

curl -O https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz
tar zxvf kibana-3.1.0.tar.gz
cd kibana-3.1.0/
python -m SimpleHTTPServer 8087

Open the preconfigured Logstash Dashboard in Kibana and check if it successfully connect to your running Elasticsearch Server. Per default it uses the URL “http://localhost:9200” (see config.js to modify it).

http://localhost:8087/index.html#/dashboard/file/logstash.json

Logstash Agent

To collect the logfiles and transport them to our log server we use Logstash. Open a new shell and execute this:

curl -O https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
tar zxvf logstash-1.4.2.tar.gz

Method 1: Parse unstructured logfiles with Grok

The mostly used method to parse the logs is to create a Grok Filter which is able to extract the relevant data from the log statement. I have created a Grok Filter for the standard configuration of Logback that is used actually in Spring Boot.

input {
  stdin {}
  file {
    path =>  [ "/tmp/server.log" ]
  }
}
filter {
   multiline {
      pattern => "^(%{TIMESTAMP_ISO8601})"
      negate => true
      what => "previous"
   }
   grok {
      # Do multiline matching with (?m) as the above mutliline filter may add newlines to the log messages.
      match => [ "message", "(?m)^%{TIMESTAMP_ISO8601:logtime}%{SPACE}%{LOGLEVEL:loglevel} %{SPACE}%{NUMBER:pid}%{SPACE}%{SYSLOG5424SD:threadname}%{SPACE}---%{SPACE}%{JAVACLASSSHORT:classname}%{SPACE}:%{SPACE}%{GREEDYDATA:logmessage}" ]
   }
}
output {
  elasticsearch { host => "localhost" }
}

To be able to parse the Java class name correctly I created an additional pattern (JAVACLASSSHORT). Add it to the agent directory of Logstash:

cp custompatterns logstash-1.4.2/patterns/

Run Logstash Agent

Start the Logstash agent with the Spring Boot log configuration from above. It’s already placed in logstash-spring-boot.conf.

./logstash-1.4.2/bin/logstash agent -v -f logstash-spring-boot.conf

Now start a job using this cURL command:

curl --data 'jobParameters=pathToFile=classpath:partner-import.csv' localhost:8090/batch/operations/jobs/flatfileJob

Open the preconfigured Logstash Dashboard in Kibana again and you will see upcoming logstatements

http://localhost:8087/index.html#/dashboard/file/logstash.json

kibana

Method 2: Use JSON Logback Encoder

One big disadvantage of Method 1 is that it’s sometimes not so easy to create a fully working Grok Pattern that is able to parse the unstructured logfiles. The Spring Boot default log format is one of the better ones, because it uses fixed columns. An alternative is to directly create the log statements in JSON Format. To achieve that you have to add the following artifact (It’s already inlcuded in the sample application!) to the pom.xml.

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>2.5</version>
</dependency>

… and add this special Logstash Encoder to the Logback Configuration File “logback.xml” (It’s also already inlcuded in the sample application!)

<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>

The new Logstash Configuration (logstash-json.conf) is now much smaller and easier to read:

input {
  file {
    path =>  [ "/tmp/server.log.json" ]
    codec =>   json {
      charset => "UTF-8"
    }
  }
}
 
output {
  elasticsearch { host => "localhost" }
}

Alternative Log Shippers

The Logstash Agent runs with a memory footprint (up to 1GB) that is not so suitable for small servers (e.g. EC2 Micro Instances). For our demo here it doesn’t matter, but especially in Microservice environments it is recommended to switch to another Log Shipper, e.g. the Logstash Forwarder (aka Lumberjack). For more Informations about it please refer to this link. Btw. for the JS Guys, there is also a Node.JS implementation of Logstash available.

To summarize it up, the ELK Stack (Elasticsearch-Logstash-Kibana) is a good combination to setup a complete Log Management Solution only with Open Source Technologies. For larger environments with a high amount of logs it’s maybe useful to add an additional transport like Redis to decouple the componentes (Log Server, Log Shipper) and make it more reliable. In the next time I will post about some other topics in the area of Microservices. So stay tuned and give some feedback 🙂

Kommentare

  • Tom

    Very good tutorial, simple and to the point, I liked the style

  • nick

    thanks for great job, please keep posting!

  • Michael Ressler

    I’ve been using AWS ElasticBeanstalk and I haven’t found a nice way to also start up a logstash log shipping agent on my application servers. To overcome this, I’ve used log4j’s SocketAppender – which logstash seems to handle very well. Specifically, I wrap the SocketAppender in an AsyncAppender for performance reasons, but everything is working well so far.

    Would you consider the SocketAppender an appropriate alternative log shipper? Anything that I might be missing?

    Thanks for the article!

  • Damian Calabresi

    2. April 2015 von Damian Calabresi

    Thanks! Installing an ELK stack is not an easy task, I could make it work following this post.

  • Dan James

    17. April 2015 von Dan James

    Thanks, very interesting and informative
    I’ve been using log4j and lately started using Stackify’s log management (http://stackify.com). They have an appender to log4j so getting it started was really simple and easy. While it is not free, it is very low cost and the benefit of having integrated log and error management together (so for example I can get in one click all the logs thrown while a specific error was reported) . We love it here and use it on a daily basis. Actually I find looking at their real-time tailing addicting 🙂

Comment

Your email address will not be published. Required fields are marked *