I’ve seen a lot of MemoryLeaks caused by wrong usage patterns of ThreadLocal variables. As I had this problem in one of our own applications yesterday, I want to blog about this case.

ThreadLocal variables are used in Java to bind a variable to the current Thread - each thread will have its own, independend instance of that variable. These kind of variables are normally used to hold status information like the user, that are needed in the entire application and you don’t want to have in every method signature.

The lifecyle of ThreadLocal variables is directly interconnected to the lifecyle of its corespondig thread. If the thread is terminated and collected by the Garbage Collector, its coresponding ThreadLocal variables will also be good candidates for Garbage Collcetion.

Memory problems will mainly occur if ThreadLocal variables are used inside of Java EE applications running in an application server.  Application Servers manage threads by using thread pools to safe resources and boost performance. (See configuration of Tomcat 6 Http Conncector as an example). If for example a HttpServletRequest is send to the ServletEngine of the Application Servers, a free worker thread is taken out of the pool and is associated with the processing of the servlet’s application logic. If the servlet or its called Java classes are using ThreadLocal variables, these variables are associated with the current worker thread. If the servlet is finished and the response is sent to the client, the associated thread is put back to the thread pool, so that it can handle another request. That means that the Thread object and its associated ThreadLocal variables are not garbage collected, as the thread object is still alive.

Depending on the number of threads in the pool (> 100 threads are normal in productive environements) and the size of the object in the ThreadLocal variable, critical memory problems can occur. If for example 200 threads are configured for the thread pool and the ThreadLocal variables are 5MB big, this could result in 1 GB of heaps space occupied by just these ThreadLocal variables. This will lead to an GC overhead and can crash the JVM with an OutOfMemoryError.

In our concrete case the servers had to be restartet every day to avoid OutOfMemoryError problems. To analyse the problem we took a heapdump during runtime  (see our recent blog entry how to do this). I did the analysis of the dump with Eclipse MAT which showed very quickly that we had a ThreadLocal memory problem.

The above screenshot shows the Dominator Tree of the dump. I’ve marked 6 Threads each occupying about 14MB of the heap.

The next screenshot shows the details of one of these threads. The high memory consumption is the result of a ThreadLocal variable that references a DOMParser and its parsed document with a total size of 14MB. Because MAT can also show the content of the objects, we could easily see that the parsers were caused by our WebServices. The WebService classes were generated by JBoss WS. Consulting Google showed that this was because of a Bug in jbossws version 1.2.0, which was fixed in 1.2.1: “DOMUtils doesn’t clear thread locals”. So we had to patch our JBoss AppServer with the latest jbossws release and the problem was gone.

This exampe shows that you have to be careful if you use ThreadLocal inside an application server. It is essential to take care that the ThreadLocal variables are dereferenced if they are not used anymore. We normally use ServletFilters to do this job.

For some month we are using Scrum in our internally development projects and at some customer sites. To deepen out knowledge 13 codecentric team members and 2 colleagues of our customer OEV Online have attended the ScumMaster Certification Training at Xebia in the Netherlands. The training was held by Jeff Sutherland one of the master minds behind Scrum. The slides of the training can be downloaded at Jeffs website.

The training itself was on of the best I’ve ever attended. Jeff is an excellent coach and story teller. His opinion and methods to software projects are characterized by great simplicity and efficiency and are based on a number of proven best practices which were combined in Scrum.

There are already a lot of good books and articles about Scrum, so this blog entry summarizes my highlights and insights of the training.

What is Scrum?

Scrum is not purely a development process, but a method to organize any kind of projects and tasks. Jeff explained us that he uses Scrum to organize his weekend activities together with his wife on a ScrumBoard in his kitchen. Examples of other people doing such kind of Scrum sessions at home can we found at Flickr. At codecentric we are using Scrum to organize our sales and marketing activities in short 2 week sprints. The velocity and transparency has improved a lot since we introduce scrum for that kind of work. I will personally put a ScrumBoard in my office so that I can plan my tasks in that way.

Jeff recommends to use eXtreme Programming in addition to Scrum for an agile development method – both together would provide maximum value for projects. Especially the XP practices like Continuous Integration, Test-Driven-Development and Coding Standards were recommended by Jeff as a proven extension to Scrum.  We are already using XP successfully in all of our projects – Jeffs training approved that we are on the right way and we should extend our agile afforts.

The Nokia Test

Jeff explained that may teams are saying that they use Scrum, but in many cases are just using parts of the whole story. It is essential to use Scrum in a complete manner and not only parts (like Daily Scrum) to see “the lights blinking” as Jeff said.  As an example he used Nokia, which has introduced Scrum in most parts of the companies software development projects. They developed the “Nokia Test” to measure if projects are using Scrum correctly. The test includes 8 questions which all team members of a Scrum team have to answer. For each answer you can get between 1-10 points – so a maximum of 60 points is possible. If you are using Scrum effectively, you should have an average of 6 points per question or a minimum of 48 points in total. The test showed me that it is hard work to introduce Scrum in all of our teams and the test provides us starting points for improvement. We are currently evaluating the status of Scrum at codecentric by doing the Nokia test with all of our development teams.

Velocity

Scrum measures the productivity of the team by introducing the velocity of a sprint. During the course I understood how import and useful this metric is. The Velocity is calculated by the sum of Story Points that have been finished by the team or team members during one sprint. After some sprints you should get a stable velocity for a team that helps you estimate the duration of the whole project and gives you a great metric to plan a sprint. It is important to find a velocity that will not over-utilize your team – this velocity is called “sustainable pace”. The velocity can also lead to conflicts as it transparently shows the productivity of each team member – transparency can hurt sometimes. It was also very intersting to see comparisons of projects that are using Scrum and those that are using the classical waterfall model. Based on a study of such projects, Jeff said that Scrum projects have been at least 7 times more productive! So if you have wondered why all the big internet companies like Google, MySpace and Yahoo! are using Scrum, these numbers will give you the answer.

Emergency Procedure

Jeff has been a jet pilot at the U.S. Air Force, which is the reason why he cares a lot about an emergency procedure for troubled projects. In a jet you will be a “smoking hole in the ground” if you don’t have an emergency procedure and the same is true for software projects. If you recognize during a sprint that you are not able to finish all the tasks in time, you should start the emergency procedure. It is important to do this as early as possible, which mean before halftime of the sprint. Otherwise you can not escape being a whole in the ground. Jeff’s emergency procedure is:

1. Make something different.
2. Get someone else.
3. Do less.
4. Stop the sprint.

The first step means that you search for alternative ways of solving your problem. Jeff said that there are often easier ways to achieve the same business value and you have to try hard to find them with your product owner. The second step is to get experts helping you achieve your goals. It is often much easier for an domain expert to solve a problem than for the team members – e.g. tuning a database is easier for an DB admin as for a developer. The third step is to throw out backlog items if possible. At the end it is important to deliver a working software that delivers business value. If step 1-3 are not helping you to finish the sprint successfully you should stop the sprint. Otherwise your will waste your customers money.

Money for Nothing – Change for Free

According to Dire Straits song Money for nothing Jeff explained a new approach to fix price projects: Money for Nothing – Change for Free. What does that mean? It means that you make an agreement with your customer that they can finish the project after every Sprint if they are satisfied with the given functionality of the software. Jeff said that in many projects a reduced functionality brings almost the same business value as the whole functionality. If you finish the project earlier your customer saves a lot of money and get the same business value as if he had payed for the whole project in the waterfall model. You can fix in the contract that you get some of the saved money: money for nothing and a classical Win-Win situation. “Change for Free” means that a customer can change the requirements at any time of the project. Therefore he has to throw out another requirement with the same story points that is prioritized less. As Jeff said, more than 60% of the requirements will be changed during a regular software project and that approach is the only approach to get real customer satisfaction. In his point of view classical Change Management is a process that prevents projects of delivering what customers want. I think that Jeff’s statement is hard but that he is right and we really have to change our mind how to deal with changes.

Scalability and distributed teams

Jeff provided examples of projects using Scrum with distributed and big teams. I personally thought that Scrum works very well with small and local teams and not so well with big or distributed teams. Jeff imposingly showed us statistics of teams that scaled perfectly, even when they were distributed to Russia or India. Scrum scales up to as many teams as you need by using virtual teams and so called Scrum of Scrums. You can use distributed Scrum an approach were teams work together locally for the first sprints and than distribute to different locations. A proportion of 50:50 (local:distributed) seems to be very efficient for that kind of projects. Xebia showed real statistic of a big project the did with Xebia India which showed no loss in velocity when the team was distributed. An interesting point was that they also measured the quality of the project in terms of bugs/lines of code. After distributing the team to India the quality was stable. It shows that we are on the right way, as we will provide nearshoring services with our branch in Doboj, Bosnien und Herzegowina based on Distributed Scrum.

Beside the good training we also staid in a beautiful place near a Dutch canal, where we could discuss the learned practices during the breaks in a nice atmosphere.

Some of our colleagues and customers will attend the next course with Jeff in November and we will definitely try to get more customers doing Scrum with us.

Peter Roßbach, core developer at the Tomcat project, held a talk at RheinJUG in Düsseldorf on Thursday. It was very delightful mixture on Tomcat architecture, best practices, Open Source community as well as current and future developments, rather than a dry technical talk.

Tuning Tomcat Performance

Regarding performance, he mentioned that the main negative factor is to use development mode for production environments. Well have a look at a few parameters and which do affect Tomcat performance.

As of tomcat 6 there is the improved protocol handler available:

<Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol"
 connectionTimeout="20000" />

additionally Java NIO powers a new improved file download option. Servlets just have to set sendfile request parameters and not output any more data. That file will then be pushed by tomcat to the user:

 org.apache.tomcat.sendfile.filename: Canonical filename of the file which will be sent as a String
 org.apache.tomcat.sendfile.start: Start offset as a Long
 org.apache.tomcat.sendfile.end: End offset as a Long

production servers should not deploy automatically:

<Host name="localhost" autoDeploy="false" deployOnStartup="false" deployXML="false">

The Jasper JSP compiler should be tuned as well:

<servlet>
  <servlet-name>jsp</servlet-name>
  <servlet-class>org.apache.jasper.servlet.JspServlet</servlet-class>
  <init-param>
<param-name>development</param-name>
<param-value>false</param-value>
  </init-param>
  <init-param>
<param-name>genStringAsCharArray</param-name>
<param-value>true</param-value>
  </init-param>
  <init-param>
<param-name>trimSpaces</param-name>
<param-value>true</param-value>
  </init-param>
</servlet>

Using mod_jk for loadbalancing

As a second main topic of his talk, he presented the module “mod_jk”, which in connection with a Apache Httpd and the Tomcat AJP protocol enables distributing java applications on multiple Tomcat servers.

He explained the different loadbalancing setups and that it might make even sense to user a Tomcat server per application, as this allows better memory management and improves overall high-availibility.

Tomcat Configuration:

<Connector port="8009" protocol="AJP/1.3"/>
<Engine name="Catalina" defaultHost="localhost" jvmRoute="node01">

mod_jk Configuration inside Httpd:

<IfModule !mod_jk.c>
  LoadModule jk_module "modules/mod_jk.so"
</IfModule>
JkShmFile "logs/mod_jk.shm"
 
JKWorkerProperty worker.list=loadbalancer
JKWorkerProperty worker.node01.port=8009
JKWorkerProperty worker.node01.host=localhost
JKWorkerProperty worker.node01.type=ajp13
JKWorkerProperty worker.loadbalancer.type=lb
JKWorkerProperty worker.loadbalancer.connection_pool_minsize=0
JKWorkerProperty worker.loadbalancer.connect_timeout=30000
JKWorkerProperty worker.loadbalancer.prepost_timeout=10000
JKWorkerProperty worker.loadbalancer.balance_workers=node01
JKWorkerProperty worker.loadbalancer.method=Request
JKWorkerProperty worker.loadbalancer.retries=2
JKWorkerProperty worker.loadbalancer.recovery_options=7
JKMount /myapps* loadbalancer

Further documentation on mod_jk worker can be found here:

http://tomcat.apache.org/connectors-doc/reference/workers.html

In total a very interesting talk, with lots of stuff to learn from, let it be the Tomcat project or the Tomcat architecture, or how Open Source “lives” and what one as developer or architect should consider when developing a Java web application.

Last weekend we went to Center Parcs in Port Zelande. That was the first company event at the sea. Not only employees of codecentric, but also their partners and children were invited to be part of the tour.

Friday evening we didn’t have much luck with the weather which didn’t stopped us from having a good party. After a battle with the electrical beer dispensing system, we had a lot of beer and our barbecue burned till the early morning.

The next day, we all had to work a little bit again: 10 km GPS Tour was not a thing for everyone, but divided into four teams and encouraged with the shiny day, wonderful seaside of the North Sea and autumn air, we went through the beautiful places of nature reserve of the peninsula. After a couple of hours we all met at the beach to dance, eat and drink into the night.

Sunday afternoon the codecentric crew started again, now direction Germany.

At that weekend you could see a lot of beer, fire and music as well as a lot of relaxed people on the small beach of Port Zelande – simply a good mach.

On the last Friday of every month codecentric hosts its “Fridaymeeting”. It’s a good opportunity to spend time from everybodys 20%-time contingent for professional development and discuss news in technology, products or other interesting topics. Every now and then we also invite companies to present their newest products.

Last Friday Progress Software was our guest; a company less known by their name but rather by their products, like the Java Messaging / Enterprise Service Bus “Sonic MQ/ESB“, or the JDBC/ODBC driver “DataDirect“. Recently Progress was in the news because of the acquisitions of IONA and Mindreef. With these two new companies Progress extends its broad SOA product portfolio, of which Eric Schaumlöffel presented the products “Actional” and “DataXtend SI“.

Progress Actional

Progress Actional provides runtime monitoring and security capabilities in a SOA with the option to actively intervene with the message flow. To achieve that, Actional instruments the participating services on application and protocol stack level (HTTP/S, IIOP, RMI, JMS, TCP) while keeping the runtime overhead low.

Without further configuration load and direction of the communication channels in the surrounding of instrumented servers can be monitored. With telling Actional where to find specific, business relevant, properties inside the message, those messages can be marked with a unique tracer. So incoming and outgoing messages can be related – also when using different protocols – and assigned to an overarching business process. This global business process can be drilled down to method level with the Actional UI. Additionally policies can be enforced, so that for example only certain systems may communicate with each other.

In Essence, Actional provides on SOA level what dynatrace is able to do for Java and .NET applications with its PurePath technology, although there certainly are overlaps in the areas in which these two products can be applied. It might feel a little strange to implement security relevant aspects of your SOA with a monitoring tool, but it makes sense if you turn this around: It’s rather so that Actional is designed and positioned as product for runtime SOA Governance. All technical necessities that are present for implementing such security relevant functions, produce the monitoring as a by-product.

Progress DataXtend SI

Progress DataXtend SI helps with the integration of services into a SOA. A reoccurring error that is made during the planning of a SOA is to concentrate on the technical connection of the services and protocols to a common communication system. This saves the costly n:m integration effort of all systems, but leaves the problem how theses systems communicate unsolved. Just because I can call and reach about any person on this planet with my mobile, it does not mean that I can communicate effectively with him or her, because we are not speaking the same language. In that sense, DataXtend is the Merriam-Webster of Esperanto, it provides for the usage of a Canonical Data Model and abilities to map it to the (legacy) services in the SOA. The modelling happens in the well known Eclipse IDE, enriched with specific plug-ins. At runtime, the data transformation can either be deployed as a stand-alone service in the SOA, or as adapter between the service and the ESB.

DataXtend can handle huge models like the Shared Information and Data (SID) Model from the TeleManagement Forum and offers all options and details, you would expect from a model mapping tool. In addition to the pure modelling you can also create validation rules, that can also be tested against the mapping. A rule could be “If incoming message is a relocation, then two addresses need to be present”. It would be interesting to know, if and how you could do that automatically in a continuous integration environment.

With the possible deployment of the DataXtend SI Runtime as Mapping und Validation Service Engine the integration into a JBI container works without problems. A welcome feature for that is that the SI Engine can be generated with all necessary mappings directly from the SI Designer.

DataXtend SI has to stand the comparison to heavy-weight products like SAP NetWeaver Master Data Management, IBM InfoSphere Master Data Management Server, or AquaLogic Data Services Platform. At first glance DataXtend is en par with the modelling possibilities, but is more flexible and lightweight at deployment. The approach of Master Data Management is more holistic in nature and aims at encapsulating all data in a SOA, to prevent multiple and different copies. While this is an desirable goal, DataXtends makes it possible to achieve this goal incrementally.

JavaRebel

As a last point on the agenda, Carsten Mjartan presented JavaRebel. With this tool Java’s hot deployment capabilities are considerably improved. Theoretically this capability exists since JDK 1.4 (in IDEs like Visual Age for Java even longer; for Smalltalk much longer), but are constrained to changes, that do not impact the class signature. With JavaRebel, which hooks into the JVM as -javaagent, you save a good share of server redeploys and restarts, according to Carstens experience. JavaRebel monitores the generated class files in your IDE and propagates the changes to the runtime environment, while the application state is completly kept. Changing EJBs and test the new logic with a simple refresh of the browser? Not a problem any more!