During the second act of this series we explained Java Memory Leaks . We elaborated how Memory Leaks can occur in Java and how they differ from traditional Memory Leaks. Todays post is going to explain how to have a closer look at all the objects residing in the Java heap. This includes references between the objects, which are required to identify memory leaks. The most accessible tool to approach analyzing memory issues are so called Java heapdumps, which can be created automatically in the event of an OutOfMemoryError by passing the JVM command line parameter
-XX:+HeapDumpOnOutOfMemoryError. In this case the JVM will write the dump right before it exits. Analyzing these kind of dumps is also called “Post mortem”, as the JVM is already dead. But of course one can also create dumps manually at any point during runtime. More on that later.
While we take on heapdumps in this post, we will see other alternatives for hunting lost memory during the next act.
A heapdump is a textual or binary representation of the Java heap which is usually written to a file. Various tools can reconstruct the objects and their references from the information stored in the heap dump file. The only information missing is the heap area the object was residing in. This is quite sad, as we just learned from act 3 the importance and meaning of the heap areas. However, this information is not required to locate Java Memory Leaks.
Wanted: Dead or alive
Heapdumps can contain to kinds of objects:
- Alive objects only (those are objects, which can be reached via a GC root reference)
- All objects (including the no longer referenced, but not yet garbage collected objects)
Because the live objects can be easily determined from various VM internal mechanisms, a heapdump containing only live objects can be created acceptable fast. Creating a full heap dump with dead objects as well takes much longer and also consumes more disk space.
However, the live objects are sufficient to search for Memory Leaks. Unreferenced objects are only of interest in respect to Object Cycling or GC Thrashing. In those cases one wants to find unnecessary or excessive object allocation.
As said initially, heap dumps can be created in 2 ways. But the creation of a dump will halt the JVM and all running processes and applications. This is required to be able to dump a consistent state of the memory. Due to this halt, it is not recommended to use this functionality on a production system very often, if at all.
- When using the JVM parameter
-XX:+HeapDumpOnOutOfMemoryErrorthe JVM will create a heap dump whenever it encounters an OutOfMemoryError before it quits. There is no reason not to use this parameter. The JVM has died and any negative impact on performance is irrelevant. But the size of the dumps can be an issue for certain servers. As the dump is very often the only point of information after a crash it should be created and preserved by the server administrator to allow a “post-mortem” analysis. Unfortunately they are very often simple deleted.
- Using the JVM tool jmap, heapdumps can be created from the command line:
jmap -dump:live,format=b,file=<filename> <PID>
The option live is important, as it will restrict the dump to live objects only. If you want to have also dead objects, simply omit this option. Format “b” (for “binary”) is recommended. While there is an ASCII version as well, manually analyzing it is almost impossible due to the amount of data.
Heap – inside out
There are a few tools for analyzing heapdumps offline. But detailing a comparison here is not worth the effort, as there is a clear winner: Eclipse Memory Analyzer Toolkit (Eclipse MAT) Eclipse MAT can read, parse and analyze dumps of various formats and sizes. It has an easy to use, but sometimes a bit too powerful, user interface and is completely free to use.
While older tools required to load the whole dump into memory, requiring very often 1.5 times the size of the dump as free memory, MAT will index the dump initially which allows fast analysis using just very few system resources. This enables users to work even with very large heapdumps.
After opening a dump with Eclipse MAT following objects are usually listed very often:
Those objects are in every dump in large amounts. But are those objects the issue? How can I find the real root cause for a memory leak?
When looking at an OutOfMemoryError induced heapdump post mortem, one can usually assume that the Memory Leak filled the majority of the memory. But how big is the leak? While the dump contains all object instances, Eclipse MAT shows in its Class Histogramm only class names, amount of instances and the sum of their shallow size. Shallow Size is basically the sum of the sizes of all contained primitive data types, as well as the size of the references to other objects. But the so called Retained Heap is more important for finding leaks, as it is the total size of all objects being kept alive by this dominator. To get this number, MAT needs to calculate the Retained Size by following all object references. As sorting by Retained Heap is very helpful for finding leaks, MAT displays the largest retained heaps in a pie chart. A prominent example for a lot of Retained Heap in web applications is usually the
The purpose is clear: keep user data for the time of their session. But very often frameworks store a lot of data here as well. We already blogged about Ajax4JSF consuming a lot memory in the session. But also custom written, and unfortunately often incorrect, caches appear very often in the pie charts. This makes it very easy to locate memory leaks and their cause quickly.
To determine who is creating these objects, or find out what the purpose of some structures is, the actual instances with their incoming and outgoing references are required. To get them, choose List Objects with incoming References from the context menu. Now a tree structure is displayed, showing all instances with all incoming references. Those references keep the object alive and prevented them from being garbage collected. Outgoing References are interesting as well, because they show the actual contents of the instances, helping to find out their purpose. They are especially interesting when the actual dominator, like the SessionManagers, should not be removed, but the excessive objects referenced by it should.
Sneaking Memory Leaks
Finding Memory Leaks long before an OutOfMemoryError occurs is possible by creating multiple snapshots with the previously described method. A single snapshot alone can seldom help deciding whether the amount of objects is ok or not. But slowly growing object counts can hint at leaks. Eclipse MAT allows comparing two heapdump snapshots and finding growing structures and instance counts. But this requires two well timed dumps. With some amount of load they should be minutes apart, with few load hours or days. Having a greater time lag helps in separating normal jitter from real issues. The following example nicely depicts the obvious growing amount of XML related classes:
But comparing snapshots is cumbersome and in some cases even misleading. And we actually never want an OutOfMemoryError to happen in production. Even worse, heapdumps do not tell us about the lines of code actually creating objects and not removing them. There has to be a better tool than heapdumps!
And there are: Profiler and some APM tools allow recoding statistics and access pathes during runtime. The information provided by those tools is often more significant and easier to act on than heapdumps. For those reasons, we are going to have a closer look at those tools during our next installment of our OutOfMemoryError tragedy.
Optimizing iText performance using AppDynamics and YourKit
The following example shows how easy it is to combine a performance monitoring solution with a profiler. On a regular patrol through our AppDynamics monitoring on our continuously integrated projects, I found this interesting HotSpot in iText. iText...
27.11.2010 | 2 Minuten Lesezeit
Phantom java logger causing major performance problems
Recently at a customer, I saw massive amounts of garbage generated, causing many garbage collections, as well as a huge slowdown inside Hibernate code. I browsed through the slow transactions recorded in production by AppDynamics, and was wondering why...
11.11.2010 | 2 Minuten Lesezeit
Easy Performance Analysis with AppDynamics Lite
AppDynamics is the rising star in the Application Performance Management sky. Mirko gives a really good description why AppDynamics delivers the right solutions for todays distributed architectures in his Post “Troubleshoot Java in production – introducing...
30.8.2010 | 1 Minuten Lesezeit
A Different Take on Sprint Retrospectives
There are many ways to do a good sprint retrospective, so we decided to try a new one every now and then. This time we took the role of a painter, painting out impression of the last sprint into a formidable piece of art. It might look strange at the...
- Agile methods
4.8.2010 | 2 Minuten Lesezeit
WordPress WPML Comments Filter Plugin
I admit, this post is a bit “off-topic”. Recently we migrated this blog from using qTranslate to WPML for publishing in German and English. Main reasons were much better updates and a cleaner separation. But one feature was missing because of that: ...
28.6.2010 | 2 Minuten Lesezeit
Style Tests using Selenium and Robotframework
In projects facing end customers style matters, often more than less. While in internal apps it doesn’t matter if the UI changes after each release, there might be customers that want their app to follow a very strict style guide to integrate with their...
15.6.2010 | 4 Minuten Lesezeit
codecentric playing at german board game championship
“Dr. codecentric und seine kranken Pfleger”, (codecentric, M.D. and his sick attendants) the codecentric board game team, Andreas Ebbert-Karroum, Torsten Rodemann, Marc Clemens and Fabian Lange (left to right) competed in Dinslakenhighly motivated for...
27.2.2010 | 2 Minuten Lesezeit
Hot Coffee and Green Builds
Automated builds and tests already have a long tradition at codecentric, but we never managed to put up build radiators in our new offices. Till today. Developers could have looked up the status in the past, but getting it pushed to you while enjoying...
- Software development
1.2.2010 | 1 Minuten Lesezeit
Meet The Experts Architecture – Open Space: Managing the JAR Chaos
This post shall sum up the results from our fruitful discussion on friday evenig. The idea for the open space discussion was sparked by Stefan Zörner who talked about modularity and what happens when you have no control over modularity. This post will...
29.11.2009 | 1 Minuten Lesezeit
#devoxx 09: map&reduce and closures
A hot topic here at the Devoxx were the upcoming Java editions with their features and changes in the language syntax. While it is nice that you will be able to switch() on Strings, have a modularized platform and other cool stuff, one thing is a bit...
19.11.2009 | 3 Minuten Lesezeit
codecentric Crew visiting #Devoxx 2009
As every year, codecentric Developers are attending the Devoxx. Devoxx in Antwerp is among the top conferences for Java in Europe, known for its hand picked Speakers and excellent topics. No surprise that the depicted 7 gents in codecentric Shirts did...
18.11.2009 | 1 Minuten Lesezeit
JUG Cologne – 5th October – Slides on Eclipse RAP
Having a presentation slot at a Java User Group is always special. Its an audience who cares, or is there just for the buffet. No kidding, todays evening was great. Besides my talk on RAP for which i attach the slides, there were new insights on what...
5.10.2009 | 1 Minuten Lesezeit
Neal Ford at RheinJUG: Emergent Design & Evolutionary Architecture
Back after having a great evening at todays RheinJUG talk held by Neal Ford. It was almost a perfect fit for our upcoming Meet the Experts – Architecture . Because Neal has the slides on his github , I just want to briefly summarize my personal takeaways...
- Software architecture
20.9.2009 | 1 Minuten Lesezeit
Commit every day, or revert – Be agile, every day
One of the biggest problems in agile development teams is “effort”. Of course it is always about effort, because effort is money and we all like our money. In planning we can cope with effort quite easily: “oh that’s a week effort”, but when it comes...
- Agile methods
2.9.2009 | 4 Minuten Lesezeit
JSP Tag Pooling Memory Leaks
JSP custom tags were once widely used, but even still nowadays they find their way into projects. Not to mention the masses of production code using them. And almost all projects I have looked at using custom tags had the same issue. When writing JSP...
13.8.2009 | 2 Minuten Lesezeit
Convert InputStream to String
Because searching for “Convert InputStream to String” still brings up solutions involving StringBuffer, byte or something like that, developers still produce large amounts of different implementations of the same conversion in their projects. In an...
10.8.2009 | 1 Minuten Lesezeit
codecentric coding night – facts & figures
Hier einige interessante Statistiken zur coding night . Da die coding night ein „Projekt im Zeitraffer“ war, sind die von Hudson bei den automatischen Builds erstellten Statistiken ganz interessant. JUnit Test Ausführung Das erste was auffällt ist,...
- Agile Methoden
15.7.2009 | 2 Minuten Lesezeit
Eclipse Galileo and SVN
To prove that I can do short posts as well, here a quick guide to SVN in latest Eclipse release. This was not that easy in previous releases, but now it works like a charm: Help Install New Software… Galileo – http://download.eclipse.org/releases/galileo...
28.6.2009 | 1 Minuten Lesezeit
Can I change this Code?
“Can I change this code?” sounds like a normal question, but in my opinion it expresses a problem in agile development that needs addressing. Foremost: This is a very good question, because it shows a noble intent: Make code you found better. Following...
- Software architecture
- Agile methods
29.4.2009 | 3 Minuten Lesezeit
Data Validation Alongside Agile Development
I would like to discuss an issue one can likely experience with agile development processes and systems which data needs to be maintened during upgrades: A customer care application for a PC retailer was developed so far and the software is running pretty...
- Software development
27.1.2009 | 2 Minuten Lesezeit
Ajax World Conference in San Jose, CA
From the 20th to 22nd of October the 6th Ajax World Conference took place in the sunny San Jose, CA. I was there those 3 days as delegate of codecentric to catch up with the newest trends and developments in Ajax and RIA. I tried to collect and write...
4.11.2008 | 1 Minuten Lesezeit
Dein Job bei codecentric?
Agile Developer & Consultant (w/d/m)
An allen Standorten
Gemeinsam bessere Projekte umsetzen.
Wir helfen Deinem Unternehmen.
Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.
Hilf uns, noch besser zu werden.
Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.
Do you still have questions? Just send me a message.