All the geeks at codecentric were pretty excited about TableTopDay, a table top board gaming holiday invented by the fine folks at GeekAndSundry, Felica Day and Wil Wheaton. As response to their extremly successful YouTube Show “TableTop”, which accumulated already over 8.5 million views, they proclaimed March 30th to be International TableTopDay. Over 2500 groups registered their event on tabletopday.com and so did we.

While I expected only codecentrics to attend, we ended with one guy working for one of our customers, one random guy who registered on our Google+ Event page and 4 strangers who showed up without any notice. But also all the codecentrics brought their families. It was successful and fun event, for us and for all board game nerds worldwide, as proven by the hashtag #TableTopDay, which was trending on Twitter the whole Saturday, and many people posting pictures or even livestreaming their event.

Games we played:

• Pandemic
• Small World
• Stone Age
• Fearsome Floors
• Castle Panic
• Hansa Teutonica
• Zombie Dice
• Munchkin

And maybe others; I was busy winning, could not watch all games being played Thanks to all people attending making this possible, and special thanks to codecentric for hosting this event! Looking forward to next year, or earlier whenever possible!

Play more games!

Let me start with a screenshot from our AppDynamics monitoring.

The green boxes are all Java 7 based services monitored with AppDynamics. The number “4″ within the blue circle indicates that at the moment there are currently 4 instances each running. This number can change when services are scaled up and down. We currently run a mix of KVM virtualized and non-virtualized CentOS instances on high end Dell Machines with a total of about 148 cores, 600GB memory and 150TB disc space.
The lines connecting the boxes indicate call directions, their average response times and how often they were made in that time interval. What is really great about AppDynamics is that the services, and how they talk to each other is automatically detected.

In the middle of the picture is the heart of our architecture “tomcat-rest”. When you talk to

api.centerdevice.de/v1

you arrive there (and see a message that tells you you are missing our oAuth authorization – we are planning to publish our API soon). It hosts all of our REST services, which are implemented with Jersey. We chose Jersey, because it is very easy to implement REST Services with it, is a proven piece of software and also easily testable in unit and integration tests as outlined in this excellent blog post by Michael Lex about Writing lightweight REST integration tests with the Jersey Test Framework.

Services accessing any kind of data use our MongoDB backend, visualized by AppDynamics as blue boxes on the top. As a scalable database with easy access and painless schema changes (because there is no schema), it suits our performance requirements, as well as the changeability we require for a young product with frequent releases. If you want to know more about MongoDB I recommend reading Thomas Jaspers MongoDB tutorial. We talk to it via the Java Driver, have implemented replicasets and shard to multiple nodes. The major challenge here is to accept the eventual consistency. Even code which writes data might not yet be able to read it again in the next line.

Uploaded documents are stored on the Gluster backed XFS file system. All data is encrypted using a per user 256 AES key before it is persisted on the file system. Gluster then takes care of replicating the data to all servers.

For clients, we developed two native clients: CenterDevice for iPad and CenterDevice for Android Phone. The reason for native apps is that some emulated or cross compiled platforms have plenty of issues. Mark Zuckerberg also recently had a widely acknowledged talk about the problems HTML5 mobile clients have at the moment. And because iOS dominates the tablet market and Android rules the phone market, that is what we started with.

Note that our screenshots show cute baby animals, instead of boring documents, because they are more adorable.
Besides our native iPad and Android Phone application, and a few third party clients, the main user of the REST Server is our Vaadin 6 based web client. In fact we have two of them

app.centerdevice.de

and

public.centerdevice.de

Both are logically hosted on “tomcat-centerdevice”.
Vaadin, as a framework, allows us to implement complex rich web interfaces easily, using SWT Style Java code. It compiles to GWT JavaScript which is then delivered to the browser. The bulk of the work can be easily done in Vaadin, which we combine with CDI-utils, a plugin that allows implementing the MVP pattern easily using Weld as CDI library. Developing complex components however requires developing GWT Widgets yourself sometimes, which is not that easy. For easy copy to clipboard functionality, Henning Treu wrote a Copy to clipboard vaadin addon, which we open sourced.

Communication between the web application servers and the rest server is only unidirectional, but sometimes we want to send back notifications (like new documents somebody just shared with you) to the webserver.

That is where RabbitMQ comes into play. The application map from AppDynamics shows our 3 usages of messaging:

• Sending Notifications from REST to Web.
• Sending requests to send e-Mails (currently sent from and consumed by the REST Server).
• Sending processing requests from REST server to doc-server.

RabbitMQ is set up using HA queues and worked flawlessly so far. Tobias Trelle wrote an introduction into RabbitMQ with Spring, which provides more background about Rabbit MQ and AMQP.

The document processing done by “doc-server”, which has multiple tasks depending on the type of input document:

• Generating PDF representations
• Generating preview images for different sizes
• Performing fulltext extraction
• Performing OCR
• Getting page count
• Obtaining additional metadata
• Detecting language

There are basically two types of documents that we use as basis: PDF and Images.
If we are getting any other format, we use Libre Office to convert it to PDF, or Imagemagick to convert it to images.
ImageMagick is then also used to generate preview images in various sizes.
Depending on the type of document we can use Apache Tika to get the fulltext from the document. For documents where Tika cannot find a fulltext, we resort to OCR running on “tomcat-ocr”. OCR will be done using tesseract. Further metadata is extracted using Tika and custom detectors.

Search capabilities are provided by Apache Solr 3, running on “tomcat-solr”. Solr is running in Master Slave mode. One neat feature of CenterDevice is that is performs super fast search on everything we can extract from documents or their metadata.

Performance

As you can see in the screenshot, the overall performance is quite nice. However it largely depends on the documents uploaded for individual processing requests. AppDynamics automatically learns the normal behavior and alerts us when there is deviation. For uploads and downloads however, we turned this learning off. Depending on the document and the clients connectivity, there is just nothing like a normal time a client takes to upload or download a document. We however gather diagnostic data for extraordinary slow up/downloads to investigate in case of customer complaints. A similar story applies to the document processing. While in most cases also the learned baselines are good, sometimes they do not match. That is why in the screenshot the connection to our external processing services are red. Some heavy document processing was going on at the time the screenshot was taken. When performance degrades, AppDynamics captures important metrics and provides us code level insight into the root cause. So far the most issues were typical like too many queries, too much API calls, inefficient indicies etc.

ToDos

We always have ideas for improving stuff and are moving fast, so the architecture will change. While currently we do not have a pressing need, these changes will be most likely coming in future:

* Introduce WebSession replication for failover using memcache or redis. (currently we loose web session data on failure (not happened so far), and deployments (happening during nights when no sessions are alive)). The major challenge so far seems to get it integrated into the Servlet 3 async pushing we do.
* Switch from Gluster to Hadoop File System (We were bitten by lots of Gluster problems, like running on ext-4 64bit, which we now changed to xfs)
* Switch from Solr 3 to Elastic Search (the master-slave failover just does not work as nicely and does not scale, plus changes to Solr 3 need downtime)
* Add a reverse proxy / loadbalancer layer to perform green / blue deployments, redirect specific users to certain versions of the software.

Join the team!

I hope you enjoyed the overview on architecture and software we use. If you are interested in helping us build and grow this stack, we have good news for you: CenterDevice and codecentric are hiring! We are especially looking for a dev-opsy Linux and Hardware enthusiast to build out what I described above. Get in touch!

Recap: Memory Footprint of Data

Question: How much memory does the String “Hello World” consume?
Answer: 62/86 Bytes (32/64 bit Java)!
This breaks down into 8/16 (Object Header for String) + 11 * 2 (characters) + [8/16 (Object Header char Array) + 4 (array length) padded to 16/24] + 4 (Offset) + 4 (Count) + 4 (HashCode) + 4/8 (Reference to char Array). [On 64Bit the size of String Object is padded to 40].

The Problem

Imagine you have a lot of Locations attached to tweets in your data store. The implementation of the location as a Java class could look like this

class Location {
	String city;
	String region;
	String countryCode;
	double long;
	double lat;
}

So if you load all the locations of tweets ever made, it is quite obvious that you load a lot of String objects, and at the scale Twitter has, there are for sure a lot of duplicate Strings. Attila said that this data did not fit into a 32 GB heap. So the question is: can we reduce memory consumption, so that all Locations fit into memory?

Let us have a look at two possible solutions, which can even augment each other.

Attilas Solution

There is a, more or less, hidden dependency between the data stored in the Location class, which, once realized, will solve the problem in an elegant way, non-technical way. We can just apply normalization to the Object and split it into two:

class SharedLocation {
	String city;
	String region;
	String countryCode;
}
class Location {
	SharedLocation sharedLocation;
	double long;
	double lat;
}

This is actually a neat solution, because cities rarely change the region and country they reside in. The combination of those Strings is unique. And it is flexible, so that even violations of those uniqueness can be handled. This makes always sense for data which could be user input. By doing so, multiple Tweets from “Solingen, NRW, DE” just consume one SharedLocation.
But still “Ratingen, NRW, DE” would store 3 additional String in memory, rather than just the new one “Ratingen”. By refactoring the data model like that the amount of data for that Twitter research project dropped to about 20GB.

String interning

But what to do when you do not want to, or simply cannot refactor the data model? Or the researcher at twitter would not have had a 20GB Heap?
The answer is String interning, which keeps every String only once in memory. But there is huge confusion about String interning. Many people ask if it speeds up equals checks, because equal strings are actually identical Strings when using interning. Yes, it might do that (like all objects should)

// java.lang.String
public boolean equals(Object anObject) {
  if (this == anObject) {
    return true;
  }
  //...
}

But equals performance is not the reason you should do interning. String interning is intended to reuse String objects to save memory.

Only use String.intern() on Strings you know are occurring multiple times, and only do it to save memory

How effective interning is, is determined by the ratio of duplicate/unique strings. And it depends on whether it is easy to change code at string generating places.

So how does it work?

String interning takes a String instance (so it already exists in the Heap) and checks if an identical copy exists already in a StringTable.
That StringTable is basically a HashSet that stores the String in the Permanent Generation. The only purpose of that Table is to keep a single instance of the String alive. If it is in there, the instance is returned. If its not, its added to the String Table:

// OpenJDK 6 code
JVM_ENTRY(jstring, JVM_InternString(JNIEnv *env, jstring str))
  JVMWrapper("JVM_InternString");
  JvmtiVMObjectAllocEventCollector oam;
  if (str == NULL) return NULL;
  oop string = JNIHandles::resolve_non_null(str);
  oop result = StringTable::intern(string, CHECK_NULL);
  return (jstring) JNIHandles::make_local(env, result);
JVM_END
 
oop StringTable::intern(Handle string_or_null, jchar* name,
                        int len, TRAPS) {
  unsigned int hashValue = hash_string(name, len);
  int index = the_table()->hash_to_index(hashValue);
  oop string = the_table()->lookup(index, name, len, hashValue);
 
  // Found
  if (string != NULL) return string;
 
  // Otherwise, add to symbol to table
  return the_table()->basic_add(index, string_or_null, name, len,
                                hashValue, CHECK_NULL);
}

As a result, this specific instance of a String only exists once.

How to use interning

The right place to use String interning is where you read from data store and add the Objects/String to a larger scope. Note that ALL Strings which are hardcoded (as constant or anywhere in code) are automatically interned by the compiler.
An example would be:

String city = resultSet.getString(1);
String region = resultSet.getString(2);
String countryCode = resultSet.getString(3);
double city = resultSet.getDouble(4);
double city = resultSet.getDouble(5);
 
Location location = new Location(city.intern(), region.intern(), countryCode.intern(), long, lat);
allLocations.add(location);

All newly created location objects will use the interned string. The temporary strings read from the database will be garbage collected.

How to find out how effective would be string interning

You are best off taking a heap dump of a quite full heap. It can even be collected at an OutOfMemoryError.
Open it in MAT and select the java.lang.String from the histogram. On that one use “Java Basics” and “Group By Value”

Depending on the heap size, this may take a long time. In the end the result will be like this. Sorting either along retained heap or number of objects will reveal interesting things:

From this screenshot we can see that empty Strings take a lot of memory! 2 million empty Strings take a total of 130MB. Then we see some amount of JavaScript that is loaded, a few more technical strings like the keys, which are used for localization. And we can see some Strings related to business logic.
These business logic strings are probably the easiest to intern, because we might know where they are loaded into memory.
For the other ones we need to use “Merge shortest Path to GC Root” to identify where they are stored, which may or may not reveal to use where it is created.

Tradeoffs

So why not always do String interning? Because it slows your code down!
Here a small example:

private static final int MAX = 40000000;
public static void main(String[] args) throws Exception {
	long t = System.currentTimeMillis();
	String[] arr = new String[MAX];
	for (int i = 0; i < MAX; i++) {
		arr[i] = new String(DB_DATA[i % 10]);
		// and: arr[i] = new String(DB_DATA[i % 10]).intern();
	}
	System.out.println((System.currentTimeMillis() - t) + "ms");
	System.gc();
	System.out.println(arr[0]);
}

The code uses a String array to keep a strong reference to the String objects. and we print the first element in the end to avoid removal of the structure due to optimization. Then we load 10 different Strings from the database. I used new String() here to illustrate the temp String allocation you always do when you read from storage. At the end I do a GC, so that the results are correct and no leftovers are included.
This was run on a 64bit Windows, JDK 1.6.0_27, i5-2520M with 8GB Ram. Run with -XX:+PrintGCDetails -Xmx6G -Xmn3G to log all GC activity. Here is the output:
Without intern()

1519ms
[GC [PSYoungGen: 2359296K->393210K(2752512K)] 2359296K->2348002K(4707456K), 5.4071058 secs] [Times: user=8.84 sys=1.00, real=5.40 secs] 
[Full GC (System) [PSYoungGen: 393210K->392902K(2752512K)] [PSOldGen: 1954792K->1954823K(1954944K)] 2348002K->2347726K(4707456K) [PSPermGen: 2707K->2707K(21248K)], 5.3242785 secs] [Times: user=3.71 sys=0.20, real=5.32 secs] 
DE
Heap
 PSYoungGen      total 2752512K, used 440088K [0x0000000740000000, 0x0000000800000000, 0x0000000800000000)
  eden space 2359296K, 18% used [0x0000000740000000,0x000000075adc6360,0x00000007d0000000)
  from space 393216K, 0% used [0x00000007d0000000,0x00000007d0000000,0x00000007e8000000)
  to   space 393216K, 0% used [0x00000007e8000000,0x00000007e8000000,0x0000000800000000)
 PSOldGen        total 1954944K, used 1954823K [0x0000000680000000, 0x00000006f7520000, 0x0000000740000000)
  object space 1954944K, 99% used [0x0000000680000000,0x00000006f7501fd8,0x00000006f7520000)
 PSPermGen       total 21248K, used 2724K [0x000000067ae00000, 0x000000067c2c0000, 0x0000000680000000)
  object space 21248K, 12% used [0x000000067ae00000,0x000000067b0a93e0,0x000000067c2c0000)

With intern()

4838ms
[GC [PSYoungGen: 2359296K->156506K(2752512K)] 2359296K->156506K(2757888K), 0.1962062 secs] [Times: user=0.69 sys=0.01, real=0.20 secs] 
[Full GC (System) [PSYoungGen: 156506K->156357K(2752512K)] [PSOldGen: 0K->18K(5376K)] 156506K->156376K(2757888K) [PSPermGen: 2708K->2708K(21248K)], 0.2576126 secs] [Times: user=0.25 sys=0.00, real=0.26 secs] 
DE
Heap
 PSYoungGen      total 2752512K, used 250729K [0x0000000740000000, 0x0000000800000000, 0x0000000800000000)
  eden space 2359296K, 10% used [0x0000000740000000,0x000000074f4da6f8,0x00000007d0000000)
  from space 393216K, 0% used [0x00000007d0000000,0x00000007d0000000,0x00000007e8000000)
  to   space 393216K, 0% used [0x00000007e8000000,0x00000007e8000000,0x0000000800000000)
 PSOldGen        total 5376K, used 18K [0x0000000680000000, 0x0000000680540000, 0x0000000740000000)
  object space 5376K, 0% used [0x0000000680000000,0x0000000680004b30,0x0000000680540000)
 PSPermGen       total 21248K, used 2725K [0x000000067ae00000, 0x000000067c2c0000, 0x0000000680000000)
  object space 21248K, 12% used [0x000000067ae00000,0x000000067b0a95d0,0x000000067c2c0000)

We can see that the difference is significant. It took 3.3 seconds longer to run with interning. But the memory saved was enormous. When the code finished, the program using interning used 253472K(250M) of Memory. The other one used 2397635K (2.4G). That is quite a difference and illustrates nicely the tradeoffs when using String interning.

So let me walk you through what they did, and show you the problem that killed their server.

An Overview of the Situation

Here is what the situation looked like on Saturday 8 in the morning.


If you don’t know AppDynamics, here a short introduction:

• The biggest area is the application map. AppDynamics autodiscovers and monitors all systems and their interaction.
• On the right side there are some statistics. We can see some Stalls and Abnormal Slow requests. AppDynamics found out that something is not right.
• On the bottom is the historical view. Because this is historical data now, we only see hourly data. During the incident, the data was more fine grained.

The historical data is of most interest for us, looking back.
We can see the server start on the left hand side (blue icons on top of green bar). The load on the system climbed rapidly (green bar). But then the system broke down. The response time increased massively (blue bar). And requests decreased. Soon the server crashed and had to be restarted.
But the restart did not help, the system did not recover. Luckily it did later after the problem was fixed

What went wrong

The system crashed with an OutOfMemory Error. In our OutOfMemoryError Series, we already discussed some of them, and here is another: “Unable to create native thread”.

This error is actually an interesting fact. Outages with this kind of Error are usually not created by inefficient code, but either by infrastructural problems or buggy threading code. So lets look at the number of threads on each server recorded by AppDynamics:

Ouch. When it crashed, the system had already created 1800 threads.
What was producing these threads? I have seen this before at customers, so I immediately knew this. And also my client was able to find out by seeing a few snapshots of stalled calls. JSP Compilation was the culprit.
All application servers have an option to precompile JSPs on startup, or when requested for the first time. But both settings are problematic when the server is started under high load.
The problem was resolved on Sunday Morning 4AM. But the ops team had not to be awake those 20 hours. They had reported the qualified issue to Oracle around noon on Saturday. The Oracle provided “fix” was simple, but not documented anywhere. Or at least not where you can find it if you do not know it.
Here is the screenshot from AppDynamics discovering the property change and restart at 4AM.

Limiting JSP Compiler Threads on Weblogic

The “secret” switch was an environment variable you have to set:

BEA_COMPILER_NUM_THREADS = 1

While 1 is for sure very pessimistic, it seems to be a way better setting than unlimited, which is the default. No one should ever create an unlimited amount of threads. If you think in a classical view, you should only have as much threads as you have number of cores. modern JVMs can easily handle like ten times that much threads, without much starvation. But in this case we had about 100 Compiler Threads per CPU.

Installation

Plumbr is as easy to install as AppDynamics. Your downloaded package contains files for various platforms, so this is an indication, that plumbr is using a JVMTI native agent in addition to the javaagent. After unzipping, I needed to add this to my Tomcat 7, which will be running the memory leak app:

SET PATH=C:\Tools\plumbr\plumbr\win\64;%PATH%
set CATALINA_OPTS=-agentlib:plumbr -javaagent:C:\Tools\plumbr\plumbr\plumbr.jar

Finding Memory Leaks Fast

Plumbr is working very fast. I deployed my leaking application and started the load script. After only a few minutes this messaged appeared:

So the only thing I needed to do was to open the generated report:

Tadaa! The leak was correctly identified. And it also lists where the leak comes from, which is very important:

The objects are created at
  de.codecentric.memleak.leak.StringQueueLeak.leak(de.codecentric.memleak.domain.Bookmark):58

When compared to AppDynamics, this is slightly less information, because the call stack is shorter. And it also does not give invocation counts. However this is still great information, which helps us to identify and fix the leak. And it is presented in a dead simple way.

Further improvements

You might wonder why there are 3 leaks identified, while AppDynamics did show only one of them. The second and third are classes used by the Derby database:

So plumbr currently assumes that a reference from java.lang.ref.Finalizer is a memory leak. There is some controversy (for example on stackoverflow) about this being possible leaks. So I mailed this observation to the plumber developers and got a quick answer: While they think Finalizers are problematic, they want to change the handling, so there will be less likely false positive reportings.

I think plumbr is a really interesting solution. It solves the very specific problem of finding memory leaks in an elegant way. It reports fast and did not add significant overhead to my application, but I still have to trial it in a real application server environment to make an educated statement.

But not all leaks will cause an OutOfMemoryError and produce a dump, or they would take a very long time to occur. For example the server and the JVM could even be restarted in a regular interval for deployments or to fight memory issues.

To find slowly growing memory leaks, we have to perform more complicated and time-consuming analysis. We could use multiple dumps, which are spread out over time. While they theoretically would allow us to recognize growing structures, it would be tedious in practice, because the difference between multiple dumps will be mostly normal fluctuation, which makes it difficult to spot the relevant delta. And you already have to know the memory leak producing use cases which you then can invoke between dump. But perhaps the biggest problem of all is that creating a dump in production is not advisable, because it can hang the system for seconds to minutes, depending on the heap size.

A better solution is to monitor the heap and the relevant objects within during the runtime of our application. By doing so we can track every structure and get notified when something keeps on growing over time. And because the application is still running fine, we can also easily get the information about the code which is interacting with the leak. This is not possible at all using heap dumps, as they do not contain information about code.

The concept

The pattern for runtime analysis of heap is pretty simple:

• Find all objects which are created by the application.
• Track those objects and record their size.
• Alert on any “abnormal” behavior.
• Provide content and invoking code for diagnosis.

Unfortunately each and every of those points brings a lot of issues in practice. As a result of this, there are only a few implementations of this concept. Already finding all the relevant objects is not an easy task. In Act 4 I recommended to focus on our own packages, like de.codecentric.memoryleak. But what do we do when standard classes leak? While in a demo application the number of objects might be manageable, in real applications, there are millions of objects. How can we ever efficiently store data on those complex structures? And what is “abnormal” behavior? Are there sizes and lifetimes of objects that we can consider “normal”?

An Implementation

As an example for this concept, I am going to showcase the Leak Detection Feature of the APM solution AppDynamics. The only other implementation of a leak detection, which does not use heap dumps, I am currently aware of, is the Introscope Leak Hunter. Should you know a tool which does it in a similar way, I would be happy to get learn about them in your comment!

Assumptions

As you can guess, a solution as outlined above cannot be realistically implemented. We need to simplify the problem using simple assumptions. Luckily there are quite a few assumptions you can do for any Java program. For example the typical age distribution of objects is used by Garbage Collectors to work with different generations, as I described in Act 3.

AppDynamics is doing the following assumptions:

• There is no need to monitor all objects. Experience shows us that most memory leaks are caused by putting data into collection type structures, like lists and maps, but not removing the data from there later. Custom cache implementations are a very typical example. Because of that, AppDynamics monitors just those classes.
• We do not need to look at collections that are not used, like internal structures created by the application server on startup. We just need the structures our code interacts with.
• From those active collections, we just need to monitor those which contain a relevant number of objects. And because we look for leaks, that number has to increase over time.
• Those long and active collections could be leaks, but to become a relevant problem for the stability of our code, that collection has to dominate a significant amount of memory.
• All those factors apply over a longer period of time.

AppDynamics uses a similar process to find leaks. By that it minimizes impact on the monitored JVM. Additionally AppDynamics uses elaborated algorithms to calculate object tree sizes efficiently with very low overhead, even under high load. Nevertheless memory analysis will always be connected to a higher overhead.

My Open Source Collection Analyzer

Because AppDynamics is a commercial solution, I wanted to have my own shot at implementing such a memory leak finder.
You can find my version of a basic Java Memory Analyzers on Github.

The fundamental idea is surprisingly easy to implement. My analyzer is coded using just two classes.

CollectionAnalyzerAspect

I do similar assumptions as AppDynamics and just watch Collections. If I wanted, I could add any possibly leaking classes here:

@Before("   call(* java.util.Map.put(..)) &&
            !this(de.codecentric.performance.memory.CollectionAnalyzerAspect)")
public void trackMapPuts(final JoinPoint thisJoinPoint) {
	Map target = (Map) thisJoinPoint.getTarget();
	CollectionStatistics stats = getStatistics(target);
	stats.recordWrite(getLocation(thisJoinPoint));
	stats.evaluate(target.size());
}

This Pointcut adds my code before all calls to Map.put(). Because I am using a map myself to store statistical data, I need to exclude myself to avoid a nasty recursion. Next, I get a statistics storage object for the collection instance I monitor to record access and evaluate its usage.
This is a simplistic approach. I think it would be much better to evaluate all statistics in a separate thread periodically than to do this synchronous on every request.
There is one additional interesting problem: How can I identify the Collection I am currently inspecting? For that I am using the “identityHashCode”, but I already know that this might not be a wise idea, as it might not be unique.

int identityHashCode = System.identityHashCode(targetCollection);

CollectionStatistics

Ok, I have recorded all the invocation counts for the methods. So what do I do with this data?

public void evaluate(int size) {
	if (size >= DANGEROUS_SIZE) {
		System.out.printf("\nInformation for Collection %s (id: %d)\n", className, id);
		System.out.printf(" * Collection is very long (%d)!\n", size);
		if (reads == 0)	System.out.printf(" * Collection was never read!\n");
		if (deletes == 0) System.out.printf(" * Collection was never reduced!\n");
		System.out.printf("Recorded usage for this Collection:\n");
		for (String code : interactingCode) {
			System.out.printf(" * %s\n", code);
		}
	}
}

I did not really decide on an elaborated criterion for when a collection behaves “abnormal”. I just took a hardcoded collection length. It would be a great idea to have a WeakReference to the collection and calculate the dominator tree of it, but to calculate the Deep Size is a pretty complex problem on its own.

Besides the length, I consider two factors as interesting:

• Is this collection ever read?
• Was something ever deleted from it?

Both are typical antipattern for caches. Nobody reading or deleting from long lists is a clear indicator for a problem. Thats why I am warning on it. Last, I print all the recorded invoking code, which is a pretty useful information!

A Testrun

Information for Collection java.util.ArrayList (id: 1813612981)
 * Collection is very long (5000)!
 * Collection was never reduced!
Recorded usage for this Collection:
 * de.codecentric.performance.LeakDemo:19
 * de.codecentric.performance.LeakDemo:17
 * de.codecentric.performance.LeakDemo:18
 
Information for Collection java.util.ArrayList (id: 1444378545)
 * Collection is very long (5000)!
 * Collection was never read!
 * Collection was never reduced!
Recorded usage for this Collection:
 * de.codecentric.performance.LeakDemo:18
 
Information for Collection java.util.HashMap (id: 515060127)
 * Collection is very long (5000)!
 * Collection was never read!
 * Collection was never reduced!
Recorded usage for this Collection:
 * de.codecentric.performance.LeakDemo:19
 
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at de.codecentric.performance.DummyData.(DummyData.java:5)
	at de.codecentric.performance.LeakDemo.runAndLeak(LeakDemo.java:17)
	at de.codecentric.performance.DemoRunner.main(DemoRunner.java:12)

Heureka, it works!

Overhead

To record each and every invocation of a collection is quite memory intensive. Perhaps I should do this only when I have the indication that this collection could be leaking. But b using those AspectJ pointcuts, my code will always run. In a real environment with hundred thousand such collections it will not be a great idea for sure. Dynamic ByteCode Instrumentation should be used to avoid this. And of course an evaluation over a long period of time makes more sense than my quick checking.

As we can see, the idea is easy to implement, but a production ready solution requires a good amount of thinking and clever algorithms. If you like to improve my analyzer, feel free to send patches via GitHub.

Memory Analysis of a Demo Application using AppDynamics

So lets have a look at what AppDynamics does in a professional solution.

10:43 – Application Server Restart

After starting the AppDynamics Leak Detection you will not get immediate results. It starts in the background analyzing collections. Only after a while possible leaks might show up.

11:00 – Collection detected

This java.util.LinkedList is monitored by AppDynamics. It has 56,881 entries, which makes it indeed interesting. But AppDynamics has no long time information yet, so it is not marked as  “potentially leaking”.

11:10 – Collection potentially leaking

Time passed, but the collection continued to grow. 98,850 entries are almost twice the amount compared to ten minutes ago. The internal heuristics now mark it as “potentially leaking”.

11:17 – The leak is growing

The overview is showing the growth of the leak. Garbage Collection activation would be also drawn here to visualize effects of using SoftReferences.

11:30 – Showing the memory leak

The Content Inspection shows us what is inside the Collection. In this case there are now 118,990 java.lang.String objects with a total size of 20MB.
AppDynamics can also dump the collection and its contents to disc to allow a more detailed analysis of the contents.

11:38 – Identifying the root cause

By using an Access Tracking Session AppDynamics finds out who is creating this memory leak. While you might come up to this point using heap dumps, the listing of the call hierarchy is something special. LinkedLists containing Strings could have been used everywhere, but this leaking LinkedList is used by the “newbookmark” business transaction.
The BookmarkDaoImpl is appending Strings to that list in line 50. However, AppDynamics did not see any code reading or deleting from this list.

So we now got all the information we need to fix this memory leak:

• We can see the potentially leaking structures.
• We get notified about leaks automatically.
• We can see the contents of those structures.
• Business transactions (Use Cases) responsible for creating the leak are identified.
• Accessing code is recorded and shown.

The final decision on whether this is a memory leak or just strange code is of course still up to the developer.

Wrap Up

It is possible to find memory leaks at runtime without creating heap dumps. The information on the invocating code is very useful for correcting memory leaks. Unfortunately, there is no free or open source product for finding leaks in such a way. As you have seen, it is not recommendable to implement it yourself.
AppDynamics has a free 30 day trial which includes the memory leak finder, so you can check for yourself if it is something you can use.