One of the biggest strength of the Java Platform is the implementation of an automatic memory management in the Java Virtual Maschine. Everybody who has programmed with languages like C/C++ knows about the problems of managing memory allocation and deallocation in the code. With Java problems like deallocating memory too early (corrupted pointer) or too late (memory leak) cannot occur by specification. The question is: Why am I writing these blog entries?

The problem is that even with an implicit memory management integrated, Java cannot prevent application of being corrupt in sense of memory management, even it is not allowed to explicitly allocate memory in Java. The result of such wrongly programmed code normally is an exception of type: java.lang.OutOfMemoryError.

This part of the blog series about Java OutOfMemoryError, will introduce the Java Memory Model in detail and shows in which memory areas an java.lang.OutOfMemoryError can occur. Details about the cause of these errors and the tools and methods for analysis will be covered in later entries.

Lets start by looking at the Javadoc of java.lang.OutOfMemoryError:

Thrown when the Java Virtual Machine cannot allocate an object because it is out of memory, and no more memory could be made available by the garbage collector.

This description copied from the actual Java API Documentation (Version 6) is not only very short, but in my point of view incomplete and therefore wrong. This description does only cover the heap of the JVM – as we will learn later, OutOfMemoryError can also occur in different areas of the JVMs memory. These errors are not mentioned in the Javadoc, but you can see them every day in real world applications.

The architecture of Java’s memory management is defined for all JVM implementations in the Java Virtual Maschine Specification. Chapters 3.5 Runtime Data Areas and 3.6 Frames are the most relevant for memory architecture. For a better understanding, I’ve drawn the following picture as a summary of the chapters on memory areas in a JVM.

We can basically distinguish memory areas that are available for all threads in a JVM and those memory areas that are exclusively accessible from only one thread. The two areas that are available from all threads are the Method Area and the Heap.

The method area is responsible for storing class information. The Class-Loader will load the bytecode of a class and will pass it to the JVM. The JVM will generate an internal class representation of the bytecode and store it in the method area. The internal representation of a class will have the following data areas:

• Runtime Constant Pool
  Numeric constants of the class of types int, long, float or double, String-constants and symbolic references to all methods, attributes and types of this class.
• Method Code
  The implementation (code) of all methods of this class including constructors etc.
• Attributes
  A list of all named attributes of this class.
• Fields
  Values of all fields of this class as references to the Runtime Constant Pool.

The method area can be part of the heap and will be created at runtime. The size of the method area can be static or dynamic and it does not have to provide a Garbage Collector.

The second memory area that is available for all threads inside the JVM is the Heap. The Java heap manages instances of classes (objects) and arrays at runtime. The heap will be created at JVM startup and the size can be static or dynamic. The JVM specification mandates a Garbage Collection mechanism for reclaiming the memory of an object on the Java heap. The implementation of the Garbage Collector is not specified, but it is not allowed to provide the programmer with an explicit mechanism for deallocating the memory of an object.

Lets have a look at the Sun HotSpot implementation as an example:

The heap is devided into two generations: The Young Generation and the Tenured Generation. The details of this “generational heap” are not relevant in the context of Java OutOfMemoryError as the design is driven by optimizations of the Garbage Collection algorithm. The method area is implemented as a separated part: The Permanent Generation. All details about configuration and monitoring of these generations will be covered in the third part of this series: “JVM Monitoring  and Configuration”.

This example of the Sun HotSpot JVM memory architecure shows that the JVM specification defines how the memory inside a JVM is organized in general, but leaves enough room for implementation specific optimizations.

In addition to the heap and method area, that are available for all threads of a JVM, every thread also has exclusivly access to memory that is created for each thread:

• PC Register
  The Program Counter register. The register points to the current JVM instruction of the method the thread is executing, if the method is not a native method. If it is a native method the content of the PC register is not defined.
• Java Virtual Maschine Stack
  Each thread gets its own stack on wich so called Frames are pushed for each method the thread currently executed. This means that there can be many frames on the stack for nested method calls – but there is only one frame active at the same time for one thread. The frame contains the local variables of the method, a reference to the Runtime Constant Pool of the method’s class and an operand stack for the execution of JVM operations. (The JVM is a stack maschine!)
• Native Methode Stack
  Native methods get its own stack, the so called „C-Stack“.

Until now you should have get an overview of the Java Memory Model including its different memory areas – this is essential, because now we will take a closer look at our java.lang.OutOfMemoryError. As mentioned before the Javadoc of this exception is not very meaningful, but the Java Virtual Maschine specification defines exactly when and where Java OutOfMemoryError can occur. The difficulty is that theses errors can occur in every memory area I’ve described before. Let’s have a look at the Sun HotSpot JVM and its concrete implementation of OutOfMemoryError errors.

In the heap we get an OutOfMemoryError, if the garbage collector cannot reclaim enough memory for a new object. In such situation the Sun HotSpot JVM shows this error message:

Exception in thread "main": java.lang.OutOfMemoryError: Java heap space

A alternative for this is

Exception in thread "main": java.lang.OutOfMemoryError: Requested array size exceeds VM limit

if the application tries to create an array on the heap that is bigger than the total heap size.

If there is not enough memory in the method area for creating a new class, the Sun HotSpot implementation gets an error in the permanent generation:

Exception in thread "main": java.lang.OutOfMemoryError: PermGen space

Both kinds of OutOfMemoryError occur very often in real life and the reasons for them are very different and will be covered in later blog entries.

OutOfMemory errors in thread exclusive memory areas occur less frequently and are identified by the following error messages in the Sun HotSpot JVM:

Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread

Exception in thread "main": java.lang.OutOfMemoryError: <reason> <stacktrace> (Native method)

The first error is thrown if there are too many threads in the JVM and there is not enough memory left to create a new thread. I’ve seen this because the memory limits of a process have been reached (especially in 32bit operating systems, e.g. on Windows 32bit it is 2GB) or the maximum number of file handles for the user that executes the java process has been reached. The second error message indicates that a memory allocation error on a native stack (JNI method call) has occured.

It is also interesting that a memory allocation error on the JVM stack (too many frames on the stack) does not throw an Java OutOfMemory error but as the JVM specification mandates: java.lang.StackOverflowError.

The last variant of the OutOfMemoryError that I know of is

Exception in thread "main": java.lang.OutOfMemoryError: request <size> bytes for <reason>. Out of swap space?

This error is thrown if there is not enough memory left on the operating system level – which is normally true if other processes are using all of the available memory or the swap space is configured too small.

This first blog entry of the Java OutOfMemoryError series covered the basics of the Java Memory Model. In my point of view it is essential to know the different memory areas of the JVM and its functions if you want to understand why a java.lang.OutOfMemoryError occured in your application. I hope that I have made clear that there can be many variations of this error with totally different possible causes. There are a lot of open questions about when and why theses errors occur and how we can monitor and analyze memory problems in our applications. This is exactly what the next episodes ot this Java OutOfMemoryError series will cover.

One day every professional Java programmer will suffer from this exception thrown by his Java application: java.lang.OutOfMemoryError.

With a little luck this error will occur during development of the application – if not and it happens in a production environment, you will quickly get a lot of unintentional attention by your management, as a Java OutOfMemoryError normally means that you had an application crash! The reasons for these memory problems in the Java Virtual Maschine are versatile and in many cases not easys to analyze. In our troubleshooting services we see workarounds for these errors every day: Servers are preventevly restarted every night as the heap gets closer and closer to its limits or – even worth – the application regulary crashes with an java.lang.OutOfMemoryError indicating that the hunger of the JVM for more memory was bigger than the available heap.

As 64bit JVMs are getting more and more available in production environments, this seems as a perfect solution for theses kind of problems, as more adressable memory means bigger heaps and less frequent crashes. This is why I see that heaps in production are moving to the GB-area without thinking about the consequences: Less performance and troughput, because of more Garbage Collection overhead and pause times. Even worth if the Java OutOfMemoryError had nothing to do with the heap size…

Our performamce study has shown that memory problems are under the most critical problems with regard to stability and performance of Java applications. Almost 50% of the 250 questioned Java specialists answered, that Java Memory Management is critical for the performance and stability of their application. I think that it is time to write a comprehensive guide to the Java Memory Model inclusing tools and methods for anlyzing Java OutOfMemory errors and this is what this blog series is about.

I am calling this series “Java OutOfMemoryError – A tragedy in seven acts”, as I liked the definition of “tragic” in the German Wikipedia entry for “tragedy”, as it says that in the context of a tradegy the word “tragic” is not used in the sense of “sad” but in the sense that somebody gets blameless guilty.

In my opinion this is often true for developers that are responsible for Java OutOfMemoryError – they get blameless guilty. In my troubleshooting sessions I often see that managers are more interested in finding the person who is responsible for the problem than in finding the reason for the problem – this is sad and I hope that I will help the readers of this blog to better understand the reasons for OutOfMemory errors and give them the right knowledge to avoid them or if they have already occured, to quickly find and resolve them.

This is an overview of my Java Memory Tragedy:

1. Act- The Java Memory Model.

The first part of this series will introduce the memory architecture inside the Java Virtual Maschine in detail and will explain where and when java.lang.OutOfMemoryError can occur.

2. Act – Java Memory Leaks et al.

The second part of the series will cover the reasons for java.lang.OutOfMemoryError in detail – especially Java Memory Leaks and high, temporary memory consumption.

3. Act – JVM Monitoring and Configuration

The third part of the series will describe the parameters to configure the JVMs  memory areas and show tools to monitore these parameters at runtime.  As the parameter are different between JVM vendors and implementation, this blog entry will focus on the Sun HotSpot JVM.

4. Act – Creating and understanding Java Heapdumps.

The fourth part of the series will show how to generate Heapdumps with the Sun HotSpot JVM and how to analyze these dumps.

5. Act – Runtime Analyzis of the Java Heap.

The fifth part of the series will cover tools and methods to analyze objects on the Java Heap at runtime, especially to find high memory allocation hotspots and slow growing memory leaks.

6. Act – Typical Java Memory problems

The sixth part of the series will show you typical Java OutOfMemoryError scenarios in practise. I will show you real examples and describe how to analyze and fix these problems by applying the tools and rules of the last 5 parts of this series.

7. Act – Summary and Outlook

The last part of the series will summarize the whole series and will give you an outlook to the next tragedy I am planning to write: The Java Garbage Collector.

I am really happy to write this series in the next weeks, as I am writing a book on performance related topics together with Alois Reitbauer from dynaTrace and I hope to get a lot of feedback to these blog entries, which are partely on the same topic as in the book.

We at codecentric are not only leaders in Java performance, but support our clients on a variety of software development challenges.

I posted in a previous entry about Comparison of PHP and Java. I said that PHP mainly lacks appropriate tooling to develop debug and maintain. To be fair with PHP this posting will today describe how to profile PHP applications to tune performance. We will see how this tooling compares to the Java ones.

For Java developers JProfiler should be a well known tool for profiling and analyzing performance of Java applications. De equivalent in PHP is called Xdebug Xdebug + CacheGrind. Xdebug is recording the calls to the PHP scripts, while WinCacheGrind is then used afterwards to evaluate the collected Data. In this example we are goung to profile a command line call of the symfony Framworks. Because I am commiter for this framework, I use Xdebug from time to time to ensure a good nonfunctional quality.

First you need to download the appropriate ‘.dll’ (or ‘.so’ for Linux) extension from the Xdebug website and put it to the other extensions into the ‘php/ext’ directory. To enable tat extension you need to add Xdebug to the ‘php.ini’ Xdebug and tell it where it should put the profiling results:

zend_extension_ts="d:\dev\php\ext\php_xdebug-2.0.3-5.2.5.dll"
xdebug.profiler_output_dir="c:\xdebug-profiling"
xdebug.profiler_enable=1

Take care to edit the correct ‘php.ini’ file. There is one for the command line calls usually found in the PHP directory and another one in the Apache directory for the PHP processes spawned by Apache. Additionally you need to create the target directory first because Xdebug will not create it if it is not existing yet.

Invoking ‘php -m’ from the command line should output something including:

[Zend Modules]
Xdebug

Now we are going to profile a simple call. For example one to the symfony sandbox to create an application:

php symfony generate:app codecentric

After this call has ended an 8MB ‘cachegrind.out.4896′ been created for us. It now can be loaded using WinCacheGrind, which looks like this:

Unfortunately WinCacheGrind has only limited functionality. KCacheGrind allows some more settings and graphical representation of the data, but also is not as complete as the Java tools we usually use. It also does not contain a memory view, which might be due to the fact that the PHP memory management is pretty messy. Quite useful though is is the timing information displayed.

On the ‘Overall’ tab you can find code which took in sum the most time of your application. This is usually the code where you can make the easiest improvements. For the symfony task we can find here the following:

Function  : sfFinder->search_in
Avg. Self : 0,2ms
Avg. Cum. : 5,1ms
Total Self: 127ms
Total Cum.: 3214ms
Calls     : 629

you can see that the search_in function takes an average of 0,2ms, but methods invoked by it add another 5,1ms in average. Because this function is called 629 times, this gives you the total time spent. A good candidate for checking if it really should be called that often, and if yes, if a single call can be made slightly faster.

As you can see, also without a lot of expertise for performance issues, one can use above described tools to optimize PHP application performance. Perhaps in some time in future PHP and its tools will be as sophisticated as it is already today status quo in Java.

I’ve seen a lot of MemoryLeaks caused by wrong usage patterns of ThreadLocal variables. As I had this problem in one of our own applications yesterday, I want to blog about this case.

ThreadLocal variables are used in Java to bind a variable to the current Thread - each thread will have its own, independend instance of that variable. These kind of variables are normally used to hold status information like the user, that are needed in the entire application and you don’t want to have in every method signature.

The lifecyle of ThreadLocal variables is directly interconnected to the lifecyle of its corespondig thread. If the thread is terminated and collected by the Garbage Collector, its coresponding ThreadLocal variables will also be good candidates for Garbage Collcetion.

Memory problems will mainly occur if ThreadLocal variables are used inside of Java EE applications running in an application server.  Application Servers manage threads by using thread pools to safe resources and boost performance. (See configuration of Tomcat 6 Http Conncector as an example). If for example a HttpServletRequest is send to the ServletEngine of the Application Servers, a free worker thread is taken out of the pool and is associated with the processing of the servlet’s application logic. If the servlet or its called Java classes are using ThreadLocal variables, these variables are associated with the current worker thread. If the servlet is finished and the response is sent to the client, the associated thread is put back to the thread pool, so that it can handle another request. That means that the Thread object and its associated ThreadLocal variables are not garbage collected, as the thread object is still alive.

Depending on the number of threads in the pool (> 100 threads are normal in productive environements) and the size of the object in the ThreadLocal variable, critical memory problems can occur. If for example 200 threads are configured for the thread pool and the ThreadLocal variables are 5MB big, this could result in 1 GB of heaps space occupied by just these ThreadLocal variables. This will lead to an GC overhead and can crash the JVM with an OutOfMemoryError.

In our concrete case the servers had to be restartet every day to avoid OutOfMemoryError problems. To analyse the problem we took a heapdump during runtime  (see our recent blog entry how to do this). I did the analysis of the dump with Eclipse MAT which showed very quickly that we had a ThreadLocal memory problem.

The above screenshot shows the Dominator Tree of the dump. I’ve marked 6 Threads each occupying about 14MB of the heap.

The next screenshot shows the details of one of these threads. The high memory consumption is the result of a ThreadLocal variable that references a DOMParser and its parsed document with a total size of 14MB. Because MAT can also show the content of the objects, we could easily see that the parsers were caused by our WebServices. The WebService classes were generated by JBoss WS. Consulting Google showed that this was because of a Bug in jbossws version 1.2.0, which was fixed in 1.2.1: “DOMUtils doesn’t clear thread locals”. So we had to patch our JBoss AppServer with the latest jbossws release and the problem was gone.

This exampe shows that you have to be careful if you use ThreadLocal inside an application server. It is essential to take care that the ThreadLocal variables are dereferenced if they are not used anymore. We normally use ServletFilters to do this job.

Today Christian and I fought hard with a misterious connection problem. As a part of a longer process a web service call was failing from time to time. Actually it failed only under load. Due to that we did just debugging outputs, rather than breakpoints, as we were afraid of breaking some runtime behaviour by stopping the threads. But that was not so trivial as we would liked it to have: The webservice client is based upon the  Apache Commons HTTPclient. The webserive itself was realized in Struts2 using its Restful2ActionMappers. Both got several services injected via Spring.

Initially we were after the unusual large size of the request and response, but after some analysis it seemed ok, however we still got flooded by variations of:

java.io.IOException: CRLF expected at end of chunk: 72/84
java.io.IOException: Bad chunk size: somexml

As we realized that this has to happen on the HTTP transport level, we had a look around and found following spring config for our bean:

<bean name="restClient" class="de.codecentric.framework.RestClient">
  <property name="httpClient">
    <bean class="org.apache.commons.httpclient.HttpClient" />
  </property>
</bean>

Doesn’t look bad, but by default the HttpClient has a ConnectionManager which will return always the same Connection instance. Even to different threads.  Once you get to know this you will find immediately

org.apache.commons.httpclient.MultiThreadedHttpConnectionManager

which unfortunately did not help much. In addition we now go an

java.io.IOException: connection closed

What might suprise you is that MultiThreadedHttpConnectionManager is multithreading capable, but does only allow 2 connections per host. After we did increase that out application was running fine again. Even under load.

The Spring config does look similar to this now:

<bean name="restClient" class="de.codecentric.framework.RestClient">
  <property name="httpClient">
    <bean class="org.apache.commons.httpclient.HttpClient">
      <property name="httpConnectionManager">
        <bean class="org.apache.commons.httpclient.MultiThreadedHttpConnectionManager" destroy-method="shutdown">
          <property name="params">
            <bean class="org.apache.commons.httpclient.params.HttpConnectionManagerParams">
              <property name="defaultMaxConnectionsPerHost" value="20" />
            </bean>
          </property>
        </bean>
      </property>
    </bean>
  </property>
</bean>

Leasson learnt: When multiple threads are using the same UrlConnection, it can go fine, but more likely, some very obscure things (exceptions) can (will) happen