Mirko gives a really good description why AppDynamics delivers the right solutions for todays distributed architectures in his Post “Troubleshoot Java in production – introducing AppDynamics Lite“. If you have not read it yet, head over to it now.

The key promise of AppDynamics is simplicity. Easy installation, easy operation and very fast results. In fact this sounds uncommon and hard to believe. But this is indeed the case, as proven by the following uncut screencasts.

Installation of AppDynamics is fast and easy like pie. In this screencast I will be showing to you how to perform the installation in less than 3 minutes (there is a full screen mode in the lower right corner):

Directly after the installation you can start investigating your performance issues. In the following screencast I show you the basic functionality and a possible workflow for finding the root cause of a slow response (there is a full screen mode in the lower right corner):

Finding the culprit

To find out whats wrong the best is to create a heap dump at peak usage to show whats consuming all the memory.
After opening that dump in the wonderful Eclipse MAT you might get a picture like this:

Ouch. 1.8 GB out of 2GB are consumed by sessions. I usually then filter by “StandardSession” for apache sessions to be able to easily browse around the sessions that have high amount of retained heap.

Ouch again… 10MB per session, this cannot scale. Now it could of course be possible that some careless programmer put in much of our data into the session, but thats simply not true, as we can easily find out:

Thats interesting. Almost all of our memory is consumed by AjaxStateHolders. So what does this actually do?

How JSF and A4J work

I try to keep it simple. Before JSF actually renders HTML to the users browser, it builds an internal representation. For each page (or view) this internal representation (called component tree) is created and run through the lifefcycle. Any user events are processed. If there is a component where a user can select one of 3 items, those 3 items are loaded and a string representation put onto the page. The component keeps track of which item is selected.

Now we do not want the user to submit the whole page to select another item, which would cause the component tree to be recreated and a different item to be selected. We want Ajax! For that A4J remembers the state of the components when the page was displayed. So it knows the component tree and those 3 items, and which one was selected. On AJAX actions the component is looked up, the state modified by selecting another item and the partial HTML representation sent back. This is also called partial page rendering.

So how does the remembering work? Well you might have guessed it: It creates a AjaxStateHolder in the user session and attaches the component tree to it.

How this can get big

Ok, this might be the current page. I can have big views but it is only one. No thats unfortunately not true. A4J does store more than one view. But why? Look at your browser, you are quite likely to find a “back” button somewhat in the to-left corner. When you click that you see the last page. But the browser did not send a request, so the server does not know that you are on that page. Imagine you click AJAX functionality: Where should the server get the component tree from for that time you have been on the page? Easy answer: This is as well in the AjaxStateHolder. By default this tracks back 16 views. and because you might come across the same view in your browsing history more than once there are up to 16 variants of a view.

As you can see this can get big. And remember, this is per user.

How to fix

Unfortunately there is no fix. This is how JSF works, and how A4J works.

There is a ticket on the Richfaces bug tracker which describes this: RF-3878 – Session memory leak. And the best answer is: reduce the number of views you store.

<context-param>
 <description></description>
 <param-name>com.sun.faces.numberOfViewsInSession</param-name>
 <param-value>1</param-value>
 </context-param>
<context-param>
 <description></description>
 <param-name>com.sun.faces.numberOfLogicalViews</param-name>
 <param-value>1</param-value>
</context-param>

You will loose the browser back button functionality, but gain a lot of memory. Second way to fix is to reduce the size of the component tree, which is most likely affected by big lists of data and complex structures. However this is easier said than done.

You can see the long way to our goal. But we are close to it. You can see that I would like to get a reward for the fantastic sprints we did, so I painted a little cake right after the finish line.
Below the path leading to the finish, I depicted our architecture and the actual goal of the project. I really like how well the architecture evolves and works our for us, and also like that our customers start to realize that their confusion was not necessary and they can realize their ideas wit it.
At the top left, there is a bad test stability. It really can be improved and we actually started already addressing this. Right below that you can see our burndown for that iteration, which is not bad, but should have run below the line instead above it.
It seems that I was as well pretty pleased with the amount of features we did create and the value (meaning money) it can generate for our customers. last but not least I draw the great sprint demo we did for our customers the day before.

So how do you like my picture (or doing this kind of retrospective at all)?

Here is an often ignored part from java.text.SimpleDateFormat JavaDoc:

 * Date formats are not synchronized.
 * It is recommended to create separate format instances for each thread.
 * If multiple threads access a format concurrently, it must be synchronized
 * externally.

Perhaps that warning isn’t strong enough, because it reads: “It is recommended”.
This leads to this highly dangerous line of code I have seen in almost every project and is likely to produce hangs in concurrently accessed code.

private static final SimpleDataFormat DATE_FORMAT = new SimpleDateFormat("yyyy.MM.dd");

But it gets worse with System.gc().
There is no warning in the JavaDoc:

* Calling the gc method suggests that the Java Virtual
* Machine expend effort toward recycling unused objects in order to
* make the memory they currently occupy available for quick reuse.
* When control returns from the method call, the Java Virtual
* Machine has made a best effort to reclaim space from all discarded
* objects.

Doesn’t sound too bad, does it? You should wonder what “suggests” and “best effert” really means. Perhaps you are tempted to use System.gc(), so you might look on the internet for other people using this, hoping for an explanation what in reality happens. You might find good insights on Stackoverflow. But what exactly this does nobody seems to know.

I think a picture tells more than thousand words:

We are looking at a pretty big system of one of our customers. It has 14GB of heap and we helped them to bring down the memory consumtion so that the app ran quite well. However there were still pretty long pause times (so called stop the world time). The graph was created from a Garbage Collection log and points us to the cause of the pause times. The blue line shows Garbage Collection activity. So why are there exactly 13 spikes with up to 35 seconds? All the other activity seems to be really fast.
The reason for those 13 spikes is simple: They all were caused by an invocation of System.gc().

So what happens? Calling System.gc() seems to effectively stop any optimization the GC is doing, forcing a full collection to start, rather than the optimized CMS Algorithm used.

Note: This system is using a Sun 1.6.0.18 64 bit HotSpot Server VM. GC is set to -XX:+UseConcMarkSweepGC and -XX:+UseParNewGC. System.gc() does not have to produce results like that. Your results might vary. However also others have reported, that when using CMS and System.gc() a Full GC is always performed.

If you cannot find yourself invoking System.gc(), they might hide deeply in a used library. To fix this, you can use the the server argument “-XX:+DisableExplicitGC”. This argument is also recommended by the SUN/Oracle performance tuning guide.

Some people say: cosmic rays! But we know that is not true. To efficiently utilize a CI system without the need to troubleshoot a long time here some common issues we encountered and ideas how to mitigate them.

• A test might do some time calculation and either in the test or in the code under test time is taken twice. While most of the time there is no difference, there might be a microsecond sometimes. A good indicator for this is a message like: Time was 23:30:00 but expected 23:30:00 (note microseconds are not shown)
• Code, Testcode or Testrunner/CI code might leave files behind. Sometimes these are log files, sometimes files produced as test output. Take the time to search the whole server for all file writes made during a test run and take care to have a cleanup in place. Don’t forget to add disk space monitoring, because build machines have hard discs that tend to get full. (Hudson can do both)
• Users logging into the CI system might lock resources, like files or ports, or do anything bad to the machine. You should not allow user logons.  All “analysis” should be made read only. Note that also read access or parallel tests can lock files.
• It is not necessarily a bug when your code or tests do not run when the system date is 1th of January 1970. It could be, but you should make sure the system always uses the current time. Set up a ntp daemon. If you must use specific points in time for testing, you should be able to set a time source for all of your code, like a spring bean called TimeProvider which normally resorts to the system time.
• If your tests need to apply evil hacks to test your software (which might be required. If not get rid of the hacks) it is often safer to let test execution fork, so tests cannot introduce side effects via the JVM (like setting System properties). Code coverage tools using bytecode manipulation count as hacks.
• If you have multiple build machines ensure that they are as similar as possible. If you can afford it, you can set up a farm of build machines with defined differences, so you can spread testing on varying hardware in case this cannot be simulated. You do not want any surprise differences on which you spend hours to find out.
• Consider setting up the system under test for integration tests nightly from scratch. Those tests tend to get messed up by exploratory testing and ad hoc demos. It is easier to automate such stuff than one would think, though it takes some time.
• If you do automatic deployment, you need to at least stop the server, copy changed artifacts and then restart the server. Any kind of “hot deployment” is unfortunately just to fragile for reliable results.
• After doing any change to configuration or infrastructure of your CI system trigger a build manually. If not done the next normal developer checkin will cause a failing build and will leave the dev wondering how that change could break this stuff.
• If you find your tests hanging in your code, especially if multiple tests were run at the same time, take a heap and thread dump of the JVM before restarting the tests. You might be lucky that you found by accident a real concurrency issue inside your code. You should be grateful for that because you hardly can deliberately test this.

Yes it is possible that the build breaks without any issues in your software, but it wastes a lot of time on investigation. Know the weaknesses of your system and try to fix them or at least document them.

We have an issue with one of our integration test suites, which connects to an external service. Sometimes this just hangs and results in a connection timeout. Until a while ago this always “broke” the build. The result of this was that every engineer had a look at the build, the logs and eventually found out it was a timeout on the external system- We discussed this and decided to stop wasting time on investigating this. So we added a mechanic that in this special timeout case the build does not turn red. It stays green, but creates a tag “timeouted” on the test. This of course has a problem. A green build with “timouted” can have a problem with the computation of the extern call results. It might be really red. But we cannot know this. Real green build are not allowed to have “timeouted” tests. But the important part is that we want “real red” builds, which turn only red when there is a issue we can fix. In RobotFramework you can define a third state for “noncritical failures”. Decide yourself if this is something for you.

But the most important takeaway is: If a broken build is not caused by test or production code, you must find the reason and address it. You cannot say: “cosmic rays” because that will lead everybody to say “broken build – cosmic rays” and you will have much less successful builds because eventually no one will care. Red should always mean: Team take action!

Because we believe in Open Source, here the sourcecode of our comment-merging plugin. If required one could also make languages configurable, but we did not need that, and it would have introduced some more complexity.

WPML comment merging plugin on wordpress.org

<?php
/*
Plugin Name: WPML comment merging
Plugin URI: http://www.codecentric.de
Description: This plugin merges comments from all translations of the posts, so that they all are displayed on each other. Comments are internally still attached to the post they were made on.
Version: 1.0
Author: Fabian Lange
Author URI: http://blog.codecentric.de/en/2010/06/wordpress-wpml-comments-filter-plugin/
License: MIT
*/
 
function sort_merged_comments($a, $b) { 
  return $a->comment_ID - $b->comment_ID;
}
 
function merge_comments($comments, $post_ID) {
	// get all the languages for which this post exists
	$languages = icl_get_languages('skip_missing=1');
	foreach($languages as $l) {
		// in $comments are already the comments from the current language
		if(!$l['active']) {
			$otherID = icl_object_id($post_ID, 'post', false, $l['language_code']);
			$othercomments = get_comments( array('post_id' => $otherID, 'status' => 'approve', 'order' => 'ASC') );
			$comments = array_merge($comments, $othercomments);
		}
    	}
	if ($languages) {
		// if we merged some comments in we need to reestablish an order
		usort($comments, 'sort_merged_comments');
	}
	return $comments;
}
 
function merge_count($count, $post_ID) {
	// get all the languages for which this post exists
	$languages = icl_get_languages('skip_missing=1');
	foreach($languages as $l) {
		// in $count is already the count from the current language
		if(!$l['active']) {
		        $otherID = icl_object_id($post_ID, 'post', false, $l['language_code']);
                        $otherpost = get_post($otherID);
			// increment comment count using translation post comment count.
                	$count = $count + $otherpost->comment_count;
        	}
	}
        return $count;
}
 
add_filter('comments_array', 'merge_comments', 100, 2);
add_filter('get_comments_number', 'merge_count', 100, 2);

The idea

When the developer created HTML and CSS for a component (like a login box) and made sure that it was looking fine in all required browsers, the developer would take a screenshot, and crop it to contain only the box. Like depicted here on the right, we would want to make sure that the logo inside the red box looks exactly like that.
Next the automatic build needs to be told on which page this box is expected to be (e.g. a page with a user that is not logged in). The test takes then a screenshot and searches it for the expected box.
We considered also telling at which position we expect it, but that would cause multiple issues: The dev needs to find the pixel position. It needs to be absolute on different screen sizes, changes are hard to track.
When thinking about this, we believe that finding the expected sub-image in the screenshot is sufficient. If you would want to make sure it is at a specific position you just include neighboring objects in your expectation image.

The Implementation

We wrote a very simple keyword that takes a screenshot of a given URL (and possibly after performing steps like putting stuff to a shopping cart) and checks the occurrences of template screenshots. The keyword is implemented in Robot BDD using Selenium.

*** Keywords ***	
 
Match Styles ${Styles}
    Execute Javascript  window.resizeTo(100,100)
    ${ID} =  Get Unique ID
    Capture Page Screenshot  ${OUTPUT DIR}${/}StyleTest-${ID}.png  css=
	:FOR  ${style}  IN  @{Styles}
    \	Match Style  ${CURDIR}${/}styles${/}${style}.png  ${OUTPUT DIR}${/}StyleTest-${ID}.png
    Remove File  ${OUTPUT DIR}${/}StyleTest-${ID}.png

The Javascript helps in bringing down the file size of the screenshot and makes searching faster, as it compresses unused whitespace which occurs in many liquid layouts. Get Unique ID generates us an ID using Java UUID.

The keyword is simple:

public static void matchStyle(String expectedFilename, String actualFilename) throws IOException {
	boolean check = StyleChecker.check(expectedFilename, actualFilename);
	if (!check) {
		throw new IllegalStateException("could not find expected image in actual screenshot");
	}
}

The actual picture searching is done in Java. The code is not superclever, but after a few rounds of optimization pretty fast (worst case: up to 300ms when expected part is not found, in real use cases averages about 200ms for existing sub-images). In fact the screenshot generation is the bottleneck.
Java code can be downloaded and is excluded here for brevity.

A test can look like this:

*** Keyword ***
Behaviour
    [Arguments]  ${Url}  @{Styles}
 
    Go To  ${Url}
    Match Styles @{Styles}
 
| *Test Case* | | *Url*                        | *style*          |
| 1 | Behaviour | http://${BASE_URL}${CONTEXT} | Login | ShoppingCart-Empty|

Caveats

1. The screenshots need to be lossless.
2. Our code is pixel perfect. Which means that every tiny detail is important. Because of that Developers cannot take screenshots of their own to crop, but need to take the screenshot that is produced by selenium. There are differences, for example clear type, or available fonts.
3. When implementing picture comparison in Java you can make it horribly wrong. I hope my code is not that bad. At least I benchmarked it and for the simplicity of the algorithm it performs fine.

Benefits

With those simple steps we are able to do regression testing on our user interfaces that have strict standards. It helps even more, because CSS is sometimes a bit fragile. And because of the simplicity of the testsetup it is also very easy to add new test withing seconds.

Debugging

What happens, if a style test fails? More often than not, the analysis is trivial, because style bugs that should be found with these kind of tests (wrong borders and margins, colors don’t match, wrong image, etc.) can be quickly detected by looking at the screenshot. But right now we had the situation, where this was not the case, and immediately suspected a refactoring gone bad. We were wrong! In order to detect differences in two images, you can use GIMP’s layer mode “difference”;

• Load screenshot (1) in GIMP
• Load image (2), that should be found in the screenshot, in GIMP
• Lay image (2) as new, half-transparent layer over the screenshot
• Apply mode “difference” to the new layer
• Merge the visible layers
• Brighten up the darker image parts, to see the differences.

So far for the short tutorial for debugging. In case you run into troubles, please comment.

<https:connector name="autoProxy">
  <spring:property name="proxyHostname">
    <spring:bean class="org.springframework.jndi.JndiObjectFactoryBean">
      <spring:property name="jndiName" value="/jndi/config/proxyHost" />
      <spring:property name="defaultObject" value="" />
    </spring:bean>
  </spring:property>
  <spring:property name="proxyPort">
    <spring:bean class="org.springframework.jndi.JndiObjectFactoryBean">
      <spring:property name="jndiName" value="/jndi/config/proxyPort" />
      <spring:property name="defaultObject" value="0" />
    </spring:bean>
  </spring:property>
  <service-overrides dispatcherFactory="de.codecentric.mule.HttpAutoProxyMessageDispatcherFactory" />
  <https:tls-key-store path="/mule.keystore" keyPassword="changeit" storePassword="changeit"/>
  <https:tls-server path="/mule.keystore" storePassword="changeit"/>
</https:connector>

As you can se we configured a Connector which creates this MessageDispatcherFactory, it passes the configuration obtained from JNDI to it and also adds a keystore for ssl connections to it (with the great default password changeit )
Using it is then straightforward:

  <outbound-endpoint address="https://external.service/endpoint" synchronous="true" connector-ref="autoProxy">

The Factory itself is dead simple:

public class HttpAutoProxyMessageDispatcherFactory
  extends AbstractMessageDispatcherFactory {
  public MessageDispatcher create(OutboundEndpoint endpoint) throws MuleException {
    return new HttpAutoProxyMessageDispatcher(endpoint);
  }
}

The HttpAutoProxyMessageDispatcher implementation is easy was well. And contains a few hardcoded hosts that should not be proxied. Feel free to improve that part:

public class HttpAutoProxyMessageDispatcher
  extends HttpClientMessageDispatcher {
 
  private final boolean hasProxy;
  private final String proxyHost;
  private final int proxyPort;
 
  public HttpAutoProxyMessageDispatcher(OutboundEndpoint endpoint) {
    super(endpoint);
    this.proxyHost = ((HttpConnector) endpoint.getConnector()).getProxyHostname();
    this.proxyPort = ((HttpConnector) endpoint.getConnector()).getProxyPort();
    this.hasProxy = StringUtils.isNotBlank(proxyHost);
  }
 
  @Override
  protected HostConfiguration getHostConfig(URI uri) throws URISyntaxException {
    String host = uri.getHost();
    HostConfiguration config = new HostConfiguration();
    config.setHost(host, uri.getPort(), Protocol.getProtocol(uri.getScheme().toLowerCase()));
    if (hasProxy && !isLocalhost(host)) {
      config.setProxy(proxyHost, proxyPort);
    }
    return config;
  }
 
  private boolean isLocalhost(String host) {
    return "localhost".equals(host) || "127.0.0.1".equals(host);
  }
 
}

When you applied this pattern you only need to make sure JNDI information regarding the proxy is correct on each environment, and the mule configuration will automatically adapt to it.

So lets have a look at our Webservice Interface. To make it a little more interesting, we make a nontrival service, which takes and returns domain objects (but be aware of the http request size on large object trees)

@WebService
public interface RecommendationService {
	@WebMethod
	public Products recommendProducts(
		@WebParam(name="user")
		User user,
		@WebParam(name="genre")
		Genre genre
	);
}

There is an implementation for this service existing as well. The next step is then to wire it up in mule.
First we need to configure mule to accept webservice calls. Because we start mule in an WAR file, we use the servlet connector, but you can use the jetty connector as well when running standalone:

<servlet:connector name="servletConnector" 
                   servletUrl="http://localhost:8080/mule/services/recommendation?wsdl" />

Next up is the config of the service itself using cxf:

<model name="recommendationServiceModel">
	<service name="recommendation">
		<inbound>
			<cxf:inbound-endpoint address="servlet://recommendation" synchronous="true" />
		</inbound>
		<component>
			<spring-object bean="RecommendationService" />
		</component>
	</service>
</model>

And of course the service:

<spring:bean id="RecommendationService"
             class="de.codecentric.RecommendationServiceImpl" />

While you can mash all into one file, I recommend to split your mule and component configuration to multiple files, so that you do not get lost in the masses of xml. You could separate them by type (service, component, mule config) or by service.
That is it already for the mule configuration part, so lets try to invoke it. Because we do not have an easy way to pass the domain objects yet, we just try to read the wsdl to veryify it is working.

http://localhost:8080/mule/services/recommendation?wsdl

Watch out for any s which tell you that the listing is not complete, but is available on a seperate url geven as attribute in the import.

Generating a java client for accessing the service is very easy using the wsdl2java command from CXF:

wsdl2java -client -d src/main/java -p de.codecentric.client 
  http://localhost:8080/mule/services/recommendation?wsdl

If you would be an external party, you now could work with the stuff that has been generated. Internally however you most likely would prefer continue working with your domain objects User, Products and Genre. This will help you dealing with updates happening in your developmentcycle and provide domain methods you implemented on the model, but are not generated. Because CXF is really smart we can just delete the following generated classes:

• Genre
• ObjectFactory
• package-info
• Products
• User

Fix the imports by using your domain objects instead and delete the @XmlSeeAlso({ObjectFactory.class}) reference.

This should leave you with the interface and implementation of the service and two wrapper objects for request and response, and a dummy client. Running the dummy client (with CXF on the classpath) should now invoke the webservice.

What it does behind the scenes when you are using

RecommendationServiceImplService ss = new RecommendationServiceImplService(wsdlURL, SERVICE_NAME);
RecommendationService port = ss.getRecommendationServiceImplPort();

is that it creates a dynamic proxy using reflection from the remote wsdl.

We could stop here now. We have a dynamic Webservice client that uses the domain objects. All is fine, but the performance really sucks.

The WSDL is read over the wire, and translated to the proxy class. We could add the WSDL locally, but that would require downloading it everytime the domain objects change. External clients of course need to do that, but we want to be less affected by the changes incremental development introduces. Also still the proxy class generation would be slow. We measured the wall time spent in the whole stack, and proxy generation by far outnumbered every other code.

To improve this we create a pool, using commons pool GenericObjectPool.

private final GenericObjectPool recommendationServicePool;
 
RecommendationServiceFactory recommendationServiceFactory = new RecommendationServiceFactory();
recommendationServicePool = new GenericObjectPool(recommendationServiceFactory, new Config());

So the pool needs a factory to make instances and a configuration. The confguration can be tweaked, but defaults should be fine for now.The factory implementation is straightforward:

public class RecommendationServiceFactory implements PoolableObjectFactory  {
public Object makeObject() throws Exception {
  RecommendationServiceImplService service = new RecommendationServiceImplService();
  RecommendationService port = service.getRecommendationServiceImplPort();
  return port;
}
public boolean validateObject(Object arg0) {
  // consider all controllers valid objects
  return true;
}
public void destroyObject(Object arg0) throws Exception {}
public void activateObject(Object arg0) throws Exception {}
public void passivateObject(Object arg0) throws Exception {}
}

Now we can invoke our service like that:

RecommendationService port = (RecommendationService) recommendationServicePool.borrowObject();
try {
  Products products = port.recommendProducts(user, genre);
} finally {
  recommendationServicePool.returnObject(port);
}

Please do not forget to return the Service you have borrowed.

Wrap up
We used MuleESB to configure and deploy a Spring component based Webservice which uses Domain objects. We exposed the service using Apache CXF and used it to generate a client as well. Afterwards we tweaked the generated client to use our domain objects instead of generated clients. Then we introduced an object pool to avoid creating the proxy classes over and over again.

perhaps you want to know if there was a real performance gain. I strongly advise to profile it yourself. The simplest method is to measure milliseconds orund our last codeblock, executing it more than once. The very first invocation of the whole stack took 360ms on my machine. All subsequent calls were down to 4-6ms. This is a 100x improvement. Remember, that this time includes a lot: Calling over HTTP (even on localhost) a ESB running inside a WAR on JBoss, finding the correct service endpoint based on the URL, invoking an instance of that Service, and all the way back.

Do not judge premature. Web Services can be pretty fast. And thanks to a lot of frameworks, also easy to setup.

Let me share my impressions on the course with you in this day by day review…

Day 1

The course started with discussing Threads, and how we should use them. Quite a complex topic for early morning. We played with a ThreadGroup that became a custom Thread pool. ThreadGroup is not the best designed class. Heinz called it a child’s drawing from the early years of Java. I found using the java.util.concurrent Locks really easy. Finally gone are the days of synchronized(). Before the lunch break, Heinz showed us his laws on concurrency, which he also presented in an abbreviated form at our meet the experts – performance event. We came across this code:

private boolean running = true;
public void dojob() {
  while(running) {
    // do something useful
 }
}
public void shutdown() {
  running = false;
}

When running in a Server VM (with java -server), this might never stop, due to optimizing running would be inlined by HotSpot with being true all the time. To avoid this you would have to make it volatile. Because I asked about debugging, we tried stopping there and the debugger showed us: running = false. Still the code continued to executed, because the debugger sees the correct field value, but the running code doesn’t. It gets more interesting with this code:

public void doJob() {
  boolean myRunning = running;
  while(running){
    // do something useful
    myRunning = running;
  }
}

When looking with the debugger we saw this:

running = false; myrunning = true;

however the loop still looped. But when forcing the line to execute via F7, code terminated. This can be a nightmare to debug, so it is good to know what you should take care of when writing multithreaded programs.

Also something to remember is to check

if (Thread.interrupted()) {
  throw new InterruptedException()
}

as first code in all methods declaring an InterruptedException.

We learned about CompletionService looks like an interesting interface for mass processing of asynchronous work. So, who needs closures?

Day 2

We started with Java new (yet another new?) IO, which brings quite a lot of new features, but somehow is not as widely used as it should be. One reason might be that it can easily get much more complicated to use. Perhaps one should try easy examples before writing an Async nonblocking server (which you can after attending the course ).

FileChannel fc = new RandomAccessFile("test.txt", "r").getChannel();
MappedByteBuffer buffer = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());

Second half was on understanding Java memory management. This of course is less often really applicable, but understanding it is quite critical to solving problems. We had a look at stuff like caching and pooling, and why this creates leaks or loitering objects. This reminds me of my older post about tag pooling in servlet containers. I never understood really why tags are pooled, and even worse, do not have proper lifecycle methods to detect this when using an IDE. We used jVisualVm and HPjMeter to watch the GC at work.

Day 3

On day 3, I learned some interesting inner mechanics of Collection classes I did not use before (like PriorityQueue), as well as some nasty tricks on classloading. Heinz explained java.lang.reflect.Proxies really well, and once understood, using them was not that difficult. Actually the best instruction is in the JavaDoc, but you must know how to read it:

Foo f = (Foo) Proxy.newProxyInstance(
		Foo.class.getClassLoader(), new Class[] { Foo.class },
		new InvocationHandler() {
		  public Object invoke(Object foo, Method method, Object[] arguments) throws Throwable {
		    return method.invoke(foo, arguments);
		  }
		});

In the afternoon we discussed about Exceptions, and I made up my mind on checked vs unchecked exceptions. Personally I will use unchecked exceptions for developer/programming errors. Catching them is not required, app may crash – Developers should fix this. However everything which is related to the Environment the app runs ins should work with checked exceptions. Ideally they provide sensible information, not just a message. Also quite important: Just rethrow Exceptions! Found yourself not able to decide what to do with an InterruptedException? Well just rethrow it And handle it in the Thread code (calling interrupted() and exiting a loop). I never did that often, because I don’t like polluting my method signature, but it should be considered. Do not be afraid of retrowing Exceptions.

Day 4

The last day of the course started with a tough performance optimization exercise. The tough part was that we were not allowed to improve the code until we had all the numbers written down and eliminated any overhead of testing code. I especially liked this, because it thought me how eager I am sometimes fixing issues that I forget proving them first. As kind of side note, we discussed the various modes the JVM can run in and found out how slow java -Xint is. After having sped up the code down to 10% of its initial runtime, we moved on to Date and Time, which was a bit short chapter. I can recommend using jodatime and icu4j, and try to stay away from java.util.Date. Before the end of the course we covered logging including some nifty tricks. The most important lesson about logging is that you need to use code guards (which was not new to me, but I like the term, I never heard before):

if (log.isDebugEnabled()){
  log.debug(complexObject.toString() + expensive.toString());
}

Wrap-Up

I can wholeheartedly recommend this course. 4 days packed with lots of information and exercises that are well done so that they are a challenge for every participant. You should have worked with Java already some time. This is definitely not a beginners course. You will rush past topics which you can only grasp when you have experienced the problem before. I can also recommend taking this training in German, because Heinz has a really funny accent