Last week I had the pleasure attended Heinz Kabutz Java Specialists Master course to sharpen my Java skills. Java Champion Heinz, is a great trainer who manages to combine anecdotes, hard facts and deep Java knowledge with engaging exercises to a well done course. The scope was the whole spectrum of Java, but focusing on the details you normally do not use, or know how to use. Some of the material he already published as part of his newsletters, which are read all around the world.

Let me share my impressions on the course with you in this day by day review…

Day 1

The course started with discussing Threads, and how we should use them. Quite a complex topic for early morning. We played with a ThreadGroup that became a custom Thread pool. ThreadGroup is not the best designed class. Heinz called it a child’s drawing from the early years of Java. I found using the java.util.concurrent Locks really easy. Finally gone are the days of synchronized(). Before the lunch break, Heinz showed us his laws on concurrency, which he also presented in an abbreviated form at our meet the experts – performance event. We came across this code:

private boolean running = true;
public void dojob() {
  while(running) {
    // do something useful
 }
}
public void shutdown() {
  running = false;
}

When running in a Server VM (with java -server), this might never stop, due to optimizing running would be inlined by HotSpot with being true all the time. To avoid this you would have to make it volatile. Because I asked about debugging, we tried stopping there and the debugger showed us: running = false. Still the code continued to executed, because the debugger sees the correct field value, but the running code doesn’t. It gets more interesting with this code:

public void doJob() {
  boolean myRunning = running;
  while(running){
    // do something useful
    myRunning = running;
  }
}

When looking with the debugger we saw this:

running = false; myrunning = true;

however the loop still looped. But when forcing the line to execute via F7, code terminated. This can be a nightmare to debug, so it is good to know what you should take care of when writing multithreaded programs.

Also something to remember is to check

if (Thread.interrupted()) {
  throw new InterruptedException()
}

as first code in all methods declaring an InterruptedException.

We learned about CompletionService looks like an interesting interface for mass processing of asynchronous work. So, who needs closures?

Day 2

We started with Java new (yet another new?) IO, which brings quite a lot of new features, but somehow is not as widely used as it should be. One reason might be that it can easily get much more complicated to use. Perhaps one should try easy examples before writing an Async nonblocking server (which you can after attending the course ).

FileChannel fc = new RandomAccessFile("test.txt", "r").getChannel();
MappedByteBuffer buffer = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());

Second half was on understanding Java memory management. This of course is less often really applicable, but understanding it is quite critical to solving problems. We had a look at stuff like caching and pooling, and why this creates leaks or loitering objects. This reminds me of my older post about tag pooling in servlet containers. I never understood really why tags are pooled, and even worse, do not have proper lifecycle methods to detect this when using an IDE. We used jVisualVm and HPjMeter to watch the GC at work.

Day 3

On day 3, I learned some interesting inner mechanics of Collection classes I did not use before (like PriorityQueue), as well as some nasty tricks on classloading. Heinz explained java.lang.reflect.Proxies really well, and once understood, using them was not that difficult. Actually the best instruction is in the JavaDoc, but you must know how to read it:

Foo f = (Foo) Proxy.newProxyInstance(
		Foo.class.getClassLoader(), new Class[] { Foo.class },
		new InvocationHandler() {
		  public Object invoke(Object foo, Method method, Object[] arguments) throws Throwable {
		    return method.invoke(foo, arguments);
		  }
		});

In the afternoon we discussed about Exceptions, and I made up my mind on checked vs unchecked exceptions. Personally I will use unchecked exceptions for developer/programming errors. Catching them is not required, app may crash – Developers should fix this. However everything which is related to the Environment the app runs ins should work with checked exceptions. Ideally they provide sensible information, not just a message. Also quite important: Just rethrow Exceptions! Found yourself not able to decide what to do with an InterruptedException? Well just rethrow it And handle it in the Thread code (calling interrupted() and exiting a loop). I never did that often, because I don’t like polluting my method signature, but it should be considered. Do not be afraid of retrowing Exceptions.

Day 4

The last day of the course started with a tough performance optimization exercise. The tough part was that we were not allowed to improve the code until we had all the numbers written down and eliminated any overhead of testing code. I especially liked this, because it thought me how eager I am sometimes fixing issues that I forget proving them first. As kind of side note, we discussed the various modes the JVM can run in and found out how slow java -Xint is.  After having sped up the code down to 10% of its initial runtime, we moved on to Date and Time, which was a bit short chapter. I can recommend using jodatime and icu4j, and try to stay away from java.util.Date. Before the end of the course we covered logging including some nifty tricks. The most important lesson about logging is that you need to use code guards (which was not new to me, but I like the term, I never heard before):

if (log.isDebugEnabled()){
  log.debug(complexObject.toString() + expensive.toString());
}

Wrap-Up

I can wholeheartedly recommend this course. 4 days packed with lots of information and exercises that are well done so that they are a challenge for every participant. You should have worked with Java already some time. This is definitely not a beginners course. You will rush past topics which you can only grasp when you have experienced the problem before. I can also recommend taking this training in German, because Heinz has a really funny accent

In 2001 Andrew Hunt and David Thomas introduced a benchmark for modern developers with their book  „The Pragmatic Programmer“. The focus of their work is on practises, tools and design-paradigms to help programmers produce better code.In my point of view being a “Pragmatic Programmer” is not enough for the challenges of agile projects. This blog entry takes a closer look at the developer in the team of a Scrum project. I explicitly distinguish between “programmer” and developer” because the word programmer tends to focus more on coding and less on all the other activities needed for implementing an application. (e.g. testing)

According to Ken Schwaber (who formulated the inital version of Scrum together with Jeff Sutherland), Scrum was mainly inspired by an article of the Harvard Business Journal that was published in January 1986: „The New New Product Development Game“. The article analyzed the criterias of successful new product developments of companies like Honda and IBM. One of the main assumptions of this article for successful product development is:

„Stop playing the relay race – play rugby“.

For software development this would mean that the old sequential approach (aka: waterfall) doesn’t work anymore and we should start using a holistic approach.

Scrum has taken the basic characteristics of successful product development teams like self-organizing project team, overlapping development phases and subtle control and integrated them into the Scrum process. Scrum combines them with ideas of iterative, incremental development and lean principles.The  Scrum Alliance provides training and certification for Scrum Master and Product Owner – but why not for the developer? Does’t the success of a project mainly depend on the team? Are there any implications on the skills of a developer by changing the work model to self-organization with overlapping development phases or by delivering high-quality software every 2-4 weeks?

My personal opinion is that agile methodologies like Scrum fundamentally change the way developers work. Therefore new skills and attitudes are required.

Let’s start by looking at one of the most important changes in agile projects: self-organizing project teams. Self-organizing means that there is no “command-and-control” paradigm anymore, where (project) managers tell the developers which work in which time sequence should be implemented. Self-organizing often sounds like stress-less freedom for the team – but often turns out to be a very tough job.

The team takes the responsibility for delivering high quality software in time by committing to the Sprint goals. This implies that team members have to discuss, solve conflicts, organize work, estimate effort, pull needed information and commit to team goals. In “classic” projects this is done by defined roles like project managers and in many cases developers have no experience how to deal with these challenges. The success of agile projects depends on good self-organization teams – I’ve seen managers quickly return to “command-and-control” if this doesn’t work out as expected. So in my point of view organizations have to provide developers with training and coaching to get the needed social skills. Today I see that these kind of trainings and coaching are only provided to management and leadership positions. By inverting “command and control” we also have to rethink the training strategy for skills like team building, coaching, training, negotiating, decision making, problem solving, active listening and discussing/debating – our developers will need them to organize themselves in an agile team.

It could be also useful to provide trainings to help team members understand the strengths and weaknesses of their colleagues to support the team building – methodologies like  Team Management System and DISC can help teams to improve.

When we started with agility at codecentric we quickly found out that we had to train not only technologies and tools but also these kind of social skills, if we want to improve our customer satisfaction even more. So we hired an external trainer and coach to develop a social skill program for our developers to sustainable improve their skills in that area – the feedback of the participants is great and we see a lot of improvement in the teams.

The second new paradigm you get with Scrum is: overlapping development phases – like self-organizing this requires new ways of thinking for developers. In the classical, sequential process models, all the requirements are documented by writing detailed specs, filling out use-case templates and provide UML diagrams which exactly describe the whole system (ok, I am a little polemic). With these specs in place the developer could start coding and if he was a good developer, he tested his code. But he shouldn’t bother too much with testing as the next team in the process would do this for him: quality assurance.

In agile projects these strict constraints are disposed and the responsibilities of a developer are widened. In my point of view developers will have to deal a lot with defining and talking about the requirements (function and non-functional) and testing if the code fullfills these requirements. This means that Unit-Tests are not enough and functional integration and acceptance tests will be needed – normally these test will be defined and implemented together with the domain experts and experienced testers in the team. But especially for automated testing a developer should be involved from the beginning to code and design for automation – e.g. to provide data-setup and deal with time and dates in the tests. For me, automated regression tests are a must-have if you are developing in short iterations – otherwise the test effort can explode. Especially when Test-Driven Development is introduced, the developers will need training and coaching to get their tasks done. Today test-driven unit tests can be found in some projects but test-driven integration- and acceptance tests are really rare.

The developer will have to communicate a lot with the domain experts to transfer the elements of the user stories into good tests. Good knowledge of the domain will be also more important as the detail level of user stories are much lower than of classical use cases.  This means that developers will need very good communication skills and a toolset of practises like Domain Driven Design. Today developers concern more about frameworks like Spring and Hibernate than with the business domain; generators  that transfer an Entity-Relationship-Model into a Domain-/Object model are a nice gimmick but more or less useless! The generator will not help to get a good structured domain that can be used as a communication bridge between development and the domain experts. DDD calls a good domain model UBIQUITOUS LANGUAGE – One team, one language! In agile teams, where domain experts, testers and developers are working in one team on the same tasks and stories, it is essential to have one language. So in my opinion we need to sharpen the domain and design skills of the developers, so that they can communicate with the domain experts in a clear and simple manner.

To deliver software every 2-4 weeks a developer should also have good knowledge of common agile practises and tools.  Continuous Integration (e.g. Hudson, CruiseControl), automated build tools (e.g.. Maven, Ant), code quality analyzers (e.g. Findbugs, PMD, Checkstyle),  test automation frameworks (e.g. JUnit, Robotframework, easyMock, DBUnit, JMeter), version control systems (e.g. git, Subversion), bug tracking (e.g. Jira, Trac), documentation tools (Wiki, UML, Javadoc) are just examples of the modern, agile software developers portfolio. These practises and tools have to be trained, adopted and optimized so that they can be used for higher productivity and high quality results. Dave Thomas, Co-Autor of „The Pragmatic Programmer“, defined so called „Code Katas“ to practise what is needed to be a professional developer.

In software we do our practicing on the job, and that’s why we make mistakes on the job. We need to find ways of splitting the practice from the profession. We need practice sessions.

So despite the training, we also have to give our developers time to practise – for example to learn TDD as seen in this Kata by Arne Roock.

In addition to all these skills, the developers have to be trained to understand the overall process and concepts. Today the focus of Scrum trainings is on Scrum Master and Product Owner (Ken Schwaber is changing this with the Scrum Developer) – as most of the meetings and artifacts are mainly for the “team”, we should sharpen their understanding of Scrum (which is also part of the ScrumMasters job). For example the team should learn that the main idea behind Daily Scrum is not to report to the Scrum Master (who does not even need to attend the Daily Scrum) but to synchronize with the other team members and make impediments transparent. I’ve seen a lot of Daily Scrums where the ScrumMaster moderates and asks each team member the three questions and updates the Scrum board for them! Also the Burndown Chart is an instrument for the team to track the progress of the team – it is not used as a control-instrument for the Scrum Master or Product Owener (or even managers). Anyway: Team members have to be trained and coached to understand Scrum and the agile concepts.

This all shows that organizations that are moving to agile methods like Scrum will have to change. The skill-set of the team and especially developers is changing and the spectrum of required skills will demand for more training, practise and knowledge transfer. „The New New Product Development Game“ claimed that: „Multilearning“ and „Organizational Transfer of Learning“ are two of six critical success factors for good product development teams. Agile organizations will have to accept that training is not an incentive but a duty for all team members of agile projects. The invest in the employees will pay off for sure!

1. Maven

Instead of maintaining a plethora of start scripts manually, you can execute jybot directly from within maven. The following pom.xml shows the details:

• The maven-dependency-plugin copies all dependant libraries into a folder, which is later included into the classpath. Now you can easily use any third party library.
• With the exec-maven-plugin you can invoke ‘jybot’ in the integration-test lifecycle phase. This requires that jybot is in your PATH of your environment. Luckily, the .sh or .bat can be omitted.
• The classpath is configured to
  

  ${project.build.directory}${file.separator}classes${path.separator}
  ${project.build.directory}${file.separator}dependency${file.separator}*

  This ensures that it will include

  1. The compiled classes of your own Java keywords
  2. All jar libraries that the pom.xml lists as dependency. (The wildcard notation is only available with Java 6!)
• Afterwards, the jybot script is parametrized:
  • -C off turns off the colored output
  • –variable … sets some variables. Developerfriedly defaults are set in the pom.xml’s properties section. Is the environment variable ‘env’ set to ‘ci’, the ci profile will be activated, which changes those variables to make use of the real test system.
  • –variable RESOURCES:${project.basedir}/src/main/robot/resource Setting a RESOURCES variable makes it possible to refer to resources from your testcases simply with ${RESOURCES}/keyword-file.txt.
  • –exclude inprogress Excludes test cases, that are currently in progress. A developer can work on a test case until it is fully working, and can commit parts of his work, without having the test case executed by default. Of course, once the testcase is fully automated and functional, that tag must be deleted..
  • –include author-* In my opinion, all tests should me tagged with an author. This includes all tests that are existing, but also offers an easy way for the developer to limit the test execution to a set of test, that he is currently working on. This could be further improved with maven properties and profiles. There’s currently a bug in Jython that expands any parameter to matching filenames. (Jython Issue 1567) – so there should be no file that matches author-*!
  • ${ROBOT_TESTS} This tells robot which tests to execute. By default this refers to all tests in ${project.basedir}/src/main/robot/suite. By using tags and folders you have two independant ways to structure and manage your test cases and select a set for execution. This should be enough flexibility for everybody.

On the long term, it would be nicer to have a full-blown maven plugin for the robot framework, in order to save the awkward script parameterization. Until this is done, the presented solution should be sufficient.

<project>
...
  <build>
<plugins>
<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <version>2.1</version>
        <executions>
          <execution>
            <id>get dependencies</id>
<phase>pre-integration-test</phase>
            <goals>
              <goal>copy-dependencies</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
<plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>1.1.1</version>
        <executions>
          <execution>
            <id>robot</id>
<phase>integration-test</phase>
            <goals>
              <goal>exec</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <executable>jybot</executable>
          <environmentVariables>
            <CLASSPATH>${project.build.directory}${file.separator}classes${path.separator}${project.build.directory}${file.separator}dependency${file.separator}*</CLASSPATH>
          </environmentVariables>
          <arguments>
            <argument>-C</argument>
            <argument>off</argument>
            <argument>--variable</argument>
            <argument>SYSTEM_UNDER_TEST:${system-under-test}</argument>
            <argument>--variable</argument>
            <argument>DB_PORT:${db.port}</argument>
            <argument>--variable</argument>
            <argument>RESOURCES:${project.basedir}/src/main/robot/resource</argument>
            <argument>--outputdir</argument>
            <argument>${project.build.directory}/robot</argument>
            <argument>--output</argument>
            <argument>${project.groupId}-${project.artifactId}-${project.version}-output.xml</argument>
            <argument>--log</argument>
            <argument>${project.groupId}-${project.artifactId}-${project.version}-log.html</argument>
            <argument>--report</argument>
            <argument>${project.groupId}-${project.artifactId}-${project.version}-report.html</argument>
            <argument>--exclude</argument>
            <argument>inprogress</argument>
            <argument>--include</argument>
            <argument>author-*</argument>
            <argument>${ROBOT_TESTS}</argument>
          </arguments>
        </configuration>
      </plugin>
    </plugins>
  </build>
<profiles>
<profile>
      <id>ci-common</id>
      <activation>
<property>
          <name>env</name>
          <value>ci</value>
        </property>
      </activation>
<properties>
        <db.port>1234</db.port>
        <system-under-test>testservername</system-under-test>
      </properties>
      </build>
    </profile>
  </profiles>
<properties>
    <ROBOT_TESTS>${project.basedir}/src/main/robot/suite</ROBOT_TESTS>
    <db.port>4711</db.port>
    <system-under-test>localhost</system-under-test>
  </properties>
</project>

2. Project Structure

The above quoted pom.xml results in Eclipse in the depicted project structure below – either with the m2eclipse-Plugin or by running the eclipse:eclipse goal of the maven-eclipse-plugin.

3. Starting Robot

Executing the automated tests is now just a matter of executing the appropriate maven lifecycle phase:

4. Selenium Server

If you test Web UIs with robot, it is nice to integrate also the selenium server as external tool into Eclipse:

5. Full Workflow

By means of a concrete Example, a short screen cast should give you an impression, how to develop test cases in eclipse and how to execute them. Since details are hardly seen in the embedded video, there also a fullscreen version of it.

Every player had to play one round of each of the games: Agricola, a pretty complex strategy game, Stoneage, a round based resource gathering game, Dominion, an interesting card game, in which players compile a custom card deck according to the available cards and their strategy, and Heckmeck am Bratwurmeck, a pretty simple dice-rolling game, which trains you calculating probabilities.

• First round of the Tournament, sporting Dominion went really well for our Team. 2 first places and a second one were great. Only one game was lost due to the high amount of curses played by witches.
• Agricola strained our nerves for over two hours. It was pretty intense, and we were unable to oppose our co-players, who were well prepared. We tried both, the farmer and rancher strategy, but only achieved a second, 2 third and a fourth place.
• After a short snack-break, we had some relaxing dice rolling activities with Heckmeck am Bratwurmeck. But it seems gambling is not our profession. 3 times a fourth place was really sub-par. The other third place could not compensate those losses.
• Last game of the day was Stoneage. 3 of us tried the “starvation” strategy, which was working fairly well, but only brought us 2 third and a fourth place. The one victory was done with some more mainstream farming and building strategy. Overheard discussions tought us that most players already have overcome “starvation” and came up with something better.

Despite the not very successful results, we had a fun day! And lots of motivation for the coming year. Perhaps we should then run a board game training bootcamp?

Yesterday evening now the release 1.0 of the Database Library was finished and it is of course really great that it is already listed on the Robot Framework homepage. As with every project the interesting part starts when people start using it and I am looking forward to feedback for additional features. And with Fabian one of my colleagues joined the project as an additional committer. This way we are well prepared for further development and potential bugfixing.

The implementation of the database-access is based on JDBC, which really feels like an anachronism nowadays as we are used to Hibernate and JPA. But for this library it fits perfectly fine and the possibility to support different databases by their (class)names makes is very easy to support various different databases. And thanks to HyperSql everything can be perfectly tested using JUnit.

Of course there are already ideas for further development. Here especially the support for the Robot Framework Remote Library is very interesting as it allows to distribute the execution of test libraries on different servers.

I have pretty high expectations, as I already read his newsletter since quite some time and also my impressions of Heinz, when he gave his talk at our meet the experts, were very positive. He really knows all the nasty details about Java, and I hope I can get some of those out of his and into my brain. The course promises “Extreme Java”. I am looking forward to very esotheric subtleties without any practical value. And of course to many caveates that you indeed would find in everyday work.

I am also looking forward to meeting other Java geeks to exchange knowledge and have a good week. Will you be there?

(note that the course is in German)

I think it is really hard to find out who is right or wrong … actually I think it is sort of impossible. Even within our small company we are not able to get a common agreement about what exactly architecture is and what design is. And if it is not possible to get to an agreement within a few people (even if some of them are strong characters … ), how should we come to an agreement within the whole IT community? It is probably impossible.

Thus, I will not try to find a precise answer for that question. Instead I will try to approach the topic from a different angle. First, why is it so hard to answer the question? I think a big part of the problem is that there is no satisfying definition for both terms available, neither for architecture nor for design. So, whenever you talk with another person about architecture or design you can almost be sure that he or she will talk about something different than you do.

I mean there are *lots* of definitions available but most of them are so specific that they do not capture the topic sufficiently or they are so vague that they do not help at all. Just for instance, Robert Martin (who said and wrote a lot of very smart things, btw) once stated “Architecture is about the important stuff. Whatever that is.”. Now, this means everything and nothing. Does it mean that design is about the unimpotant stuff? Probably not. And, if coffee is really important for the developers (which it usually is), does it mean, that architecture is about coffee? Probably not. As you can see this definition does not really help.

I will not go into detail for other definitions because I think it will not lead anywhere. It won’t be possible to find a definition for architecture and design that everybody will agree upon *and* that will help us to figure out the difference between architecture and design. Especially I think it won’t be possible to find a clear separating line between architecture and design, where the one thing ends and the other thing starts. Everybody draws his line somewhere else and many people probably rather have some kind of fuzzy cloud in mind instead of a sharp line. “Somewhere within that cloud architecture stops and design starts” they will say.

Okay, but there must be something that might help us. At least there should be something …

Maybe it does help to step away from that question for a moment. From my experience I figured out that there are two basic tasks that make up that whole architecture and design thing (without trying to find the separating line). The whole thing is about transforming all the requirements that are around into a result. (Btw: If you would ask where the coding is in there, I think I would leave it up to you to find the separating line between design and coding; I think it is a similar question … )

The first part of that transformation task is to understand all the requirements (not just the business and user requirements, but also the requirements of the mangement, the project management, the development team, operations, system planning, the security department, the testers, and so on), really understand them, to balance them (there are always conflicting and inconsistent requirements across the different domains and stakeholders to deal with) and to find an appropriate solution idea for all those balanced requirements. I usually call this part of the work “alignment”.

The second part of the transformation into a working solution consists of turning the solution idea into an appropriate structure that supports maintainability and changeability, and refine the structure over and over again until you are down at the code level without losing the properties described before. I usually call this part of the work “structuring”.

Now, is it important to distinguish those two tasks? I think, yes! The reason for this is that I figured out that those two tasks require very different skills. While alignment has much to do with communication and conflict management skills, structuring requires more formal strengths. Also alignment requires kind of a broad knowledge about all the different requirement domains while structuring requires more of a deep knowledge about the solution domain. As a consequence, you will find quite few persons who are capable (or willing) of doing both tasks with the same strength, and if you neglect one of the two tasks you will most likely get an unsatisfying result.

Okay, getting back to the initial question. Will alignment and structuring help us to find a separating line. I think, sort of …

From my perspective, alignment is definitely part of any good architecture while structuring is definitely part of any good design. You cannot be a good architect without doing a lot of alignment and you also cannot be a good designer without doing a lot of structuring. How much of structuring is required in architecture and how much of alignment is required in design, I think that is mostly a matter of taste and there won’t be a definite answer to this.

So, I also cannot provide you with “the” answer to the initial question. But I know, if somebody talks about architecture without talking about alignment he does something wrong and if somebody talks about design without talking about structuring he also does something wrong. And that is the most important thing from my point of view …

PS: My apologies to Robert Martin for picking on his definition of architecture. As I wrote before he said and wrote a lot of really smart things, even though I do not agree with all of them. So, if you have the opportunity to read some of his stuff or listen to a presentation of him, just do it! Now, that should be enough of indemnification …