The context of this session is a container management system, which is rather big, speaking about 200 person years effort. The model that was used in order to create the test cases included scenarios, steps, adapters, components, and simulators.

I am skipping much of the content, once because I really typed a lot in the last days, but even more because Sersik and Schröder gave a good overview over how to design and create integration tests with the above model, which was interesting but too much to copy everything here. Slides including comments will be shared, and if possible I will update this blog and include a link.

Key points: both, scenarios and steps can and should be parametrized. Also the adapters have to be parameterized to take care of different configuration that vary from integration test environments amongst themselves or over time. To automate the test execution, starting from 2002, a time when the Robot Framework was not existing yet, and neither was FitNesse (and if it was they didn’t know about it), they created a framework on their own: iValidator was created (in an agile fashion of course). This is pretty much how Robot Framework started out within Nokia Maybe it will also be open sourced at some point?

• How many children have been in your family?
• What was the most challenging thing you faced as a child? (pick something that is not too painful)

At least that exercise was really bringing some noise and action to the room as people started discussing right away . But what it shows: You really get to know something, one visitor was saying: More interesting than a lot of talks as such. This technique can also be used for example as a starter in a retrospective where no one wants to start “talking”.

A lot of ideas came from the book “The Fifth Dicipline”, which is on my wishlist definitly after this talk.

“We are having a dualcore brain, but we are using only half of it!” (Declan on the left and right side of our brains.)

It continues with the idea of Shu Ha Ri which means translated in short Following, Breaking Away, Fluent. Probably I will have to do some googling on this still.

“In the beginner’s mind there are many possibilites, in the expert’s mind there are few.” by Shunryu Suzuki

Exercise: Draw a hand. The interesting thing is that hardly anyone in the audience was looking at a hand physically instead a mental model has been used, which is limiting us to some extend.

“We have a goal, but often not a purpose.”

The purpose is what really motivates people. There was still a lot more, but I was too much absorbed to make any meaningful notes.

From the discussion it was turning out that having a “learning culture” might be difficult to have in some organisations. But google was given as a concrete example where this has been extremely successful and of course I have to say that I am in the lucky position to work for a company where learning is also really valued.

You should have a Definition of Done on different levels, for Tasks, Stories, Sprints and Releases. For example:

Definition of Done for a Story:

• all code checked in
• all unit tests pass
• …

Definition of Done for a Sprint

• all story plus
• PB updated
• performance testing
• architectue diagrams updated
• all bugs closed
• …

Definition of Done for Release to INT

• Installation Package created
• operations guide updated
• trouble shooting guide updated
• …

Definition of Done for a Release to PROD

• stress testing
• performance tuning
• network diagram updated
• …

This is starting to look a lot like milestone exit criteria, documents to update and create, because they are on the checklist. In my opinion, many of the tasks from Release to Int/Prod should actually be tasks within the Sprint, when the document needs to be updated, and not something that happens after and outside the Sprint, as the same people are needed to do it.

Interesting figures: 25% of the teams post the DoD on the wall and 25% of the teams are getting the most out of agile, of course that does not mean that they are the same 25%, but there should be some overlap.

The test plan can easily be derived from the Definition of Done and the Product Risk Analysis. A good point that Jimmink makes was that with the Definition of Done in place, you could possibly spot lacking skills. Also the Definition of Done is a team commitment, even a company commitment and best practice. If the DoD is violated, everybody should care. After some points about what the value of the DoD is, how it can be introduced and what the role of the tester is, the session concludes with the finding that a quality standard should not be optional, and teams should embrace the concepts.

Good points in the discussion that followed the session:

• Mitigate the risk that you blindly check off the points in the higher level DoDs (Sprint / Release), by taking a probably existing company policy as imput, but agree with the team and customer on what is really needed from that.
• Don’t be afraid to take things off of the DoD
• Revisit it regularly in the Sprint Retrospective. Do we still meet it? Does it make sense?

Definition of Done for a Story:

• all code checked in
• all unit tests pass
• …

Definition of Done for a Sprint

• all story plus
• PB updated
• performance testing
• architectue diagrams updated
• all bugs closed
• …

Definition of Done for Release to INT

• Installation Package created
• operations guide updated
• trouble shooting guide updated
• …

Definition of Done for a Release to PROD

• stress testing
• performance tuning
• network diagram updated
• …

This is starting to look a lot like milestone exit criteria, documents to update and create, because they are on the checklist. In my opinion, many of the tasks from Release to Int/Prod should actually be tasks within the Sprint, when the document needs to be updated, and not something that happens after and outside the Sprint, as the same people are needed to do it.

Interesting figures: 25% of the teams post the DoD on the wall and 25% of the teams are getting the most out of agile, of course that does not mean that they are the same 25%, but there should be some overlap.

The test plan can easily be derived from the Definition of Done and the Product Risk Analysis. A good point that Jimmink makes was that with the Definition of Done in place, you could possibly spot lacking skills. Also the Definition of Done is a team commitment, even a company commitment and best practice. If the DoD is violated, everybody should care. After some points about what the value of the DoD is, how it can be introduced and what the role of the tester is, the session concludes with the finding that a quality standard should not be optional, and teams should embrace the concepts.

Good points in the discussion that followed the session:

• Mitigate the risk that you blindly check off the points in the higher level DoDs (Sprint / Release), by taking a probably existing company policy as imput, but agree with the team and customer on what is really needed from that.
• Don’t be afraid to take things off of the DoD
• Revisit it regularly in the Sprint Retrospective. Do we still meet it? Does it make sense?

Cecile starts with asking about what characteristics programmers have, and claims these are the ones (not sure where this is leading, yet)

• Intelligence
• Creativity
• Responsibility
• Collaboration
• Flexibility

Some well known keywords now follow: Agile Manifesto; Triangle of Cost (fixed), Time (fixed) and Scope (can vary); More about the People; Self-Organizing; Skills & Expertise.

Interesting point now, comparing trust in agile and traditional environments. Traditional: team has to trust the management for making the right decisions. Agile: Management has to trust the team. Ok, so it’s basically the known Command & Control vs. Self-Organizing argument.

Pitfalls for agile testing are … so far an unfinished mindmap and Cecile is looking for more input from the audience. So far the map contains keywords like: Team (involvement, skills, knowledge, competencies), Tester (skills, knowledge, competencies, attitude), Stakeholders (involvement, lack of knowledge, integration, commitment), General (little waterfalls, test environment), Environment (maturity, investment, culture). So there’s a lot missing. The separation of Testers from the Team is only to highlight some special pitfalls and not for saying they are not part of the team.

A Test Maturity Matrix, a long list of criteria, for which you can score on various level (controlled, mature, optimizing) (download from Sogeti). From these sixteen key areas, these six were selected from the facilitating and encouraging mindset, which support agile development environment:

• Test Strategy: What, how, risk analysis, priorities.
  Suggestion for improvement: Set up and maintain a regression test.
• Tester Professionalism: Team responsibility, open mind, knowledge, skill, competency, motivation, culture
  Suggestion for improvement: Discuss the role of testing within the team and define the specific skills needed.
• Degree of Involvement: Moment, Degree
  Suggestion for improvement: Let testers be involved in unit testing (should give good benefits to programmer and tester).
• Test Environment: Delivering working software, greater impact, short iterations.
  Suggestion for improvement: Try to be as much self-sufficient as possible: a lot of environmental issues can be solved more easily.
• Testware Management: Responding to Change, maintainability, transferability, reusability
  Suggestion for improvement: Make maintainability, transferability and reusability of testware part of the test strategy.
• Stakeholder Commitment: Continuous involvement, feedback, availability
  Suggestion for improvement: Gather metrics to show the costs and benefits of testing efforts

After Davis made her points on how to improve the testing process, she kicks off a discussion and asks for feedback how we should improve the testing. Ideas generated were:

• Start with the mindset of the people
• Make it part of the regular retrospective, so be sure that optimizing the testing process is what would currently bring most benefit

And the real answer is: “It depends, buy our consultacy services or at least buy the book ” There is a new revision soon to be published … but it has not been revised to take Agile into account Cecile starts with asking about what characteristics programmers have, and claims these are the ones (not sure where this is leading, yet)

• Intelligence
• Creativity
• Responsibility
• Collaboration
• Flexibility

Some well known keywords now follow: Agile Manifesto; Triangle of Cost (fixed), Time (fixed) and Scope (can vary); More about the People; Self-Organizing; Skills & Expertise.

Interesting point now, comparing trust in agile and traditional environments. Traditional: team has to trust the management for making the right decisions. Agile: Management has to trust the team. Ok, so it’s basically the known Command & Control vs. Self-Organizing argument.

Pitfalls for agile testing are … so far an unfinished mindmap and Cecile is looking for more input from the audience. So far the map contains keywords like: Team (involvement, skills, knowledge, competencies), Tester (skills, knowledge, competencies, attitude), Stakeholders (involvement, lack of knowledge, integration, commitment), General (little waterfalls, test environment), Environment (maturity, investment, culture). So there’s a lot missing. The separation of Testers from the Team is only to highlight some special pitfalls and not for saying they are not part of the team.

A Test Maturity Matrix, a long list of criteria, for which you can score on various level (controlled, mature, optimizing) (download from Sogeti). From these sixteen key areas, these six were selected from the facilitating and encouraging mindset, which support agile development environment:

• Test Strategy: What, how, risk analysis, priorities.
  Suggestion for improvement: Set up and maintain a regression test.
• Tester Professionalism: Team responsibility, open mind, knowledge, skill, competency, motivation, culture
  Suggestion for improvement: Discuss the role of testing within the team and define the specific skills needed.
• Degree of Involvement: Moment, Degree
  Suggestion for improvement: Let testers be involved in unit testing (should give good benefits to programmer and tester).
• Test Environment: Delivering working software, greater impact, short iterations.
  Suggestion for improvement: Try to be as much self-sufficient as possible: a lot of environmental issues can be solved more easily.
• Testware Management: Responding to Change, maintainability, transferability, reusability
  Suggestion for improvement: Make maintainability, transferability and reusability of testware part of the test strategy.
• Stakeholder Commitment: Continuous involvement, feedback, availability
  Suggestion for improvement: Gather metrics to show the costs and benefits of testing efforts

After Davis made her points on how to improve the testing process, she kicks off a discussion and asks for feedback how we should improve the testing. Ideas generated were:

• Start with the mindset of the people
• Make it part of the regular retrospective, so be sure that optimizing the testing process is what would currently bring most benefit

And the real answer is: “It depends, buy our consultacy services or at least buy the book “

Correctness can be confirmed at any time is the subtitle of Mary’s talk and it start off with the clear statement that Waterfall does simply not work. So let’s hear what works .

The question is how to deal with complexity and the answer to some extend is divide and conquer. Mary continues with some history on how to archieve this. Basically everything has been “structured” back in the 70s, which does also not really solve the problem as we know today. But what we really want is not to inject defects already from the very beginning. An early answer to this was given by Edsger Disjkstra which basically was a form of layered architecture.

After the long discussion about architecture with Tom Gilb yesterday night in the hotel lobby my fingers are not too calm while writing about architecture here … but back to Mary. The question is how has the software to work on various levels? E.g. unit level testing and business level testing.

Next in the history is Harlan Mills with his Top-Down Programming approach, which should help avoiding any difficult integration. It’s a kind of step-wise integration, which could be seen as an early version of continuous integration what we discussing today in agile. So as with the discussion wth Tom yesterday it seems that quite some agile ideas are not really that new.

Next in history is Dave Parnas and his “Criteria of Decomposition”. This immedietly reminds me that Tom was saying that too many archtecture are doing nothing but decomposition and this is not architecture, but this is again another story. The idea was to divide the program to ease future changes … so this comes close to the idea Uwe is promoting, which is “Architecture for Maintenance”. And sorry I have to admit here I was too slow to keep up with the hstorical view, but we will catch up. (At this point Andreas and me are pairing on this blog entry, very cool.)

1976 it is analysed that the main problem in software development is maintenance costs (again) and the solution to this problem is to find defects early! For this shorter development cycles are needed than in the known Waterfall process.

1981 Tom Gilb is hitting the scene with the concept of starting with a short summary of the measurable business goals of the system. Those must be unambigious, testable and must not contain design. Hmm, sounds very familiar as it is basically the core statement of his lecture yesterday. It is always very nice to get these kind of “cross connections”. So we are back at:

“Prevent defects injection in the first place!”

How to learn how to avoid defect injection? Right, let’s think of a “Defect Injection Process”. It starts with a Specification and one team writing the code and another team writing the Tests. Sorry, this does not match at the end and we have a defect injection process. Next idea: Combine writing the tests together with the specification and then write the code to implement the expactations imposed by the tests.

Back to “Conquering Complexity” and the question how to deal with complexity and the answer is divide by reposibilities and value. This evolves to the three rules of lean development:

• Design matters
• Correctness can be confirmed at any time
• Failure is a learning opportunity

Mary continues with some quite high-level examples how to develop a system. Of course she is making “bonus points” by referring to the “iPhone” as a good example for such a good design of an overall system.

Now it gets more concrete again with the question how much time of the release cycle is used for finding and removing bugs (system hardening)? The answer is that top companies are only using less than 10% for this while sometimes even 50% of the time in a release cylcle is used for this. But it cannot be the aim to use that much of your valuable development time for this, so find defects earlier in the process. And with this Mary comes back to the topic that you can confirm correctness of your software at any time and not only at the end of your release cycle.

“No correlation between size of the code base and bug rate: Complexity handled completely.” (from an example developing an embedded system)

Now we come to Agile: We have short and stable iterations. And the stakeholder can give and gets feedback every iteration. And this would not be called “Agile Testing Days” without a good example how automated tests have let to a very successfully project when IBM was introducing agile. Sounds like continuous integration and early automated testing, even without FitNesse and Robot.

“People do not want software, they want something else.”

Basically people want their top objectives implemented and thus problems solved. Mary is also promoting TDD to archieve this.

“The biggest defect is tolerating defects.”

And Mary is emphasizing that you must learn from failures as it indicates a lack of understanding. This is always an opportunity to learn. Anyway, in a very diciplined program only a few defects will escape.

Ok, as we are talking about lean here there must be some example from Toyota . Mary describes how Toyota is coming from the current condition to their vision. For this you need to overcome one obstacle with every step to come closer to your vision.

Andreas sums this up that a lot of this was already discussed on the tutorial on Monday, but it was good to have it repeated again.

Visited and written by: Andreas & Thomas

The answer to the question from the tile of this keynote is: Yes! As already partly in the tutorials yesterday Lisa has again emphasized how important it is that there are no silos in agile teams. Instead it is all about solving the given task (developing quality software) in a collaborative mode. To do so (agile) testers are as important as (agile) developers and all other potential members of the team.

Lisa Crispin: “We (testers) don’t break code, it comes to us already broken!” (im Original von Cem Kaner)

All in all Lisa’s keynote was very inspiring as she put a strong focus on agile values and ideas and how they help having more success in software projects and in the end also more fun.

Luckily other visitors of the conference are also blogging and thus we can keep it a bit shorter here – as this will anyway be a huge blogpost – and link to some posts on Lisa’s keynote.

• Gojko Adzic: Are agile testers different?
• Anthony Gardiner: Lisa Crispin – Are agile testers different?

Open Source Agile Testing by Péter Vajda

Visited and written by: Andreas

This session should have been called “How to not implement Scrum”. Péter Vajda presented his experiences as agile coach in the Nokia Maemo project, the linux based operating system for mobile devices of the future, like the well known and valued internet tablet N900.

Two years ago, 200 employees were part of the project, by now they grew to nearly 1000 people and 50 teams (which implies a teamsize of 20). While the gap between open source communites and a mobile device manufacturer has to be bridged somehow, it was decided to go agile and introduce Scrum, use XP practices and Lean thinking.

Beside some well known facts (scrum teams should be co-located), some interesting facts were shared. For example they realized in Sprint 20 that unit tests alone do not suffice, automated acceptance tests were nice but not existing. What to do? Take a break and create acceptance tests for all functionality from the last 20 Sprints or leave that functionality uncovered? The compromise was good from the content, having a “refactoring backlog item”, but also if you have “system user stories” you should always make clear what the value is. And the value of that is not the test itself, but that you gain confidence and flexibility to integrate new features more easily.

The demo, which showed how to test a Qt calculator with cucumber (the real new toy / gadget could not be disclosed, unfortunately) at last brought some new insight: With cucumber you can write behaviour tests, which not only support the format “given / when / then”, but also support a postfixed table with additional test data. An idea I was exposed to already at Agile 2009 in Chicago, but there in the context of FitNesse: One problem with given-when-then… the answer… “Other Examples Include”. I think that this is really a great idea to describe behaviour in the appropriate context and provide additional test data / examples.

Agile Testing: A Report from the Frontline by Joke Hettema und Ralph van Roosmalen

Visited and written by: Thomas

Joke is starting her presentation with a short introduction why the company she is working for has switched to agile methods for their projects. Here a common pattern was repeated that traditional methods could no longer give the required visibility as the projects became more complex. To solve this Scrum has been introduced as an agile method and mixed teams have been setup with developers, testers and architects in one Scrum team. Up to here not too much new I would say.

With respect to Testers and Testing some patterns from the tutorials and Lisa’s keynote are repeated: No explicit testteams anymore, shared responsibilities for developing and testing the software. But one special thing here is setting up a “Scrum of Scrums” for all testers of all the different teams only. Of course information sharing is important, but for me this goes into the wrong direction as it again draws a borderline between testers and developers instead of bringing them together. This borderline is getting even bigger by establishing special Sprint planning meetings for testers only where testing tasks are planned independently from the (other) development tasks. After a question from one visitor it also turns out that developers are only writing unit tests, but for all further testing they rely on the testers to implement those. Again this sounds as if there is still quite some borderline between testers and developers inside the teams.

Then Joke describes the problem of “Mini Waterfalls”. Those are happening still too often for them due to the fact that developers are not checking in their work frequently enough, pilling up bigger chunks of work that are then committed in one go. On this topic I can only recommend Fabian’s very good post on committing frequently that he has written some weeks ago.

When it comes to sprint planning their is again a level of indirection I personally do not like. This comes due to the fact that only some “Functional Specialist” is talking with the product owner and is then talking to the team instead of having the team talk directly with the product owner. A similar question came up yesterday in Elisabeth’s tutorial and here statement was clearly that as many team members should be “allowed” to join the sprint planning meeting as possbile. In addition I personally doubt that it is a good idea to have domain knowledge bundled only in a specifc role. But we are slowly leaving the area of agile testing, so back to topic.

Joke continues by showing the documents that are produced and the tools that are used for testing, which – again – does not make me feel too agile: Test Strategy, Risk Matrix, Test Approach Document, Checklist, Session Tester and Bug Reporting Software. With respect to the “Definition of Done” she points out that it is very often the “task” of the testers to tell the rest of the team that the definition of done is violated while the rest of the team is arguing against this. It also seems that the trust in the definition of done is limited as their are additional sprint only related to release testing where then a lot of manual tests are executed. Sounds a bit like classical release testing which would also fit that there is a dedicated role of a test manager.

All in all I think the presentation is a good example that – at least in this case – there is still some way to go at the “frontline” to get rid of the existing QA methods and going even more into the direction of agile testing.

BDD approaches for Web Development by Thomas Lundström

Visited and written by: Andreas

At first Thomas Lundström layed some basics for BDD and set the context for the following live demo. This first part of the presentation took a little bit too long for my taste, you could expect that on a conference for agile testing, everybody has a rough understanding what behaviour driven development is. And then, some parts were explained differently to what I consider them to be correct, for example the “golden triangle”, the fusion of requirements, tests and … done?! To the best of my knowledge, and this is also how Gojko Adzic explains it in his book “Bridgin the Communications Gap”, executable specifications union requirement, test and examples (to explain the requirements). The presented combination of cucumber and webrat, ruby based tools, were capable of testing a simple guest book webapp. Ok, user stories are called features in cucumber (or rather in Gherkin, the DSL), and acceptance tests are not called acceptance tests, but scenarios, but these are just minor issues. Webrat does not support Ajax, so it is really only usable for simpler use cases, but cucumber also supports Selenium and HtmlUnit.

A very important point was underlined by Lundström, and I don’t want to miss amplifying it also here. It is important that automated acceptance tests / executable specifications describe the rules that should be tested – including the context and prerequisites, but without the concrete steps / the script how to get there. Creating the context is what the fixture should do. Lundström gave the following example and differentiated between the imperativ style (the script) and the declarative style. By restricting the test on the logic that should be tested, the declarative stype is much more robust. An additional advantage is in my opinion that it opens the possibilities to automatically refactor the fixture (the binding between the test and the code under test), if it is written in the same programming language as the system under test. An impossible thing, when the context creating logic is part of the test script.

Imperative Style

Story: Animal Submission
 
  As a Zoologist
  I want to add a new animal to the site
  So that I can share my animal knowledge with the community
 
  Scenario: successful submission
  Given I'm on the animal creation page
 
  When I fill in Name with 'Alligator'
  And select Phylum as 'Chordata'
  And fill in Animal Class with 'Sauropsida'
  And fill in Order with 'Crocodilia'
  And fill in Family with 'Alligatoridae'
  And fill in Genus with 'Alligator'
  And check Lay Eggs
  And click the Create button
 
  Then I should see the notice 'Thank you for your animal submission!'
  And the page should include the animal's name, phylum, animal class, order, family, and genus

Deklarative Style

Story: Animal Submission
 
  As a Zoologist
  I want to add a new animal to the site
  So that I can share my animal knowledge with the community
 
  Scenario: successful submission
  Given I'm on the animal creation page
 
  When I add a new animal
 
  Then I should see the page for my newly created animal
  And the notice 'Thank you for your animal submission!'

Quelle: Imperative vs Declarative Scenarios in User Stories

Introduction to Robot Framework von Pekka Klärck

Visited and written by: Thomas

Even though Andreas was telling me before the session a thousand times that I anyway know everything about Robot Framework and there is no need for me to visit this track, I could not resist to see Pekka in action and getting some first hand information on the latest features in Robot. Pekka did not disappointed me.

First of all a few basic facts: Keyword-driven, based on Python, supports Java using Jython. Support for other languages via XML-RPC remote interface. Robot is Open Source under the Apache 2.0 licence. The tool evolved out of a thesis work that was started at Nokia Neworks and is today still supported by Nokia Siemens Networks. But I think this is enough of repeating the basic stuff as the slides are anyway available here .

The presentation continues with the explanation of keywords. Of course here Andreas is right that there is not too much new for me, as I am dreaming already in keywords. It goes on with a very short comparison with FitNesse and Pekka points out that in Robot everything is a keyword, which makes things very easy to understand in comparison to FitNesse where there are different kind of fixtures one has to understand. The example that is shown afterwards looks quite familiar as Elisabeth was using it as well for her tutorial yesterday. Now Pekka is emphasizing a great strenght of the Robot Framework, which is combining lower-level keywords to more powerful higher-level keywords. A feature in which I also personally see as a very big strength of the tool. By the way, it is quite facinating following the presentation on some Unix machine where all the menus and related items are completely in finnish language.

Pekka Klärck: “I have some confidence that really every tester – even without the slightest programming skills – can combine existing keywords to build new testcases.”

Pekka continues by starting Robot from a shell. It is pointed out that this is especially important for integrating Robot with CI-environments, which is really an easy task to do with Robot. Then the reporting and statistik capabilities of the tool are shown, especially the tagging. This is building a bridge to the tutorial yesterday where there was the question how to prevent that the build fails due to already implemented tests for which no functionality exists yet. So tagging is one half of the answer together with the possibility to define that tests marked with certain tags are not critical. Thus when those tests are failing the build is not marked as broken, which is not only a wanted effect in ATDD, but more a mandatory thing any agile testing tool has to support.

The graphical editor RIDE has also made great progress. It is now possible to extend keywords using CTRL-Space. RIDE shows then all keywords starting with the given characters together with the documentation of those keywords. This is really looking very promising and I personally think that the “final” version of RIDE will be another big boost in the usability of the Robot Framework. The same is true for the new possibility of “Executable Specifications” which will be worth a blog post of its own still.

After the presentation and during lunch there was some time for a quick chat with Pekka, which I was very happy about as we worked at the same company at the time Robot was developed and Pekka was visiting us a lot to promote the tool. It also showed me again that the development of Robot is very actively happening and I am looking forward to the things happening with RIDE and and the ATDD feature (and everything else).

Key Note Elisabeth Hendrickson – Agile Testing, Uncertainty, Risks and Why It All Works

Visited and written by: Andreas & Thomas

First of all Elisabeth really has a very good and active presentation style. Her presentation was mainly about what she calls the “7 Key Principles of Agile Testing”. Those are:

• ATDD
• TDD
• Exploratory Testing
• Automated System Tests
• Automated Unit Tests
• Collective Test Ownership
• Continuous Integration

Anthony Gardiner has already written more on those in his Blog Post ob this keynote.

Elisabeth Hendrickson: “The only way to tell if you are agile is if you are successfully delivering software.”
Elisabeth Hendrickson: “Exploratory Testing is a dicipline. It is not just blindly banging on the keyboard.”

I am not sure whether or not it is some kind of sign that I was initially writing “keyword” instead of “keyboard” in the quote above . But back to the keynote: It sontinues with CI and an explanation that tests are artefacts that belong to the project. I fully support this and I find it hard to understand that there are still projects that do not have there tests and test documents under version control same way the code is under version control.

Agile Quality Management – Axiom or Oxymoron? by David Evans

Visited and written by: Andreas

David Evans presented some traditional testers’ concerns that are moving into an agile environment and put them into a different light. For that he used an interesting structure, every concern was formulated from the pessimistic perspective (as oxymoron) and from the optimistic perspective (as axiom). Which of the two is the right one, I leave that as a homework to the reader

1. Developers Test Their Own Code
   • Oxymoron: Marking your own homework
   • Axiom: Unit tests are the first of many qualtiy strategies
2. Developers and Testers are co-dependant
   • Oxymoron: Independence is fundamental to quality governance
   • Axiom: whole team share the goal of creating quality software
3. Developers know the acceptance tests (are defined and agreed up-front)
   • Oxymoron: they will only write code to make the tests pass
   • Axiom: they won’t write code that makes the tests fail
4. No Signed-Off Requirements (.. seem to change with the weather)
   • Oxymoron: I can’t possibly certify that the requirements have been met
   • Axiom: Each requirement is defined by its acceptance tests: these are negotiated with and confirmed by the customer
5. No Test Plan (IEEE 829 Test Plans are in short supply on agile projects)
   • Oxymoron: Failing to plan is planning to fail! How will we know what to test?
   • Axiom: The Test Plan is impled in the product backlog: everything we build, we test (don’t build tests you don’t care if they fail!)
6. No Test Manager (or Test Management Tool) (Test Manager is not a role that exists in Agile literature)
   • Oxymoron: so how can we possibly manage testing?
   • Axiom: Testers == team; tests == specifications; … they don’t need separate management

After that he offered some recommendations for quality principles, but there have not been bigger surprises hidden (fast feedback, shared “done” understanding, etc). The right testing attitude was exposed for example in these quotes:

“Defects are evidence of missing tests”

“Any test worth writing is worth running all the time”

Means, when a test is “red” and the customer does not care … delete the Test. It was not worth writing in the first place, since apparently it was testing not needed or wanted behaviour.

Quality and Short Release Cycle von Lior Friedman

Visited and written by: Thomas

Lior starts his presentation with the statement: It is not the task of a developer to just create code, but functional and quality software. I can only say “yes”, but why is it so that this must be emphasized so often? It seems crystal clear to me that developers want to create value for the customer and it not programming for the sake of programming (this can be done in some spare time ). For me this is all about a professional approach to software development.

It continues with a discussion about Time versus Quality and the fact that in traditional projects there must be always compromises if the amount of Resourcen is fixed. With agile methods time should be saved by producing better quality at the same time.

I think for the rest I can keep it short as there was not really too much new. The presentation discusses the advantages of short release cylcles, better quality due to CI, less bugs, better efficiency, more trust from the customer side and less risk. To sum it up: The reasons why we are doing agile software development anyway.

“Testify” – One-button Test-Driven Development tooling & setup by Mike Scott

Visited and written by: Andreas

In the beginning there was some overlap with the earlier presentation from the SQS collegue David Evans. But quickly the live demonstration of “Testify” started, a tool to create a fully configured, with all bells and whistles, TDD/ATDD (Acceptance TDD) project setup; either in .Net or Java. This SQS-internal tool is open source from today on. Thanks Scott!

• .Net integrated tools: nant, nunit, fit, fitnesse, richnesse, selenium, ncover, nsis, fx-cop, cruisecontrol
• Java: here the toolsbelt looked similar, but I was too slow to note it down

Essentially, Testify is a carefully crafted project, in which some files are turned into velocity templates to replace the project name (the only parameter the testify wizard accepts) in some files and filenames. Personally I would have chosen Maven Archetypes to do that, because technically it is doing exactly that (and a little bit more), but ok, you don’t have a UI for that. If you don’t have any tool preference or other constraints, you should invest 2 hours into the generated project and see if it fit your needs.

Test Driven TDD – TDD quality improvement based on test design technique and other testing theory von Wonil Kwon

Visited and written by: Thomas

Sorry, but here a summary does not really make muh sense as the whole presentation was really low-level and there was nothing really new. Furthermore the examples did not really make much sense. Sorry Wonil.

Key Note Tom Gilb – Agile Inspections

Visited by: Andreas & Thomas, Written by: Thomas

Yiihaaa, it is 6pm and Tom start his keynote. So far I never had the opportunity to see Tom live and half an hour later it seems as if this is not really what some visitors have been expecting as it seems that agile is no topic at all in his presentation. More to the opposite as at the end of this talk he is firing with big canons at agile saying that “yeah it’s nice you guys do your standup meetings and have fun and everything but no one needs those in real projects. There you need measurable results!”

Tom Gilb: “It is you’re fault if you accept garbage in!” (about testers accepting bad requirement documents)

And that is the key message of Tom’s talk: How can I find the amount of potential defects in requirement specifications and minimize them? Because for two defects in the requirement specification there will be one bug later on in the software. At this point in time Andreas was sitting next to me and was already a little bit agonised as he was missing the agile background and at the same time he was a bit flashed by the slides. But I have to say: I really liked it and therefore I wanted to write a bit more on it still .

Tom Gilb: “Garbage in guarantees some garbage out no matter how clever you are.”

Ok, so requirements are the topic. No “Executable Specificqations”, no fancy automated tests but a way to find potential defects in requirement documents. I think it is quite obvious that Tom wants to be – and succeeded to be – very provocative with his statements, especially at a conference focusing on “Agile Testing”. But I am a big fan of looking at things from different angles and I think Toms’ approach can very well be seen as something that can be used in addition and it does not really contradict with the agile testing practices - used anyway at a later stage in the project - discussed so far.

Tom Gilb: “How lucky do you feel today, punk?” (About the 50:50 chance that a bug in the requirements results in a bug in the final software.)

The discussed method is analysing specification documents with respect to the following three rules:

• Clear enough to test is.
• Unambigious with respect to intendend readers.
• Contains no design.

With this approach you can now analyse some pages of a given requirement specification and get “the amount of defects per page” as a measurable size. For this it is enough to have a group of stakeholders analyse the document for about 10 minutes. I skip some of the details here, but in the end you get some number indicating the amount of defects which gets multiplied with some Tom-Experience one page of requirement specifications containing roughly 300 words in average. (Tom had some even more extreme examples with up to 600 defect in a page containing only 300 words.) This seems to be a normal value when looking at specifications that have not been written with any special focus on the rules shown beforehand. With 60-100 defects on one page this would mean already 600-1000 defects on a specification with only 10 pages. With 50% of them ending up as real bugs in your system that must be found and fixed this shows how dangerous this is for projects … and how expensive.

I really think this is a very good approach to check requirement documents (also together with a customer) and is worth trying out on ourselves. Of course Tom also has some solution for this problem by describing requirements in a more formalized way. For more information just use google to find Tom’s book. Of course there will be also some other ways of solving this once the problem is identified. In the end this means that with this we will get excellent input for our “executable specifications” and then I think it is also ok for Tom that we have some fun using our agile methods and everything … as deep within him he can for sure as well see some value in this .

Chill out/Dinner Buffet – Oktoberfest

Visited and written by: Andreas & Thomas

The evening event was done using the Oktoberfest as a theme. Well, we are both not the biggest fans of this and the loud music was making it very hard to continue any meaningful discussions. But anyway it was fun and was of course a nice idea.

Later the evening we are moving to the hotel bar to find still some people for some discussions and to finish this “monster baby” of a blog entry. This is of courese worth a quick agile retrospective and we decide to use a different approach tomorrow by publishing smaller blog posts more frequently. This will minimize the risk to produce something that in the end no one needs as it might not get ready in time for the end of the conference .

A dinner downer: The speaker all took of for the speaker dinner, which left all of the regular visitors on themselves. Thomas and I will check were we can find other agile testers for dinner.

[Show as slideshow]

Designing a Lean Software Development Process, by Mary and Tom Poppendieck

Visited and written by: Andreas

According to the abstract of that tutorial, it should show possibilities how plateauing productivity improvements of agile teams can be analysed. Also options for improvements should be shown.

For analysing the existing process, it is feasible to visualize the value stream end to end: from the unhappy customer with a dedicated need for a new feature to the happy customer. The value stream includes all activities in between: for example marketing, product management, budget planning, recruiting, project staffing, design, development, test, system test, delivery, deployment or installation. After that you can look at how much calendar time is really used for producing value and how much waste you have in the process:

Process Cycle Efficiency = Value Added Time / Total Cycle Time

The problem with this metric is: the deeper you dig, the lower the number gets, because you will always find more things that are waste and not generating value. Still you can identify which parts of the process you should concentrate on for optimization.

The unhappy customer, the new requirements for the project, can be motivated by two factors: the “value demand” is demand based on a need for new features. Here you can surprise and satisfy the customer by exceeding the expectations. The “failure demand” is caused by mistakes you make. Here you can consider which actions could reduce the effort you have to put into satisfying the “failure demand” in order to get more throughput for the “value demand” (for example no 1st/2nd level support, customers talk to developers, they get direct customer feedback and can learn).

“Support calls are caused by a difference in mental models of developers and customers”

A strategy that does not work to increase the throughput is to just define new goals: Time to resolve a customer defect must not take longer than 20 days. You have to change the system, otherwise you will see no changes in the symptoms.

What happens with requirements that will not be processed in the near timeframe? You should accept that there is not only a “Now” and a “Next” release queue, but also a “never” queue for features. Customers do not really like to get a “no” for a feature request, but there are so more happier to get a reliable “yes, within the next 4 weeks”.

One of the most important questions during product and software development is: when will it be ready? The answer to this should not be, at least not according to the content of the tutorial, a schedule, but a reliable workflow. A schedule has the characteristic to utilize everybody for 100%, what will rarely happen in real life due to unforeseen events. Additionally a plan is trying to include all dependencies in a complex system, while a workflow can be controlled more easily. The “lessons learnt” from building the Empire State Building (about one year!) were:

• Design a system to meet the constraints; don’t derive the constraints from the design
• Decouple workflows; break dependencies
• Workflows can be controlled more easily are more predictable than schedules

For sure it is a question of lacking practice, but right now I can hardly imagine how to design decoupled workflows in software development – this is why in Scrum all thing neccessary to get a feature done are within one Sprint, in a sequence that is most appropriate for the Story at hand. Speaking of Scrum, an interesting quote: Scrum was designed to adress the scarce resource of software developers.

In retrospect I think that the stories and examples in the tutorial (Construction of the Empire State Building and Southwest Airlines not focussing on load factor) were entertaining and inspiring, but I think I would have liked examples from the world of software development better, just to see that the principles all apply also to software development.

Acceptance Test Driven Development (ATDD) in Practice, by Elisabeth Hendrickson

Visited and written by:Thomas

Ok, what do I take home from the tutorial “Acceptance Test Driven Development (ATDD) in Practice” by Elisabeth Hendrickson? First of all roughly 10 Din A4 pages of notes which clearly indicates that this was really a very interesting event. But beside a lot of paper there have been also a lot of concrete ideas how to take ATDD into use more in our own projects. On the other hand it is also very interesting to hear the different experiences on this topic from other participants of the tutorial. By this it was becoming quite clear that currently for most people there a still a lot of problems. Here it is not that much a problem of technical issues or the tooling, but a lot more on the organisational side. With this it is very much the same as with introducing other agile practices to organisations.

The tools and processes behind ATDD very well fit to organisations that have already been successfully adopted agile practices and here it really offers the chance to create additional value. This is supoorted by the fact that also the tools are making big progress all the time and of course I was very excited to see some new feature of the Robot Framework in action that allows to define requirements in sentences written purely in English language (of course there is some magic behind). With this tests can be implemtend as “Executable Specifications” and thus form a common foundation for the product owner (= customer) and the development team.

In the following a short list of Quotes collected during the tutorial.

• The Technical Team is “The tribe of the how” while the Product Owner is “The Voice of the What”. See also Agile Anarchy
• A bug is a valid expectation of the Product Owner that is violated.
• How do we test automated tests? Explorative Testing
• Lean does not mean: Be stupid!
• Gold plating is a big risk in Software Development, as it creates things no one actually wanted.
• Agile is the most diciplined software developing approach.

It was also getting clear – once again – that a successful development team (e.g. a Scrum team) must have all required skills within the team. It is really bad to have any kind of silos, e.g. by putting testers or architects to own teams. This goes to the same direction that Boris Gloger was pointing out at our last Meet the Experts where he was saying that every member of a team should be able to perform 90% of the tasks that team has to perform. During the whole time it really became clear how difficult it is to start with ATDD unless you have some agile foundation to build on. Elisabeth was also emphasizing that you need both: Test automation and an agile approach. Those things do not work without each other. Probably this is a good closing word for the summary of the tutorial.

Given the amount of – unfortunelty in parallel running – sessions it is good that my collegue Andreas has already arrived in Berlin. This way we can visit different sessions. Of course this means we have to negotiate still who will visit which session . On Tuesday I am very interested in “Agile Testing: A Report from the Frontline” by Joke Hettema and Ralph Roosmalen in addition to the session by Pekka. Wednesday tough decisions will be needed as well, but “Bridging the gap between agile and traditional testing” by Anko Tijman is on my list of favorites. For the rest of the slots I am not yet really sure.

Ok, as my collegues are anyway telling me all the time that my blog entries are way too long I make it short for today and keep the really long entries for the following days at the conference. This brings me back to the decision whether to have a look at my SCWCD book or some matches of “Go” on my iPhone. But well, considering my experinces with travelling by train I will probably be back to sleep as soon as I put the laptop back to my bag .