Apache Software Foundation: Behind the Scenes

Ever thought about getting involved in an open source community? The Apache Software Foundation (ASF) is one of the places to do so. But what’s in for you? We talked to someone who should know: Benedikt Ritter is involved in the ASF and answered our questions in the following interview.

The following interview was originally published in the corporate magazin Softwerker vol. 01/15.

From your point of view, what is the Apache Software Foundation all about?

Well, there are a number of principles which are important at the ASF. One of them is “Community over Code”. It means that a vivid, friendly community is more important than the code itself. The rationale behind this is that a good community can continuously improve bad code. But without a community even the best code will not evolve.
Furthermore the ASF is a meritocracy. This implies that anybody has to prove his abilities before he gets a say in the development. Once this proof has been provided, there is no way to evade say. The term “do-ocracy” it tightly coupled to this principle, meaning that those who do something are to decide. So it’s better to introduce a change that works for 90% of the use cases already than not to do it.
Personally I like that the ASF is not driven by politics. There are no big players who pursue their own agenda.

How did you find the ASF?

Anybody working in IT will sooner or later have to deal with the Apache http webserver . Apache httpd is probably the most common product of the ASF. When I worked with a LAMP stack for the first time during my time at university I didn’t know what that was all about…

Apropos httpd – this is not only the most common product, but it is the the first project from which the ASF emerged, isn’t it?

That’s right. The ASF was founded by a group of developers in the mid-nineties. They came together to write patches for the NSCA public domain HTTP daemon. In April 1995 this group released version 0.6.2 of Apache httpd. In 1999 the members of this so-called Apache Group founded the Apache Software Foundation. The myth that they chose the name “Apache” to describe the poor quality of Apache httpd (“a patchy webserver”) dates back to that time. The truth is that the name was chosen to honor the native tribes of America .

Let’s get back to how you got involved in the ASF…

Later during my studies, I worked as a student assistant in a small software company. Coming right from university, I had the urge to always reinvent the wheel. But my team leader told me: “Before you start implementing things yourself, check if somebody has already implemented it for you.” He showed me Apache Commons and I was amazed by the software components that were available there.
Sometime later I subscribed to the Apache Commons mailing list, because I wanted to know how it all worked. I started discussing and I wrote some patches. On a Saturday morning I got an e-mail by the Apache Commons PMC (Project Management Committee) inviting me to become a committer. That was a great moment for me because I convinced others of my abilities by doing good work. In the meantime I’ve become a member of the PMC myself and I send invitation e-mails to others who have convinced me of their abilities.

What is a Project Management Committee?

Every project at the ASF has a Project Management Committee. The PMC is responsible for advancing the project. Members of the PMC may propose new committers. Decisions concerning new committers are made through secret ballot. In general, votes raised by members of the PMC are binding when a vote is held. For example, when a new release is about to be made, the release manager prepares a release candidate (RC) for review. Anybody is invited to review the RC and raise any concerns he or she has or problems he or she discovers. But in order to release a RC the release manager needs three positives votes by members of the PMC. All other votes cannot prevent or accelerate the release. Of course it is pretty unlikely that the release manager decides to release something if there are only three positives votes by PMC members but ten negative votes by other community members. But formally this vote would pass by the rules of the ASF.
Another responsibility of the PMC is to prepare a quarterly report for the Board of Directors, informing the board about the state of the project. Such a report contains information about new committers and PMC members, successful releases, and anything else which may be of interest to the board. The quarterly report provides an opportunity to ask the board of directors for help if a project has problems it cannot deal with on its own. This can be, for example, incapacity due to a lack of active community members or internal conflicts among project members which cannot be resolved. I’ve to admit that I haven’t experienced such a situation until now.

You just introduced a whole lot of new terms: committer, release manager, release candidate, Board of Directors. Can you please elaborate on these?

Sure! Generally there is a difference between contributors and committers. Becoming a contributor is fairly easy: You just contribute something to a project. For example, creating a bug ticket is already a contribution. Other examples include contributions to discussions on one of the mailing lists, improvements of the documentation and providing patches for issues in the bug tracker. Patches can be provided in form of SVN/git diffs or for some time now as pull request against one of the GitHub mirrors of the ASF . Patches will then be reviewed by one of the committers regarding relevance, quality and code style. Often this leads to a discussion about what has to be improved for the patch to be accepted. When the patch has reached the desired quality, it is applied by one of the committers. The diff of each commit is sent to a separate mailing list so that other project members (committers as well as contributors) can review it. If somebody has concerns about a change, he or she can raise them on the mailing list, giving the author of the change the opportunity explain the change. This is a means to keep the quality high.

The release manager does his or her job later in the development process. At some point in time a committer can volunteer to create a new release of a project. Usually he or she will write a short notice to the mailing list like “It has been some time since we released foo 1.1, so I’d like to release foo 1.2 soon. If you’re planning to fix something that should be included, please do it now”. To start the release process, the release manager prepares a release candidate and sends another notice to the mailing list . A release candidate is made up of several parts: An SVN/git tag of the source code, which should be released, binary and source archives, artifacts that will be published to Maven Central, release notes, the project website as well as some code quality reports. The release candidate represents something like “if we decide to release foo 1.2, it will look like this”. If the vote about the release candidate is successful, it will be published. If not, the code has to be improved and the release manager prepares another release candidate.

To explaining the role of the Board of Directors I have to go back a little and talk about the organizational structure of the ASF (editor’s note: the following part is to describe the term “foundation” to German readers). The English term “foundation” can be translated to “Stiftung” in German. Nevertheless under German law the Apache Software Foundation corresponds more to a registered society. There are members in this society. You become members – following the principle of meritocracy – by proving your abilities to the other members through good work and by showing that you care about the ASF as a whole. The members annually elect the Board of Directors, which governs the ASF’s activities in all areas. The board is made up of officers for the various areas, such as fundraising, marketing, trademarks, infrastructure, etc. This is the organizational view of the ASF (see fig. 1). The PMCs take care of the technical matters and report to the Board of Directors. Furthermore new projects are initiated by the Board of Directors.

Can you describe the way new projects are handled at the ASF?

Basically there are two ways new projects are started:

• An existing project wishes to become part of the ASF. Examples of this are Apache SVN or Apache OpenOffice.
• A new project is initiated from within the ASF. For example Apache BatchEE was a result of the work on the Apache TomEE project.

No matter how a project is brought to the ASF, the starting point will always be the Incubator . The Incubator has various responsibilities. The most important one is to work on intellectual property (IP) clearance. All code at the ASF has to be released under the Apache License 2.0 . When existing projects are to become part of the ASF, there are several “problems” that may arise:

1. The code of the project has an incompatible license (e.g. GPL). In this case the IP owners may simply change the license.
2. The project has an incompatible license and it contains code whose IP is not owned by members of the project. In this case the affected parts of the project have to be reimplemented.
3. The project has dependencies to libraries with incompatible licenses. In this case those dependencies have to be exchanged or have be reimplemented.

It’s extremely important for the ASF to resolve all issues around IP before accepting a new project, because the ASF guarantees that their products may be used in commercial contexts.
Another responsibility of the Incubator is to make sure no projects are bound to fail because there is no vivid community supporting them. For this reason there are several defined points in the incubator process where the project activity is checked. In addition to that the Incubator is a place for ASF newcomers to learn “the Apache Way”.

What is “the Apache Way”?

Part of “the Apache Way” besides the principles I talked about in the beginning, is the way people work on projects. There is (with very few exceptions) no secret-mongering at the ASF. All decisions are discussed publicly and they can later be looked up in the mailing list archives. Furthermore development follows the commit-then-review process. I’ve already talked about that. It’s a way to keep the quality high. A positive side effect is that you adopt an extremely tidy way of working. Commits become very small, so that they are easy to review. Moreover they only contain changes that are part of the bugfix or feature you’re working on. Reformatting source code or reorganization of imports go to separate commits. Things are separated from each other. This helps me deliver good results in my daily work.

What are the benefits of being involved in the ASF?

First and foremost I learn a lot. Since my work is continuously reviewed by very talented people, I see where I have to improve my skills. As a committer it is a lot of fun to work with contributors and to introduce them to the Apache way. By reviewing patches and commits I can see how others approach problems. In addition to that, I get in touch with people from all around the world. Britons behave and communicate very differently from for example Italians. Having to communicate completely via mailing lists doesn’t make things easier in this regard. On top of that I’m always amazed by whom I get to work with. At the Apache Commons project I met Henri Yandell. He is responsible for the Open Source activities at Amazon. Phil Steitz has been CTO at American Express and Gary Gregory is the author of “JUnit in Action” and “Hibernate in Action”. And probably everybody has already heard about Roy Fielding… And last but not least it’s a great feeling to get positive feedback by users, e.g. “Thanks for this library, it saved me a lot of work.”

Do you have any advice to those who want to get involved in the ASF?

My tip would be “fear not, just do it”. At first I was just following discussions, because I was afraid to say something stupid. But only those who ask can be helped. I’d say that you have to be a bit frustration tolerant. Very few patches are accepted right away. It’s completely normal that you have to get used to the style of ASF projects. You shouldn’t get discouraged, but perceive discussions around your contributions as a chance to learn. Concrete assistance can be found on the ASF websites . The most important rules for patches are probably:

• All files have to contain an Apache License header.
• Use spaces instead of tabs for indentation.

Thank you very much for this interview, Benedikt. Next time I want to hear more about the Apache Commons project!