The Boost Development Process - A proposal

Abstract

With this document I would like to present my contribution to the currently ongoing discussion on how to improve the Boost development process. I tried to focus on changes that may be applied now, in the 1.35 release time frame, rather than envision a more revolutionary approach that would take a long time to be put in practice. For this reason I tried to avoid placing requirements on the availability of tools that would take a significant amount of time to be developed. Most of the concrete changes I suggest revolve around the future structure of the Boost Subversion repository and the ways to use it, as I expect that this will already require a significant amount of work and impact the every day activity of each developer.
The key elements of my proposal are a three part Subversion repository structure and a staged development process, inspired by the one used for GCC.
I also concentrated on the changes with respect to the current process and did not attempt to provide a complete description.

The Subversion repository

The Subversion repository should be divided in the following areas:

• development - This shall contain one directory for each library, under which the customary Subversion directories shall be found: trunk, tags and branches. Subdirectories will be under full control of their respective library owners who shall be free to adopt the tagging/branching policy of their preference.

development
 +spirit
 | +trunk
 | | +boost
 | | | +spirit
 | | +libs
 | |   +spirit
 | +tags
 | +branches
 +serialization

• integration - This shall be structured as the current Boost development tree in CVS and will play the role currently played by both HEAD and the Release Candidate branches.
• release - This shall also be structured as the current Boost development tree and will mostly serve as a work area for the release manager and as an official source for the most current stable version.

Developers will be free to choose whether to base their development environment on the Integration branch, the Release branch or some previous revision of either, combined with a development branch of their library, and possibly also of some other libraries theirs depends on.
Tests will be continuously run on the integration branch to which developers will merge their work when they see fit, provided they follow the restrictions placed by the current development stage.

The release cycle

Each release cycle should take three months and be divided in three one month stages:

• During Stage 1 new library additions and major changes to existing ones shall be allowed. It is understood that during this stage the integration branch may remain "broken" for short periods of time, where by broken it is meant that checkins related to one library may cause tests in other libraries or the overall build to fail. Developers are not expected to give precedence to handling failed tests in their own libraries during stage 1.
• In Stage 2 only less extensive changes to existing libraries shall be allowed. Developers whose checkins to integration cause breakage are expected to give precedence to fixing such problems. The Release Manager may ask developers to back out their changes if s/he feels that either they are too extensive or fixing problems takes too long. Developers are encouraged to deal with failing tests in their libraries, but "all green" test results are not an objective for this stage.
• Stage 3 is devoted to bug fixing and to preparing for release. All activity on the integration branch shall be devoted to ensure that tests either pass or are marked expected on all the primary platforms. When "all green" test results are achieved, the Release Manager will merge the integration branch  to the release one. Towards the end of this stage the Release Manager will evaluate, together with the developers, whether the latest revision on the Release branch is a reasonable candidate for official release or additional time should be devoted to try and reach an additional, more complete "all green" revision. When the agreed upon revision becomes available the Release Manager proceeds to package the release. In case of delays an attempt shall be made to preserve the subsequent release cycle by compressing its stage 1 and possibly stage 2 phases; should a release be completed earlier than scheduled the time saved shall be devoted to the subsequent stage 1 and 2 phases.

Supported platforms and testing

At the start of a release cycle the set of currently supported platforms should be evaluated. The minimum non negotiable criterium for the inclusion of a platform is the availability of a volunteer to run the regression tests for the duration of the release cycle. Ideally one or more platform mantainers should be identified who are willing to help developers with problems specific to their platform. This is especially important for less common platforms or compilers. Other criteria for the selection of supported platforms are beyond the scope of this document.
The chosen platforms are assigned to a core set of testing volunteers, who commit to running tests regularly for the duration of the release cycle. Each single platform/compiler/variant combination should only be tested by a single volunteer. During stages 1 and 2 test results for additional combinations are accepted and included in the result reports, even if they are sporadical. Tests that are older than a given amount of time are excluded from the reports. During stage 3 only test results for the supported platforms are included in the reports.
Tests are run continuously on the integration branch, except that the Release Manager may ask that a test run be performed on the release branch instead, to ensure that merging didn't cause any problem.

Tools present and future

Apart from issues connected with the move to Subversion, the present proposal is meant to be applicable without introducing new tools or modifying the existing ones. On the other hand there are a few improvements that would be desirable.
First and foremost the reliability of incremental tests should be verified and fixed.
Developers should be encouraged to run tests locally in their development environment on as many compilers as possible. This is already possible with the present tool combination, but in perspective it would be a very good thing if the very same tools used for official regression tests could be made to work locally.
Another area where an increase in ease of use would be desirable is the specification of expected errors. At the very least splitting the current single file configuration among the libraries should be considered.

Evolution

Many interesting ideas have emerged from the ongoing discussion that, while highly desirable, cannot be put in practice in a short time, mainly due to the requirements they place on the development infrastructure. While the main focus of this proposal is what can be put in practice for the next release cycle, I believe that some of these idea should be considered as an evolution path.

Dependency management and partitioning

In the not so long run the necessity of partitioning Boost in more manageable packages will have to be addressed. In this respect work on a plan to takle issues such as inter-library dependency management and package level versioning should start as soon as possible.
On the other hand even a sensible decision on the course to take is going to take months and we cannot wait that long to address the problems of the current development process. In order to limit the risk of causing delays a complete strategy will have to be devised and prototyped and all the required tools shall have to be developed and tested before moving in such a direction.
Personally I'm not convinced that partitioning should take place at the single library level, except for some of the larger ones. I also fear that per library, on demand testing is going to prove too complicate to be viable, however I admit that such topics should be given more consideration. I'm also convinced that proponents of switching to a single library oriented process haven't addressed integration issues in a convincing way.   

Financial support

One topic that wasn't addressed so far is how to ensure that sufficient time is devoted, not only to library development but above all to management. In my opinion the only definitive answer is professional involvement, i.e. Boost needs a few individuals to devote paid time to it. I realize that this is a thorny issue, because it contrasts with Boost's current status of non-entity, but I'm convinced that part of the problem has to do with the fact that voluntary work alone cannot guarantee the degree of continuous commitment that a project this size requires, the heroic efforts put in by some of the most experienced people around here notwithstanding.
At the very least companies that use Boost for commercial development should consider hiring experts if they require a superior support level or some different form of packaging.

Open issues

This proposal fails to address several important topics:

• Testing of release variants. It is certainly desirable that at least for the most popular platforms regression tests are run for both the debug and the release variant. The main obstacle is resource availability, even though the rationalization of the current resources may produce some benefit in this respect.
• It has been correctly pointed out that release quality criteria should include more than just regression test results. 
• Providing tool support for the enforcement of stage policy should be considered.
• Some of the problems faced during the development of 1.34.0 have to do with communication among developers. These should also be addressed. The languishing of bug reports is in my opinion related to these problems.
• This proposal should address how bugfix releases should be handled. Hoping that they won't be necessary is not enough :-)
• I believe it's way to early to address changes in our numbering scheme.
• I have no idea about what can be found at http://beta.boost.org:8081/ ; I'm currently behind a firewall that blocks all http traffic that doesn't use port 80.

Acknowledgements

The ideas in this document draws on Beman Dawes's and Gennadiy Rozental's proposals, and on the comments of many participants in the Boost Development Environment proposal discussion thread.