{{{ #!html

Independent libraries versioning proposal for boost development environment, testing and release procedures

}}} == Introduction == This document presents the proposal for [#svn_structure restructuring] boost development environment along with new [#test_procedures testing] and [#release_procedures release] procedures. The primary idea of this proposal is to "bite the bullet" and completely separate boost into multiple independent components and use [#independent_versioning independent versioning] for each one. == Objectives == The objectives of this proposal include: * Independent development of all the components that constitute boost libraries set (accepted and proposed), so that each developer may work on his own pace and boost can scale up to an ever-larger number of libraries, and so do without the incremental weight of each new library having a negative impact on other libraries or on release management * Explicit support for libraries dependencies * Decoupling individual library testing and release [#individual_lib_release procedures] from [#boost_umbrella_release boost umbrella releases] * Minimizing requirements for the test environment while making it flexible to support all the necessary tests * Boost releases that are professional and reliable, and suitable for users ranging from simple projects by individuals on up to very large multi-developer projects in large organizations. * Timely releases of Boost are available on a predictable schedule, with no possibility that problems with a few libraries can delay a release of Boost as a whole. * Release preparation does not require super-human effort by the release manager and does not involve neither testers nor developers. * It is easy to release of subsets of Boost == SVN repository structure == #svn_structure The repository structure have to be changed to reflect an "independent nature" of this proposal. Essentially the change convert existing boost tree that look like this: {{{ boost/ lib1/ lib2/ lib3/ .... some_header1.hpp some_header2.hpp .... libs/ lib1/ lib2/ lib3/ .... }}} to {{{ lib1/ boost/lib1/ boost/lib1.hpp libs/lib1/ lib2/ boost/lib2/ boost/lib2.hpp libs/lib3/ ... }}} The SVN repository directory structure for every library should look like this: {{{ libX/ trunk/ boost/ libX/ libX.hpp libs/ libX/ src/ test/ Jamfile example/ Jamfile build/ Jamfile doc/ releases/ 1.0.0/ 1.0.1/ 1.0 -> 1.0.1 1.1.0/ 1.1 -> 1.1.0 ... LATEST -> 1.4.2 branches/ dev_branch }}} Releases directory should only be populated by automated [#individual_lib_release script] during libX release. Branches are created by developer when needed. === Boost web site content === #web_content The content of the boost web site should be made a first class citizen library that has it's own [##independent_versioning version] and [#lib_dependency dependency] on all others accepted boost libraries. === Boost tools === #boost_tools The tools developed by boost (if any) should be made a first class citizen libraries that has it's own [##independent_versioning version]. This includes boost.build and boost.doc. === Optional extensions === {{{ #!html

New library script

}}} To automate the procedure of new library creation it might be convenient to create a script that create above directory structure in a single command. {{{ #!html

Combined tree as a reflection

}}} To support the need for a single command update of several (all) boost libraries it might be convenient to create combined boost tree (similar to our current tree) using svn externals based reflections. {{{ boost/ lib1 -> lib1/trunk/boost/lib1 lib1.hpp -> lib1/trunk/boost/lib1.hpp lib2 -> lib2/trunk/boost/lib2 lib2.hpp -> lib2/trunk/boost/lib2.hpp lib3 -> lib3/trunk/boost/lib3 lib3.hpp -> lib3/trunk/boost/lib3.hpp ... }}} Note that this tree will reflect the trunk versions of all the components {{{ #!html

Sandbox/proposed libraries support

}}} The proposed directory structure allows us to avoid introduction of independent notion of sandbox. The same structure that is used for accepted libraries have to be used for any other component. This should make library incorporation into accepted libraries set especially easy. {{{ #!html

Boost release view

}}} It might be convenient (especially from [#build_procedures build procedures] standpoint) to create [#boost_umbrella_release boost umbrella release] reflections that combines all released libraries with their versions: {{{ boost_release/ 1.35.0/ lib1 -> lib1//boost lib2 -> lib2//boost lib3 -> lib3//boost ... 1.35.1/ }}} For example, above tree allows make system to refer to the version of library lib1 released as part of 1.35.0 as $(BOOST_DEV_ENV_HOME)/1.35.0/lib1. Alternatively boost release reflection can be made to match the deployment structure. {{{ boost_release/ 1.35.0/ boost/ lib1 -> lib1//boost/lib1 lib1.hpp -> lib1//boost/lib1.hpp lib2 -> lib2//boost/lib2 lib3.hpp -> lib2//boost/lib2.hpp lib3 -> lib3//boost/lib3 lib3.hpp -> lib3//boost/lib3.hpp ... 1.35.1/ }}} == Independent versioning == #independent_versioning Each library is independently versioned and [#individual_lib_release released]. The [#boost_umbrella_release boost release] is just a thin packaging layer on top of set of already releases components. Concrete versions naming scheme is up for discussion. Each library may [#lib_dependency depend] not only on truck version of other library, but also on any other released versions. Optionally we may consider support for depending on development braches. == Explicit dependencies support == #lib_dependency There several ways to maintain the library dependencies. Concrete implementation may depend on tools we use/develop. For the sake if this proposal let's make them part of the library Jamfile (It may require further revision, for we need to access this information from outside make system). Each library build/Jamfile file should include dependency statement in a form like this: {{{libX.dependant_libs = lib1: lib2: lib3:}}} For example: {{{A.dependant_libs = B:1.2.1 C:1.5.0 D:1.4.3}}} The Jamfile have to be created even if library is header only. All the targets in test and example directories have to include similar dependency statements. The dependency for a test/example of library X have to include the host library X version as well. For example: {{{A_test_1.dependant_libs = A:TRUNK B:1.2.1 C:1.5.0 D:1.4.3}}} For offline libraries and test/examples the dependencies list generation is straightforward and enforced during [#build_procedures compilation]. For the header only libraries it is library developer responsibility to maintain it (usually it should be easily deducible from the dependency list in the library tests). === Strong dependency === #strong_dependency If library A required library B for all the configurations in which it can be used, B is called a strong dependency of library A. For example all boost libraries have strong dependency on Boost.Config. === Weak dependency === #weak_dependency If library B is required by library A only for particular configuration or if particular feature is used, B is called a weak dependency of library A. This is also the case if library B is only required by library A tests/examples. === Supported version formats === * .. Refers to the specific version of library X. * . Refers to the latest patch release of library X in . version * LATEST Refers to the LATEST released version of library X * Boost: Refers to the version of library X that was released as part of referred version of Boost * Boost:LATEST Refers to the version of library X that was released as part of latest released version of Boost * TRUNK Points to the library X trunk. To be used to test against development version of another component. Is not recommended and would prevent releasing of your library. == Development/testing environment == #dev_tree The environment used by developers and testers can represent a subset of SVN repository and only refer to the libraries/versions actually used: {{{ my_lib_under_development/ trunk/ lib1/ releases/ 1.1.0/ lib5/ releases/ 1.5.1/ ... }}} In addition the environment may include [#boost_release_view boost umbrella release reflections]. Boost make system [#auto_fetch automatically fetch] any dependency that is not present in the environment. == Build procedures == #build_procedures Boost make system have to recognize different locations of boost libraries and their independent versioning. === Automated dependency fetch === #auto_fetch If build target depends on the libX:Nx and the version Nx of the library libX is not present in development environment boost make system pulls it automatically from the svn and place in an appropriate location. For the header only library there should be a way to define the target in build/Jamfile that does only that dependencies fetching. === Automated include/lib population === #include_lib_population Boost make system shouldn't add $(BOOST_ROOT) to the list of includes. Instead if build target depends on the libX:Nx then $(BOOST_DEV_ENV_HOME)/libX/releases/Nx should be added to the list of includes. Similarly appropriate library have to be added to the list of ling objects and link path have to be added to the library search dirs list. This procedure should allows us to easily identify all necessary dependant libraries for all the targets. == Development procedure == Let's say you are a developer of library X. Your development tree can include only your own library (at least this is the only thing you need to pull from svn manually, the rest is done by make system automatically). There are several modes you can work in using new development environment === Active development mode === If you are doing active development you have to change build/Jamfile to depend on stable versions of dependant component. Easiest choice is the versions used in last boost leases: {{{libX.dependant_libs = A:Boost:LATEST B:Boost:LATEST C:Boost:LATEST D:Boost:LATEST }}} Now you can do your own development with no regards to other developers activity. Your tests are validated against stable version of boost and any failures are introduces exclusively by your changes. If let say you found a problem in library A and after discussion library A developer releases new version of his library you may change dependency list to {{{libX.dependant_libs = A: B:Boost:LATEST C:Boost:LATEST D:Boost:LATEST }}} Note that by this time new version of A is already tested and released. And yet again you are testing against stable version of boost. If you are in a hurry, you can change dependency list to {{{libX.dependant_libs = A:TRUNK B:Boost:LATEST C:Boost:LATEST D:Boost:LATEST }}} But in this case you are on your own to figure whose fault is new failure, so in general this is not recommended. Once all your tests are green you proceed with library [#individual_lib_release release] and switch to maintenance mode (or another round of development). === Maintenance mode === If you are not doing any active development you have to change build/Jamfile to {{{libX.dependant_libs = A:LATEST B:LATEST C:LATEST D:LATEST }}} Now you are testing against latest released version of all dependant libraries. This mode allows you to [#auto_test_restart immediately] see any regressions caused by other libraries changes, allows [#auto_promotion automated promotion releases] and is required to prevent library [#lib_deprecation deprecation]. If regression is detected and your library require changes, you switch to the development mode. If one of the dependencies is at fault you may switch to the last stable version of that library and wait for the update release announcement or just ignore it for now. Once dependency is fixed you can do your own patch version release. But this is optional since it going to be taken care of by [#auto_promotion automated promotion release] feature of the [#boost_umbrella_release boost umbrella release] procedure. === Deprecation mode === #lib_deprecation If you are interested in maintaining library anymore you have to change build/Jamfile to {{{libX.dependant_libs = A: B: C: D: }}} The [#boost_umbrella_release boost umbrella release] will remove your library from the supported boost libraries set. == Individual library release == #individual_lib_release As soon as regression test for your library is green, you are free to make release. You don't have to synchronize with/wait for anyone. It's done by single script that does following tasks: 1. Checks that all unit tests are green 2. Checks that you don't depend on development version of other components (version TRUNK) 3. Create release tag in libX/releases directory 4. Save version of all dependant component. Even it you point on the LATEST, this will save actual version at the time of release. This is done by modifying libX.dependant_libs to refer to concrete version at the time of release 5. Post announcements "Library A version N is released" on dev list. No binaries are released at this time unless we invent another procedure for independent component packaging. Ideally this should be server side script to prevent manual "hack" releases. == Boost umbrella release == #boost_umbrella_release The boost release is done automatically every predefined period of time (my preference would be three month). It requires NO testing, NO branching and NO merging/reverting. It's done by single script (a bit oversimplification, but close) that expect at least one optional argument: list of libraries to be released. By default all accepted boost libraries are released. But the same script can be used to release any subset of boost. The script performs following tasks: 1. For every library to be released it checks that all libraries it depends on are part of released libs set and if not adds it. 2. Does [#auto_promotion automated promotion release] for all the released libraries that requires it. 3. For every library to be released it finds last version that can be included in the release based on [#version_selection version selection algorithm] 4. Deduces next version of boost (unless specified explicitly, which is what you should be doing for subset releases) If, in comparison with last boost release, any library was released with version update this will be version update of the Boost. If all libraries released with version updates, it's going to be release of boost. 5. Merge appropriate versions of all released libraries into single deployment tree: {{{ boost/ lib1/ lib1.hpp lib2/ lib2.hpp lib3/ lib3.hpp .... libs/ lib1/ lib2/ lib3/ .... }}} 6. Produce deployment package and post dev list notification. === Automatic promotion releases === #auto_promotion Let's say S is the set of libraries to be released for each library X in S we can do a "auto-promotion release" in case if following conditions are satisfied: 1. For the latest version of library X - Nx - there is dependant library Y, so that release Nx depends on version Ny1, but latest version of library Y is Ny2 > Ny1. This essentially means that one of the dependencies was released after X is released. 2. Trunk version of library X depends on LATEST version of library X and all the tests for the trunk version of library X are green. This essentially means that library X still works with new version of library Y. In this case we can make an automatic "promotion" and release patch version update for library X without any changes that depends on latest version of library Y Ny2. In practice for at the beginning we may have to do this manually. In a long run it have to be automated. === Version selection algorithm === #version_selection Let's say S is the set of libraries to be released. Following algorithm is used to select the versions of all the libraries in S that can be safely released together. TODO == Testing procedures == #testing_procedures This proposal doesn't require significant modification in testing procedures. Test driver script have to be changed to: * Accommodate different [#dev_tree testing environment] structure * Accommodate to the fact that test environment may include only subset of all the boost libraries * Initiate library A targets testing in case if any library A depends on (through LATEST reference) was released In addition the test results have to be accessible programmatically from the single location for scripts managing releases. The ideal solution would be to have some external DB based server that hosts interfaces for tests reporting and querying. === Optional extensions === In addition we may consider in long term implementing following extensions: * On-demand test requests.[[BR]]Exact scope of this feature is up to discussion * Proposed library testing.[[BR]]It might be helpful to allow some not-yet-accepted libraries to be tested in the same environment as regular libs. It shouldn't be difficult to support, since they are going to be no different in directory structure. * "Non-incremental testing" mode.[[BR]]For the test hosts running under limited disc space resources test system configuration have support option to clear all the results (including built binaries) after testing is completed. Exact scope of this feature is up for discussion.