wiki:IndependentLibraryVersioning

Context Navigation

Version 5 (modified by Gennadiy Rozental, 15 years ago) ( diff )
--

Independent libraries versioning proposal for boost development environment, testing and release procedures

Introduction

This document presents the proposal for restructuring boost development environment along with new testing and release procedures. The primary idea of this proposal is to "bite the bullet" and completely separate boost into multiple independent components and use independent versioning for each one.

Objectives

The objectives of this proposal include:

Independent development of all the components that constitute boost libraries set (accepted and proposed), so that each developer may work on his own pace and boost can scale up to an ever-larger number of libraries, and so do without the incremental weight of each new library having a negative impact on other libraries or on release management

Explicit support for libraries dependencies

Decoupling individual library testing and release procedures from boost umbrella releases

Minimizing requirements for the test environment while making it flexible to support all the necessary tests

Boost releases that are professional and reliable, and suitable for users ranging from simple projects by individuals on up to very large multi-developer projects in large organizations.

Timely releases of Boost are available on a predictable schedule, with no possibility that problems with a few libraries can delay a release of Boost as a whole.

Release preparation does not require super-human effort by the release manager and does not involve neither testers nor developers.

It is easy to release of subsets of Boost

SVN repository structure

The repository structure have to be changed to reflect an "independent nature" of this proposal. Essentially the change convert existing boost tree that look like this:

boost/
   lib1/
   lib2/
   lib3/
....
   some_header1.hpp
   some_header2.hpp
....

libs/
   lib1/
   lib2/
   lib3/
....

lib1/
   boost/lib1/
   boost/lib1.hpp
   libs/lib1/
lib2/
   boost/lib2/
   boost/lib2.hpp
   libs/lib3/
...

The SVN repository directory structure for every library should look like this:

libX/
  trunk/
    boost/
      libX/
      libX.hpp
    libs/
      libX/
        src/
        test/
            Jamfile
        example/
            Jamfile
        build/
            Jamfile
        doc/
  releases/
     1.0.0/
     1.0.1/
     1.0      -> 1.0.1
     1.1.0/
     1.1      -> 1.1.0
  branches/
     dev_branch

Releases directory should only be populated by automated script during libX release. Branches are created by developer when needed.

Boost web site content

The content of the boost web site should be made a first class citizen library that has it's own version and dependency on all others accepted boost libraries.

Boost tools

The tools developed by boost (if any) should be made a first class citizen libraries that has it's own version. This includes boost.build and boost.doc.

Optional extensions

New library script

To automate the procedure of new library creation it might be convenient to create a script that create above directory structure in a single command.

Combined tree as a reflection

To support the need for a single command update of several (all) boost libraries it might be convenient to create combined boost tree (similar to our current tree) using svn externals based reflections.

boost/
   lib1      -> lib1/trunk/boost/lib1
   lib1.hpp  -> lib1/trunk/boost/lib1.hpp
   lib2      -> lib2/trunk/boost/lib2
   lib2.hpp  -> lib2/trunk/boost/lib2.hpp
   lib3      -> lib3/trunk/boost/lib3
   lib3.hpp  -> lib3/trunk/boost/lib3.hpp
   ...

Note that this tree will reflect the trunk versions of all the components

Sandbox/proposed libraries support

The proposed directory structure allows us to avoid introduction of independent notion of sandbox. The same structure that is used for accepted libraries have to be used for any other component. This should make library incorporation into accepted libraries set especially easy.

Boost release view

It might be convenient (especially from build procedures standpoint) to create boost umbrella release reflections that combines all released libraries with their versions:

boost_release/
   1.35.0/
     lib1      -> lib1/<lib1-version>/boost
     lib2      -> lib2/<lib2-version>/boost
     lib3      -> lib3/<lib3-version>/boost
   ...
   1.35.1/

For example, above tree allows make system to refer to the version of library lib1 released as part of 1.35.0 as $(BOOST_DEV_ENV_HOME)/1.35.0/lib1.

Alternatively boost release reflection can be made to match the deployment structure.

boost_release/
   1.35.0/
     boost/ 
       lib1      -> lib1/<lib1-version>/boost/lib1
       lib1.hpp  -> lib1/<lib1-version>/boost/lib1.hpp
       lib2      -> lib2/<lib2-version>/boost/lib2
       lib3.hpp  -> lib2/<lib1-version>/boost/lib2.hpp
       lib3      -> lib3/<lib3-version>/boost/lib3
       lib3.hpp  -> lib3/<lib1-version>/boost/lib3.hpp
   ...
   1.35.1/

Independent versioning

Each library is independently versioned and released. The boost release is just a thin packaging layer on top of set of already releases components. Concrete versions naming scheme is up for discussion.

Each library may depend not only on truck version of other library, but also on any other released versions. Optionally we may consider support for depending on development braches.

Explicit dependencies support

There several ways to maintain the library dependencies. Concrete implementation may depend on tools we use/develop. For the sake if this proposal let's make them part of the library Jamfile (It may require further revision, for we need to access this information from outside make system).

Each library build/Jamfile file should include dependency statement in a form like this:

libX.dependant_libs = lib1:<version of lib1> lib2:<version of lib2> lib3:<version of lib3>

For example:

A.dependant_libs = B:1.2.1 C:1.5.0 D:1.4.3

The Jamfile have to be created even if library is header only. All the targets in test and example directories have to include similar dependency statements. The dependency for a test/example of library X have to include the host library X version as well.

For example:

A_test_1.dependant_libs = A:TRUNK B:1.2.1 C:1.5.0 D:1.4.3

For offline libraries and test/examples the dependencies list generation is straightforward and enforced during compilation. For the header only libraries it is library developer responsibility to maintain it (usually it should be easily deducible from the dependency list in the library tests).

Strong dependency

If library A required library B for all the configurations in which it can be used, B is called a strong dependency of library A. For example all boost libraries have strong dependency on Boost.Config.

Weak dependency

If library B is required by library A only for particular configuration or if particular feature is used, B is called a weak dependency of library A. This is also the case if library B is only required by library A tests/examples.

Supported version formats

<major>.<minor>.<patch>

Refers to the specific version of library X.

<major>.<minor>

Refers to the latest patch release of library X in <major>.<minor> version

LATEST

Refers to the LATEST released version of library X

Boost:<version of boost>

Refers to the version of library X that was released as part of referred version of Boost

Boost:LATEST

Refers to the version of library X that was released as part of latest released version of Boost

TRUNK

Points to the library X trunk. To be used to test against development version of another component. Is not recommended and would prevent releasing of your library.

Development/testing environment

The environment used by developers and testers can represent a subset of SVN repository and only refer to the libraries/versions actually used:

my_lib_under_development/
  trunk/
lib1/
  releases/
     1.1.0/
lib5/
  releases/
     1.5.1/
...

In addition the environment may include boost umbrella release reflections.

Boost make system automatically fetch any dependency that is not present in the environment.

Build procedures

Boost make system have to recognize different locations of boost libraries and their independent versioning.

Automated dependency fetch

If build target depends on the libX:Nx and the version Nx of the library libX is not present in development environment boost make system pulls it automatically from the svn and place in an appropriate location.

For the header only library there should be a way to define the target in build/Jamfile that does only that dependencies fetching.

Automated include/lib population

Boost make system shouldn't add $(BOOST_ROOT) to the list of includes. Instead if build target depends on the libX:Nx then $(BOOST_DEV_ENV_HOME)/libX/releases/Nx should be added to the list of includes. Similarly appropriate library have to be added to the list of ling objects and link path have to be added to the library search dirs list.

This procedure should allows us to easily identify all necessary dependant libraries for all the targets.

Development procedure

Let's say you are a developer of library X. Your development tree can include only your own library (at least this is the only thing you need to pull from svn manually, the rest is done by make system automatically). There are several modes you can work in using new development environment

Active development mode

If you are doing active development you have to change build/Jamfile to depend on stable versions of dependant component. Easiest choice is the versions used in last boost leases:

libX.dependant_libs = A:Boost:LATEST B:Boost:LATEST C:Boost:LATEST D:Boost:LATEST

Now you can do your own development with no regards to other developers activity. Your tests are validated against stable version of boost and any failures are introduces exclusively by your changes.

If let say you found a problem in library A and after discussion library A developer releases new version of his library you may change dependency list to

libX.dependant_libs = A:<new version of A> B:Boost:LATEST C:Boost:LATEST D:Boost:LATEST

Note that by this time new version of A is already tested and released. And yet again you are testing against stable version of boost.

If you are in a hurry, you can change dependency list to

libX.dependant_libs = A:TRUNK B:Boost:LATEST C:Boost:LATEST D:Boost:LATEST

But in this case you are on your own to figure whose fault is new failure, so in general this is not recommended.

Once all your tests are green you proceed with library release and switch to maintenance mode (or another round of development).

Maintenance mode

If you are not doing any active development you have to change build/Jamfile to

libX.dependant_libs = A:LATEST B:LATEST C:LATEST D:LATEST

Now you are testing against latest released version of all dependant libraries. This mode allows you to immediately see any regressions caused by other libraries changes, allows automated promotion releases and is required to prevent library deprecation.

If regression is detected and your library require changes, you switch to the development mode. If one of the dependencies is at fault you may switch to the last stable version of that library and wait for the update release announcement or just ignore it for now. Once dependency is fixed you can do your own patch version release. But this is optional since it going to be taken care of by automated promotion release feature of the boost umbrella release procedure.

Deprecation mode

If you are interested in maintaining library anymore you have to change build/Jamfile to

libX.dependant_libs = A:<last tested against> B:<last tested against> C:<last tested against> D:<last tested against>

The boost umbrella release will remove your library from the supported boost libraries set.

Individual library release

As soon as regression test for your library is green, you are free to make release. You don't have to synchronize with/wait for anyone. It's done by single script that does following tasks:

Checks that all unit tests are green

Checks that you don't depend on development version of other components (version TRUNK)

Create release tag in libX/releases directory

Save version of all dependant component. Even it you point on the LATEST, this will save actual version at the time of release. This is done by modifying libX.dependant_libs to refer to concrete version at the time of release

Post announcements "Library A version N is released" on dev list.

No binaries are released at this time unless we invent another procedure for independent component packaging. Ideally this should be server side script to prevent manual "hack" releases.

Boost umbrella release

The boost release is done automatically every predefined period of time (my preference would be three month). It requires NO testing, NO branching and NO merging/reverting. It's done by single script (a bit oversimplification, but close) that expect at least one optional argument: list of libraries to be released. By default all accepted boost libraries are released. But the same script can be used to release any subset of boost.

The script performs following tasks:

For every library to be released it checks that all libraries it depends on are part of released libs set and if not adds it.

Does automated promotion release for all the released libraries that requires it.

For every library to be released it finds last version that can be included in the release based on version selection algorithm

Deduces next version of boost (unless specified explicitly, which is what you should be doing for subset releases)

If, in comparison with last boost release, any library was released with <minor> version update this will be <minor> version update of the Boost. If all libraries released with <patch> version updates, it's going to be <patch> release of boost.

Merge appropriate versions of all released libraries into single deployment tree:

boost/
   lib1/
   lib1.hpp
   lib2/
   lib2.hpp
   lib3/
   lib3.hpp
....
libs/
   lib1/
   lib2/
   lib3/
....

Produce deployment package and post dev list notification.

Automatic promotion releases

Let's say S is the set of libraries to be released

for each library X in S we can do a "auto-promotion release" in case if following conditions are satisfied:

For the latest version of library X - Nx - there is dependant library Y, so that release Nx depends on version Ny1, but latest version of library Y is Ny2 > Ny1. This essentially means that one of the dependencies was released after X is released.

Trunk version of library X depends on LATEST version of library X and all the tests for the trunk version of library X are green. This essentially means that library X still works with new version of library Y.

In this case we can make an automatic "promotion" and release patch version update for library X without any changes that depends on latest version of library Y Ny2.

In practice for at the beginning we may have to do this manually. In a long run it have to be automated.

Version selection algorithm

Let's say S is the set of libraries to be released. Following algorithm is used to select the versions of all the libraries in S that can be safely released together.

TODO

Testing procedures

This proposal doesn't require significant modification in testing procedures. Test driver script have to be changed to:

Accommodate different testing environment structure

Accommodate to the fact that test environment may include only subset of all the boost libraries

Initiate library A targets testing in case if any library A depends on (throw LATEST reference) was released

In addition the test results have to be accessible programmatically from the single location for scripts managing releases. The ideal solution would be to have some external DB based server that hosts interfaces for tests reporting and querying.

Optional extensions

In addition we may consider in long term implementing following extensions:

On-demand test requests.
Exact scope of this feature is up to discussion

Proposed library testing.
It might be helpful to allow some not-yet-accepted libraries to be tested in the same environment as regular libs. It shouldn't be difficult to support, since they are going to be no different in directory structure.

"Non-incremental testing" mode.
For the test hosts running under limited disc space resources test system configuration have support option to clear all the results (including built binaries) after testing is completed. Exact scope of this feature is up for discussion.

Note: See TracWiki for help on using the wiki.

Download in other formats:

Plain Text