CMake and Modularization Status

Officially, all Boost libraries currently inhabit a ​single subversion repository. This page documents the status of the effort to split Boost up into separate modules. Those working on the modularization are simultaneously porting the build/test system from ​Boost.Build to ​CMake.

Automation

​Modularized Boost is continuously maintained and tested by a system that automatically does two things:

1. Modularization: shuffle files from their standard places in a Boost distribution into separate ​per-library git repositories
2. Integration Testing: run each libraries' tests to make sure its modularized state is healthy.

You can review the current status of these steps ​here. The rest of this section details what the steps are doing.

Modularization

Each time there's a new commit to Boost's trunk, ​this script modularizes a ​Git repository mirroring Boost SVN and distributes the updated files into separate ​per-library git repositories. As ​you can see, the script is being kept up-to-date by ​Daniel Pfeifer.

The column labelled “Boost.Modularize” on the right side of the ​automated test page shows the modularization process itself. If that column is green, it tells you that every file is currently “accounted for” and has been assigned to a module. If there are files that don't have a modularized home, we get an error like ​this one. (in that run, it looks like someone just added the Boost.Heap library). When that happens, Daniel gets an email from the system and he fixes it.

To prepare to run modularization on a local machine:

# get the the modularized boost superproject
git clone http://github.com/boost-lib/boost.git modules

# Update all the sub-modules
cd modules
git submodule update --init                     # takes a couple of minutes
cd ..

# get the boost svn mirror
git clone http://github.com/ryppl/boost-svn.git     # takes a couple of minutes

# get the modularization script
git clone http://github.com/ryppl/boost-modularize.git

As a reference on Git sub-modules, see also the corresponding section of the ​Pro Git book. For instance, if you intend to customize the sub-module URLs (e.g., replace ​git:// by ​http:// when you are behind a corporate firewall), you have to replace the git submodule update --init by:

# Initialize the Git local configuration file
git submodule init
# Customize the sub-modules URLs in the .git/config file (here, replace git:// by http://)
sed -i -e 's|git://|http://|g' .git/config
# Fetch all the data from the super-project and check out the appropriate commit listed in it (.gitmodules)
git submodule update                  # takes a couple of minutes

To do the modularization once you've prepared these repositories:

cd boost-modularize
python -u modularize.py --src=../boost-svn --dst=../modules setup
python -u modularize.py --src=../boost-svn --dst=../modules update

You won't want to do this, but the automated process goes on to update the ​modularized state at GitHub:

python -u modularize.py --src=../boost-svn --dst=../modules push

CMake Support

​Modularized Boost is a Git superproject that references each modularized boost library as a Git submodule. There is a parallel ​Modularized and CMake-ified Boost project that has the same submodules, plus an ​"overlay" of CMakeLists.txt files and an ​additional support submodule. This is the project that you can use to build Boost with CMake. It is maintained by the same script as regular ​Modularized Boost.

Integration Testing

The other columns represent the results of Boost "integration tests" of the modularized, CMake-ified state on several platforms. An integration test is essentially equivalent to Boost's official unmodularized tests: we run all of the libraries' tests together, with the latest state of each library. Each time there's a change in the modularized state, we kick off integration tests.

To run integration test on a local machine, start in a clone of the modularized state. If you ran modularization locally following the directions above, you can just cd ../modules. Otherwise:

# get the the modularized boost superproject
git clone http://github.com/ryppl/boost-zero modules

# Update all the submodules
cd modules
git submodule update --init                     # takes a couple of minutes

Then create a build directory so you don't mess up the source tree:

mkdir build

Call CMake from the build directory TWICE. The two invocations are necessary for resolving circular dependencies until CMake does not provide better support for ​usage requirements.

If everything goes well, the first call to cmake will end with an error message saying "Initial pass successfully completed, now run again!".

cd build
cmake ..
cmake ..

You can now build and test:

cmake --build .
cmake --build . --target test

TODOs

Porting Test Jamfiles

The integration testing columns are currently not all green in part because several libraries have not had the Jamfiles for their tests ported to CMake yet (all Jamfiles for building library binaries have been ported). ​These tickets document the libraries whose test Jamfiles have been ported, and ​these show the libraries whose Jamfiles still need to be ported. Most of these should be really straightforward to handle based on looking at Jamfiles and examples of ​commits that closed unit test tickets (follow the link in each ticket).

For a more complex example, see the ​CMake file for Boost.Python, graciously ported by Ravikiran Rajagopal. These are among the most complex because they have to invoke Python instead of just building and running C++ executables.

Porting Documentation Jamfiles

Libraries with generated documentation (e.g. via Quickbook, ​Docbook, or ​RestructuredText) need to have their Jamfiles ported as well. There's a list of tickets ​here.

Rewriting Boost History

Several methods have been discussed for preserving the history of Boost in the modularized git repositories; we still have to settle on and implement one approach. The easiest to implement, though not necessarily the most useful possible arrangement, would be to simply allow people to graft the Boost SVN history onto their modularized Git repository.

Further Testing

There are several levels at which we might want to confirm that the CMake build instructions are doing the same thing that the Boost.Build Jamfiles are.

How Can I Help?

Set up an integration testing installation by following the directions above. While that is building, read the section on porting test Jamfiles. Then, close a few of the ​open tickets. After looking at a few of the ​completed CMakeLists.txt files (follow the commit links in the tickets), most of these should be easy and obvious textual transformations even if you don't know much about Boost.Build or CMake! For details, read “How to Contribute” in ​this document.