Testing Boost with CMake

Boost's CMake-based build system provides regression testing via ​CTest, which comes as part of CMake. This regression testing system can be used by Boost developers to test their libraries locally and also by testers and users to submit regression tests to a ​Dart server, which collects and reports regression-testing results from different sites. This document assumes that the reader has already learned how to build and configure Boost using CMake.

• Re-run the CMake configuration for Boost. Set BUILD_REGRESSION_TESTS to ON. You may notice that configuration takes significantly longer when we are building all of the regression tests. If you want you can enable regression testing for a subset of Boost libraries by listing them, separated by semicolons, in the BOOST_TEST_LIBRARIES option that appears once BUILD_REGRESSION_TESTS is ON. Be sure to re-configure CMake once you are done tweaking these options, and generate makefiles or project files, by clicking "OK" (on Microsoft Windows) or pressing g (on Unix).

• Build, following the same building process described in the section CMakeConfigAndBuild. For Unix users, we suggest passing the -i option to make, and also possibly -j 2 (or more) to run the build process in parallel. Building all of the regression tests for the Boost libraries can take a long time.

  make -i -j 2
  

Note: If you change Boost source files in a way that affects your tests, you will need to re-run the build process to update the libraries and tests before moving on to the next step.

• Once regression tests have finished building, go into the command line or command prompt and enter the Boost binary directory. Then, run the command

  ctest
  

to execute all of the regression tests. The ctest executable will be stored in the binary directory for CMake. On Unix platforms, this is the same place where ccmake resides. On Windows platforms, it will be in C:\Program Files\CMake\bin. The ctest program should produce output like the following:

Start processing tests
Test project /Users/dgregor/Projects/boost-darwin
  1/ 22 Testing any::any_test                    Passed
  2/ 22 Testing any::any_to_ref_test             Passed
  3/ 22 Testing function::lib_function_test      Passed
  4/ 22 Testing function::function_n_test        Passed
  5/ 22 Testing function::allocator_test         Passed
  6/ 22 Testing function::stateless_test         Passed
  7/ 22 Testing function::lambda_test            Passed
  8/ 22 Testing function::function_test_fail1 ***Failed - supposed to fail
  9/ 22 Testing function::function_test_fail2 ***Failed - supposed to fail
 10/ 22 Testing function::function_30            Passed
 11/ 22 Testing function::function_arith_cxx98   Passed
 12/ 22 Testing function::function_arith_porta   Passed
 13/ 22 Testing function::sum_avg_cxx98          Passed
 14/ 22 Testing function::sum_avg_portable       Passed
 15/ 22 Testing function::mem_fun_cxx98          Passed
 16/ 22 Testing function::mem_fun_portable       Passed
 17/ 22 Testing function::std_bind_cxx98         Passed
 18/ 22 Testing function::std_bind_portable      Passed
 19/ 22 Testing function::function_ref_cxx98     Passed
 20/ 22 Testing function::function_ref_portabl   Passed
 21/ 22 Testing function::contains_test          Passed
 22/ 22 Testing function::contains2_test         Passed

100% tests passed, 0 tests failed out of 22

Here, we have only enabled testing of the Boost.Any and Boost.Function libraries, by setting BOOST_TEST_LIBRARIES to “any;function.”

Again: This "ctest" step runs the tests without first running a build. If you change a source file and run the ctest step you will see that no build is invoked.

• To run just a subset of the tests, pass -R and a regular expression to ctest. For example, to run all of the Python tests, use:

  ctest -R python::
  

There is also a -E (exclude) option which does the inverse of -R. ctest --help shows the full list of options.

Submitting Regression Test Results

The ctest command can be used by individual developers to test local changes to their libraries. The same program can also be used to build all of Boost, run its regression tests, and submit the results to a central server where others can view them. Currently, regression test results based on the CMake build system are displayed on the Dart server at ​http://dart.resophonic.com/boost_1_34_0/Dashboard/.

To submit "experimental" results to the Dart server, configure a Boost binary tree by following the configuration instructions in the section ​Configuring and Building Boost and then enable regression testing via the BOOST_TESTING option, as described above. At this point, don't build anything! We'll let CTest do that work. You may want to customize some of the advanced CMake options, such as SITE (to give your site name), and MAKECOMMAND (which, for makefile targets, governs the top-level make call when building Boost). Finally, go into the Boost binary directory and execute:

ctest -D Experimental

CTest will then reconfigure Boost, build all of the Boost libraries and regression tests, execute the regression tests, and submit the results to the Dart dashboard at ​http://dart.resophonic.com/boost_1_34_0/Dashboard/. Results submitted to the dashboard usually show up in 15-20 minutes, and can be browsed from the web site given.

Note: Although we are running regression tests on several flavors of Unix on a nightly basis, we have done very little work to ensure that regression testing runs smoothly on other platforms. We will remedy this problem in the near future.