Version 4 (modified by 15 years ago) ( diff ) | ,
---|
Testing Boost with CMake
Boost's CMake-based build system provides regression testing via CTest, which comes as part of CMake. This regression testing system can be used by Boost developers to test their libraries locally and also by testers and users to submit regression tests to a Dart server, which collects and reports regression-testing results from different sites. This document assumes that the reader has already learned how to build and configure Boost using CMake.
- Re-run the CMake configuration for Boost. Set
BUILD_TESTING
toON
. You may notice that configuration takes significantly longer when we are building all of the regression tests. If you want you can disable regression testing for some or all Boost libraries by toggling theTEST_BOOST_
libname options that appear onceBUILD_TESTING
isON
. Be sure to re-configure CMake once you are done tweaking these options, and generate makefiles or project files, by clicking "OK" (on Microsoft Windows) or pressingg
(on Unix).
- Build, following the same building process described in the section CMakeConfigAndBuild. For Unix users, we suggest passing the
-i
option tomake
, and also possibly-j 2
(or more) to run the build process in parallel. Building all of the regression tests for the Boost libraries can take a long time.make -i -j 2
Note: If you change Boost source files in a way that affects your tests, you will need to re-run the build process to update the libraries and tests before moving on to the next step.
- Once regression tests have finished building, go into the command line or command prompt and enter the Boost binary directory. Then, run the command
ctest
to execute all of the regression tests. The
ctest
executable will be stored in the binary directory for CMake. On Unix platforms, this is the same place whereccmake
resides. On Windows platforms, it will be inC:\Program Files\CMake\bin
. The ctest program should produce output like the following:Start processing tests Test project /Users/dgregor/Projects/boost-darwin 1/ 22 Testing any::any_test Passed 2/ 22 Testing any::any_to_ref_test Passed 3/ 22 Testing function::lib_function_test Passed 4/ 22 Testing function::function_n_test Passed 5/ 22 Testing function::allocator_test Passed 6/ 22 Testing function::stateless_test Passed 7/ 22 Testing function::lambda_test Passed 8/ 22 Testing function::function_test_fail1 ***Failed - supposed to fail 9/ 22 Testing function::function_test_fail2 ***Failed - supposed to fail 10/ 22 Testing function::function_30 Passed 11/ 22 Testing function::function_arith_cxx98 Passed 12/ 22 Testing function::function_arith_porta Passed 13/ 22 Testing function::sum_avg_cxx98 Passed 14/ 22 Testing function::sum_avg_portable Passed 15/ 22 Testing function::mem_fun_cxx98 Passed 16/ 22 Testing function::mem_fun_portable Passed 17/ 22 Testing function::std_bind_cxx98 Passed 18/ 22 Testing function::std_bind_portable Passed 19/ 22 Testing function::function_ref_cxx98 Passed 20/ 22 Testing function::function_ref_portabl Passed 21/ 22 Testing function::contains_test Passed 22/ 22 Testing function::contains2_test Passed 100% tests passed, 0 tests failed out of 22
Here, we have only enabled testing of the Boost.Any and Boost.Function libraries, by setting
TEST_BOOST_ANY
andTEST_BOOST_FUNCTION
toON
while all of the otherTEST_BOOST_
LIBNAME options are set toOFF
.
Again: This "
ctest
" step runs the tests without first running a build. If you change a source file and run thectest
step you will see that no build is invoked.
- To run just a subset of the tests, pass
-R
and a regular expression toctest
. For example, to run all of the Python tests, use:ctest -R python::
There is also a -E
(exclude) option which does the inverse of -R
. ctest --help
shows the full list of options.
Submitting Regression Test Results
The ctest
command can be used by individual developers to test local changes to their libraries. The same program can also be used to build all of Boost, run its regression tests, and submit the results to a central server where others can view them. Currently, regression test results based on the CMake build system are displayed on the Dart server at http://dart.resophonic.com/boost_1_34_0/Dashboard/.
To submit "experimental" results to the Dart server, configure a Boost binary tree by following the configuration instructions in the section Configuring and Building Boost and then enable regression testing via the BOOST_TESTING
option, as described above. At this point, don't build anything! We'll let CTest do that work. You may want to customize some of the advanced CMake options, such as SITE
(to give your site name), and MAKECOMMAND
(which, for makefile targets, governs the top-level make call when building Boost). Finally, go into the Boost binary directory and execute:
ctest -D Experimental
CTest will then reconfigure Boost, build all of the Boost libraries and regression tests, execute the regression tests, and submit the results to the Dart dashboard at http://dart.resophonic.com/boost_1_34_0/Dashboard/. Results submitted to the dashboard usually show up in 15-20 minutes, and can be browsed from the web site given.
Note: Although we are running regression tests on several flavors of Unix on a nightly basis, we have done very little work to ensure that regression testing runs smoothly on other platforms. We will remedy this problem in the near future.