| | 1 | = Testing Boost with CMake = |
| | 2 | Boost's CMake-based build system provides regression testing via [http://www.cmake.org/Wiki/CMake_Testing_With_CTest CTest], which comes as part of CMake. This regression testing system can be used by Boost developers to test their libraries locally and also by testers and users to submit regression tests to a [http://public.kitware.com/Dart Dart server], which collects and reports regression-testing results from different sites. This document assumes that the reader has already learned how to [http://svn.boost.org/trac/boost/wiki/CMakeConfigAndBuild configure and build] Boost using CMake. |
| | 3 | |
| | 4 | 1. Re-run the CMake configuration for Boost, either by running the CMake configuration program from the Start menu (on Microsoft Windows) or executing the command-line CMake configuration in Unix: |
| | 5 | {{{ |
| | 6 | ccmake $BOOST_ROOT |
| | 7 | }}} |
| | 8 | |
| | 9 | 2. Toggle the `BUILD_TESTING` option in the CMake configuration to `ON`, and then re-configure (either by clicking "Configure", on Microsoft Windows, or pressing `c`, on Unix). You may notice that configuration takes significantly longer when we are building all of the regression tests. |
| | 10 | |
| | 11 | 3. If you want, you can now disable regression testing for some or all Boost libraries by toggling the `TEST_BOOST_LIBNAME` options that appear. Be sure to re-configure CMake once you are done tweaking these options. |
| | 12 | |
| | 13 | 4. Generate makefiles or project files, by clicking "OK" (on Microsoft Windows) or pressing `g` (on Unix). |
| | 14 | |
| | 15 | 5. Follow the same building process described in the section [http://svn.boost.org/trac/boost/wiki/CMakeConfigAndBuild Configuring and Building Boost], either be running your make program from the Boost binary directory or compiling from within your IDE (e.g., Microsoft Visual Studio). For Unix users, we suggest passing the `-i` option to `make`, and also possibly `-j 2` (or more) to run the build process in parallel. Building all of the regression tests for the Boost libraries can take a long time. |
| | 16 | {{{ |
| | 17 | make -i -j 2 |
| | 18 | }}} |
| | 19 | |
| | 20 | '''Note''': If you change Boost source files in a way that affects your tests, you will need to re-run the build process to update the libraries and tests before moving on to the next step. |
| | 21 | |
| | 22 | 6. Once regression tests have finished building, go into the command line or command prompt and enter the Boost binary directory. Then, run the command |
| | 23 | {{{ |
| | 24 | ctest |
| | 25 | }}} |
| | 26 | |
| | 27 | to execute all of the regression tests. The `ctest` executable will be stored in the binary directory for CMake. On Unix platforms, this is the same place where `ccmake` resides. On Windows platforms, it will be in `C:\Program Files\CMake\bin`. The ctest program should produce output like the following: |
| | 28 | {{{ |
| | 29 | Start processing tests |
| | 30 | Test project /Users/dgregor/Projects/boost-darwin |
| | 31 | 1/ 22 Testing any::any_test Passed |
| | 32 | 2/ 22 Testing any::any_to_ref_test Passed |
| | 33 | 3/ 22 Testing function::lib_function_test Passed |
| | 34 | 4/ 22 Testing function::function_n_test Passed |
| | 35 | 5/ 22 Testing function::allocator_test Passed |
| | 36 | 6/ 22 Testing function::stateless_test Passed |
| | 37 | 7/ 22 Testing function::lambda_test Passed |
| | 38 | 8/ 22 Testing function::function_test_fail1 ***Failed - supposed to fail |
| | 39 | 9/ 22 Testing function::function_test_fail2 ***Failed - supposed to fail |
| | 40 | 10/ 22 Testing function::function_30 Passed |
| | 41 | 11/ 22 Testing function::function_arith_cxx98 Passed |
| | 42 | 12/ 22 Testing function::function_arith_porta Passed |
| | 43 | 13/ 22 Testing function::sum_avg_cxx98 Passed |
| | 44 | 14/ 22 Testing function::sum_avg_portable Passed |
| | 45 | 15/ 22 Testing function::mem_fun_cxx98 Passed |
| | 46 | 16/ 22 Testing function::mem_fun_portable Passed |
| | 47 | 17/ 22 Testing function::std_bind_cxx98 Passed |
| | 48 | 18/ 22 Testing function::std_bind_portable Passed |
| | 49 | 19/ 22 Testing function::function_ref_cxx98 Passed |
| | 50 | 20/ 22 Testing function::function_ref_portabl Passed |
| | 51 | 21/ 22 Testing function::contains_test Passed |
| | 52 | 22/ 22 Testing function::contains2_test Passed |
| | 53 | |
| | 54 | 100% tests passed, 0 tests failed out of 22 |
| | 55 | }}} |
| | 56 | |
| | 57 | Here, we have only enabled testing of the Boost.Any and Boost.Function libraries, by setting `TEST_BOOST_ANY` and `TEST_BOOST_FUNCTION` to `ON` while all of the other `TEST_BOOST_LIBNAME` options are set to `OFF`. |
| | 58 | |
| | 59 | == Submitting Regression Test Results == |
| | 60 | The `ctest` command can be used by individual developers to test local changes to their libraries. The same program can also be used to build all of Boost, run its regression tests, and submit the results to a central server where others can view them. Currently, regression test results based on the CMake build system are displayed on the Dart server at http://dart.resophonic.com/boost_1_34_0/Dashboard/. |
| | 61 | |
| | 62 | To submit "experimental" results to the Dart server, configure a Boost binary tree by following the configuration instructions in the section [http://svn.boost.org/trac/boost/wiki/CMakeConfigAndBuild Configuring and Building Boost] and then enable regression testing via the `BOOST_TESTING` option, as described above. At this point, don't build anything! We'll let CTest do that work. You may want to customize some of the advanced CMake options, such as `SITE` (to give your site name), and `MAKECOMMAND` (which, for makefile targets, governs the top-level make call when building Boost). Finally, go into the Boost binary directory and execute: |
| | 63 | {{{ |
| | 64 | ctest -D Experimental |
| | 65 | }}} |
| | 66 | |
| | 67 | CTest will then reconfigure Boost, build all of the Boost libraries and regression tests, execute the regression tests, and submit the results to the Dart dashboard at http://dart.resophonic.com/boost_1_34_0/Dashboard/. Results submitted to the dashboard usually show up in 15-20 minutes, and can be browsed from the web site given. |
| | 68 | |
| | 69 | '''Note''': Although we are running regression tests on several flavors of Unix on a nightly basis, we have done very little work to ensure that regression testing runs smoothly on other platforms. We will remedy this problem in the near future. |
