Changes between Initial Version and Version 1 of CMakeAddingRegressionTests

Timestamp:

Jun 12, 2007, 4:55:58 AM (15 years ago)

Author:

Douglas Gregor

Comment:

Initial stab at documentation for adding regression tests

CMakeAddingRegressionTests

@@ +1 @@
+= Adding Regression Tests with CMake =
+This page describes how to add regression tests for a Boost library in the CMake-based build system. Before adding regression tests, make sure you have already followed the directions for [wiki:CMakeAddingALibrary adding a library to CMake], so that the CMake system recognizes your Boost library, and (if necessary) [wiki:CMakeAddingCompiledLibrary adding a compiled library to CMake]. We also assume that you have already configured your build tree for regression testing, by enabling the `BUILD_TESTING` option described in the section [wiki:CMakeTesting Testing Boost with CMake].
+
+In this page, we will assume that your library resides in the subdirectory `libs/libname`, and that tests for this library are stored in `libs/libname/test`. The test directory should be listed via `TESTDIRS` in the use of the [wiki:CMakeLibraryProject boost_library_project macro], as described in an earlier section, [wiki:CMakeAddingALibrary "Adding a Library to CMake"]. Follow these steps to add this new library into Boost's build system. If your library has multiple testing directories listed after `TESTDIRS`, follow these steps for each one.
+
+  1. Create a new file `libs/libname/test/CMakeLists.txt` file with your favorite text editor. This file will contain instructions for building and running each of the regression tests for your library. If you reconfigure CMake now (i.e., by running `ccmake` or `cmake` from the command line, or using the Windows CMake GUI), you will see a new option `BOOST_TEST_LIBNAME`. This option allows you to turn on or off regression testing for your library. 
+
+  2. For each test that only needs to be compiled (but not executed), add a `compile` or `compile_fail` test using the [wiki:CMakeTestCompile boost_test_compile] or [wiki:CMakeTestCompileFail boost_test_compile_fail] macros, respectively. The most basic usage of these macros provides only the test name, e.g.,
+{{{
+  boost_test_compile(compile_test)
+  boost_test_compile_fail(compile_fail_test)
+}}}
+
+  This code will create two regression tests. The first, `compile_test`, will try to compile the source file `compile_test.cpp` in the current source directory. If the compile is successful, the regression test passes. If the compile fails, the regression test fails. The second regression test works the opposite way: it will try to compile `compile_fail_test.cpp`: if the compilation is successful, the regression test fails. When you run the regression tests (e.g., by calling `ctest` from the build directory), the regression tests will execute and produce output like the following:
+
+{{{
+Running tests...
+Start processing tests
+Test project /Users/dgregor/Projects/boost-darwin
+  1/  2 Testing libname::compile_test            Passed
+  2/  2 Testing libname::compile_fail_test     ***Failed - supposed to fail
+
+100% tests passed, 0 tests failed out of 2
+}}}
+
+  3. For any tests that need to be built and executed, use the [wiki:CMakeTestRun boost_test_run] or [wiki:CMakeTestRunFail boost_test_run_fail] macros. Both tests will build, link and execute a regression test. The `boost_test_run` macro expects that executable to return an exit code of zero, while the `boost_test_run_fail` macro expects that executable to return a non-zero exit code. For example, we might build a simple test `simple_test` from the source file `simple_test.cpp`:
+    {{{
+boost_test_run(simple_test)
+    }}}
+
+  4. Often, we'll want to link against our own Boost library, which we do using the `DEPENDS` argument to `boost_test_run`:
+    {{{
+boost_test_run(big_test big_test1.cpp big_test2.cpp
+  DEPENDS boost_libname-static
+    }}}
+
+  Here, we have created a test `big_test`, built from the source files `big_test1.cpp` and `big_test2.cpp`, which will link against the static library for `boost_libname`. We could create a similar test that links against the shared library for `boost_libname`, passing along compilation flags specific to the shared library:
+    {{{
+boost_test_run(big_test_dll big_test1.cpp big_test2.cpp
+  DEPENDS boost_libname-shared
+  COMPILE_FLAGS "-DBOOST_LIBNAME_DYN_LINK=1"
+    }}}
+
+  5. Some tests require command-line arguments. For example, say we want to pass `-loop 1000` to a randomized test. We can do so using the `ARGS` argument to `boost_test_run` (or `boost_test_run_fail`):
+  {{{
+boost_test_run(random_test ARGS "-loop" "1000" DEPENDS boost_libname-static)
+  }}}
+
+
+Once you have finished describing your regression tests to the CMake system, you're done! Your library will now build, test, and install with CMake and this behavior should be portable across many different platforms.