| | 1 | [[PageOutline]] |
| | 2 | |
| | 3 | = Boost Test Policies and Protocols = #intro |
| | 4 | |
| | 5 | The Boost libraries are intended to be both reliable and |
| | 6 | portable. Every experienced programmer knows that means each |
| | 7 | library must be tested against a suitable number of test cases, |
| | 8 | on a wide range of platforms, and then tested again (regression |
| | 9 | tested) every time a change is made and before every |
| | 10 | release. |
| | 11 | |
| | 12 | "Quality assurance based on a wide range of targeted tests" |
| | 13 | as one of the key answers to C.A.R Hoare's question "How did |
| | 14 | software get so reliable without proof." |
| | 15 | |
| | 16 | == Regression test == #regression |
| | 17 | |
| | 18 | Boost uses an automatic |
| | 19 | [http://www.boost.org/doc/libs/release/tools/regression/doc/index.html regression test suite] |
| | 20 | which generates HTML |
| | 21 | [http://www.boost.org/development/testing.html#RegressionTesting compiler status tables]. |
| | 22 | |
| | 23 | == Test Policy == #policy |
| | 24 | |
| | 25 | === Required === #policy.required |
| | 26 | |
| | 27 | * Every Boost library should supply one or more suitable |
| | 28 | test programs to be exercised by the Boost |
| | 29 | [http://www.boost.org/doc/libs/release/tools/regression/doc/index.html regression test suite]. |
| | 30 | In addition to the usual compile-link-run |
| | 31 | tests expecting successful completion, compile-only or |
| | 32 | compile-and-link-only tests may be performed, and success for |
| | 33 | the test may be defined as failure of the steps. |
| | 34 | * Test program execution must report errors by returning a |
| | 35 | non-zero value. They may also write to stdout or stderr, but |
| | 36 | that output should be relatively brief. Regardless of other |
| | 37 | output, a non-zero return value is the only way the |
| | 38 | regression test framework will recognize an error has |
| | 39 | occurred. Note that test programs to be included in the |
| | 40 | status tables must compile, link, and run quickly since the |
| | 41 | tests are executed many, many, times. |
| | 42 | * Libraries with time consuming tests should be divided |
| | 43 | into a fast-execution basic test program for the status |
| | 44 | tables, and a separate full-coverage test program for |
| | 45 | exhaustive test cases. The basic test should concentrate on |
| | 46 | compilation issues so that the status tables accurately |
| | 47 | reflect the library's likelihood of correct compilation on a |
| | 48 | platform. |
| | 49 | * If for any reason the usual test policies do not apply to |
| | 50 | a particular library, an alternate test strategy must be |
| | 51 | implemented. |
| | 52 | * A |
| | 53 | [http://www.boost.org/doc/libs/release/tools/regression/doc/index.html#Adding_new_test Jamfile] |
| | 54 | to drive the regression tests for the |
| | 55 | library. |
| | 56 | |
| | 57 | === Optional (but highly recommended) === #policy.optional |
| | 58 | |
| | 59 | The |
| | 60 | [http://www.boost.org/doc/libs/release/libs/test/index.html Boost Test Library] |
| | 61 | provides many useful components which ease the |
| | 62 | construction of test programs. |
| | 63 | |
| | 64 | * Use the library's |
| | 65 | [http://www.boost.org/doc/libs/release/libs/test/doc/components/test_tools/index.html Test Tools] |
| | 66 | for the construction of simple test programs |
| | 67 | that do not need much structure. |
| | 68 | * Use the library's |
| | 69 | [http://www.boost.org/doc/libs/release/libs/test/doc/components/utf/index.html Unit Test Framework] |
| | 70 | for the construction of more complex test |
| | 71 | programs that need to be structured into individual tests and |
| | 72 | test suites. |
| | 73 | |
| | 74 | == Suggested Protocol for Fixing Bugs or Adding Features. == #fixing-bugs |
| | 75 | |
| | 76 | * First, add regression test cases that detects the bug or |
| | 77 | tests the feature. Sometimes adding one case suggests similar |
| | 78 | untested cases, and they are added too. |
| | 79 | * Second, for bugs, run the regression test and verify that |
| | 80 | the bug is now detected. |
| | 81 | * Third, then, and only then, fix the bug or add the feature. |
| | 82 | * Finally, rerun the full regression tests - sometimes the |
| | 83 | change breaks something else. |
| | 84 | |
| | 85 | == History == #history |
| | 86 | |
| | 87 | [http://www.boost.org/doc/libs/release/tools/regression/doc/index.html#History See Regression Test History]. |
| | 88 | |
| | 89 | == Acknowledgements == #acknowledgements |
| | 90 | |
| | 91 | Written by Beman Dawes. Jens Maurer, Paul Moore, Gary Powell |
| | 92 | and Jeremy Siek contributed helpful suggestions. |
