| 5 | | The Boost libraries are intended to be both reliable and |
| 6 | | portable. Every experienced programmer knows that means each |
| 7 | | library must be tested against a suitable number of test cases, |
| 8 | | on a wide range of platforms, and then tested again (regression |
| 9 | | tested) every time a change is made and before every |
| 10 | | release. |
| 11 | | |
| 12 | | "Quality assurance based on a wide range of targeted tests" |
| 13 | | as one of the key answers to C.A.R Hoare's question "How did |
| 14 | | software get so reliable without proof." |
| 15 | | |
| 16 | | == Regression test == #regression |
| 17 | | |
| 18 | | Boost uses an automatic |
| 19 | | [http://www.boost.org/doc/libs/release/tools/regression/doc/index.html regression test suite] |
| 20 | | which generates HTML |
| 21 | | [http://www.boost.org/development/testing.html#RegressionTesting compiler status tables]. |
| 22 | | |
| 23 | | == Test Policy == #policy |
| 24 | | |
| 25 | | === Required === #policy.required |
| 26 | | |
| 27 | | * Every Boost library should supply one or more suitable |
| 28 | | test programs to be exercised by the Boost |
| 29 | | [http://www.boost.org/doc/libs/release/tools/regression/doc/index.html regression test suite]. |
| 30 | | In addition to the usual compile-link-run |
| 31 | | tests expecting successful completion, compile-only or |
| 32 | | compile-and-link-only tests may be performed, and success for |
| 33 | | the test may be defined as failure of the steps. |
| 34 | | * Test program execution must report errors by returning a |
| 35 | | non-zero value. They may also write to stdout or stderr, but |
| 36 | | that output should be relatively brief. Regardless of other |
| 37 | | output, a non-zero return value is the only way the |
| 38 | | regression test framework will recognize an error has |
| 39 | | occurred. Note that test programs to be included in the |
| 40 | | status tables must compile, link, and run quickly since the |
| 41 | | tests are executed many, many, times. |
| 42 | | * Libraries with time consuming tests should be divided |
| 43 | | into a fast-execution basic test program for the status |
| 44 | | tables, and a separate full-coverage test program for |
| 45 | | exhaustive test cases. The basic test should concentrate on |
| 46 | | compilation issues so that the status tables accurately |
| 47 | | reflect the library's likelihood of correct compilation on a |
| 48 | | platform. |
| 49 | | * If for any reason the usual test policies do not apply to |
| 50 | | a particular library, an alternate test strategy must be |
| 51 | | implemented. |
| 52 | | * A |
| 53 | | [http://www.boost.org/doc/libs/release/tools/regression/doc/index.html#Adding_new_test Jamfile] |
| 54 | | to drive the regression tests for the |
| 55 | | library. |
| 56 | | |
| 57 | | === Optional (but highly recommended) === #policy.optional |
| 58 | | |
| 59 | | The |
| 60 | | [http://www.boost.org/doc/libs/release/libs/test/index.html Boost Test Library] |
| 61 | | provides many useful components which ease the |
| 62 | | construction of test programs. |
| 63 | | |
| 64 | | * Use the library's |
| 65 | | [http://www.boost.org/doc/libs/release/libs/test/doc/components/test_tools/index.html Test Tools] |
| 66 | | for the construction of simple test programs |
| 67 | | that do not need much structure. |
| 68 | | * Use the library's |
| 69 | | [http://www.boost.org/doc/libs/release/libs/test/doc/components/utf/index.html Unit Test Framework] |
| 70 | | for the construction of more complex test |
| 71 | | programs that need to be structured into individual tests and |
| 72 | | test suites. |
| 73 | | |
| 74 | | == Suggested Protocol for Fixing Bugs or Adding Features. == #fixing-bugs |
| 75 | | |
| 76 | | * First, add regression test cases that detects the bug or |
| 77 | | tests the feature. Sometimes adding one case suggests similar |
| 78 | | untested cases, and they are added too. |
| 79 | | * Second, for bugs, run the regression test and verify that |
| 80 | | the bug is now detected. |
| 81 | | * Third, then, and only then, fix the bug or add the feature. |
| 82 | | * Finally, rerun the full regression tests - sometimes the |
| 83 | | change breaks something else. |
| 84 | | |
| 85 | | == History == #history |
| 86 | | |
| 87 | | [http://www.boost.org/doc/libs/release/tools/regression/doc/index.html#History See Regression Test History]. |
| 88 | | |
| 89 | | == Acknowledgements == #acknowledgements |
| 90 | | |
| 91 | | Written by Beman Dawes. Jens Maurer, Paul Moore, Gary Powell |
| 92 | | and Jeremy Siek contributed helpful suggestions. |
| 93 | | |
| 94 | | Copyright Beman Dawes 2001. |
| | 3 | This has been move back to the [http://www.boost.org/development/test.html main site] |
