Opened 7 years ago
Closed 7 years ago
#11416 closed Bugs (invalid)
BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES should result in a fail verdict if the expectation is not fullfilled
| Reported by: | Owned by: | Gennadiy Rozental | |
|---|---|---|---|
| Milestone: | To Be Determined | Component: | test |
| Version: | Boost 1.57.0 | Severity: | Problem |
| Keywords: | Cc: |
Description
Given this test:
BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES(fail_test, 1)
BOOST_AUTO_TEST_CASE( fail_test ) {
BOOST_CHECK(1 == 1);
}
I would assume that this test results in a overall failed test, but the macro only prints a message on the console: "Test case fail_test has fewer failures than expected" and does not touch the verdict of the test. This is IMHO useless, because its easy to miss (Especially if CI is used)
I introduced that test case into our test suite to ensure that the tests are working correctly (before we had tests that did nothing and resulted in PASS, because NDEBUG was defined)
Note:
See TracTickets
for help on using tickets.
This is by design: an error is raised only when the number of failures inside the test case is higher than the expected one. For checking the strict number of failures, regular assertions should be used.
This is explained in details here: http://www.boost.org/doc/libs/develop/libs/test/doc/html/boost_test/testing_tools/expected_failures.html