Include warning counts in reports

[olaf.meeuwissen@…]

I ​mentioned this on the mailing list before, but I would like to be able to report the number of times assertions in the BOOST_WARN "family" have failed.

For test cases, the (XML) report includes information similar to:

result="passed"
assertions_passed="x" assertions_failed="x" expected_failures="x"

and for test suites

result="passed"
assertions_passed="x" assertions_failed="x" expected_failures="x"
test_cases_passed="x" test_cases_failed="x"
test_cases_skipped="x" test_cases_aborted="x"

I guess I would like to see a count of something like assertions_ignored added to the test cases and an additional ignored value for their result attribute. This ignored value is used for test cases that fail one or more of their BOOST_WARN type assertions but pass all of their BOOST_REQUIRE and BOOST_CHECK assertions. If all assertions pass, the test case passes. If a BOOST_CHECK assertion fails the test case fails. If a BOOST_REQUIRE assertion fails, the test case is aborted and fails.

For the test suites, the same story goes for the assertions_ignored and ignored additions, where an ignored result is assigned if one or more test cases with that result were encountered and none of the test cases failed. If all test cases passed, the test suite passes. If one or more test cases failed, the test suite fails. In addition to the existing test_cases_* counts, I'd like to see a test_cases_ignored count.

Summarizing, you'd get something like this:

• counts of assertions passed, ignored or failed
• a value of passed, ignored or failed for the result attribute
• counts of test cases passed, ignored, failed, skipped or aborted
• counts of test suites passed, ignored, failed or skipped?

Please note the distinction between skipped (was not even started so no assertions have been evaluated) and ignored (ran to completion with one or more failed BOOST_WARN assertions).

AFAIK, test suites cannot be aborted. Can they?

In reviewing my ticket, I noticed that skipped and aborted should perhaps be added as test case result values as well. That way, the report includes everything you ever wanted to know about all test cases' and test suites' states.

Adding the output of failed assertions would be nice too, but that's outside the scope of this ticket.

[Gennadiy Rozental]

Fixed in trunk in 81320. Thank you for the submission

[olaf.meeuwissen@…]

Thanks! I had a quick look through changeset [81320]. Too bad there's quite a few unrelated changes mixed in but in general it seems to add what I requested. I don't have time now to give it a spin though :-(