id,summary,reporter,owner,description,type,status,milestone,component,version,severity,resolution,keywords,cc
6129,Include warning counts in reports,olaf.meeuwissen@…,Gennadiy Rozental,"I [http://lists.boost.org/boost-users/2011/11/71744.php mentioned] this on the mailing list before, but I would like to be able to report the number of times assertions in the `BOOST_WARN` ""family"" have failed.

For test cases, the (XML) report includes information similar to:
{{{
result=""passed""
assertions_passed=""x"" assertions_failed=""x"" expected_failures=""x""
}}}
and for test suites
{{{
result=""passed""
assertions_passed=""x"" assertions_failed=""x"" expected_failures=""x""
test_cases_passed=""x"" test_cases_failed=""x""
test_cases_skipped=""x"" test_cases_aborted=""x""
}}}

I guess I would like to see a count of something like `assertions_ignored` added to the test cases and an additional `ignored` value for their `result` attribute.  This `ignored` value is used for test cases that fail one or more of their `BOOST_WARN` type assertions but pass ''all'' of their `BOOST_REQUIRE` and `BOOST_CHECK` assertions.  If all assertions pass, the test case passes.  If a `BOOST_CHECK` assertion fails the test case fails.  If a `BOOST_REQUIRE` assertion fails, the test case is aborted and fails.

For the test suites, the same story goes for the `assertions_ignored` and `ignored` additions, where an `ignored` result is assigned if one or more test cases with that result were encountered and none of the test cases `failed`.  If all test cases passed, the test suite passes.  If one or more test cases failed, the test suite fails.  In addition to the existing `test_cases_*` counts, I'd like to see a `test_cases_ignored` count.

Summarizing, you'd get something like this:

 - counts of assertions `passed`, `ignored` or `failed`
 - a value of `passed`, `ignored` or `failed` for the `result` attribute
 - counts of test cases `passed`, `ignored`, `failed`, `skipped` or `aborted`
 - counts of test suites `passed`, `ignored`, `failed` or `skipped`?

Please note the distinction between `skipped` (was not even started so no assertions have been evaluated) and `ignored` (ran to completion with one or more failed `BOOST_WARN` assertions).

AFAIK, test suites cannot be aborted.  Can they?

In reviewing my ticket, I noticed that `skipped` and `aborted` should perhaps be added as test case result values as well.  That way, the report includes everything you ever wanted to know about all test cases' and test suites' states.

Adding the output of failed assertions would be nice too, but that's outside the scope of this ticket.",Feature Requests,closed,To Be Determined,test,Boost 1.47.0,Problem,fixed,,