BOOST_CHECK_MESSAGE - 2 problems

[Maxim Yanchenko <Maxim.Yanchenko@…>]

1) The 2nd argument is always evaluated. I believe it's natural for a user to expect that BOOST_CHECK_MESSAGE follows short circuit logic, i.e. if the check was satisfied, no more user code is called.

Example:

BOOST_CHECK_MESSAGE( x == y, report_non_equal(x,y) )

report_non_equal(x,y) should be evaluated only if the check failed, as a natural precondition for the report_non_equal is that its arguments are not equal (e.g. a user can divide something by the difference of x and y without checking that it's not zero).

2) The message is printed even in case of success with --log_level=success. People usually place there a message like "failed" in the message. For example, see the Boost regression test suite (just picked a couple of examples):

./libs/test/test/test_tools_test.cpp:210:    BOOST_CHECK_MESSAGE( test_pred1(), "Checking predicate failed" )
./libs/iostreams/test/large_file_test.cpp:382:    BOOST_CHECK_MESSAGE(amt == 1, "failed reading character");
./libs/serialization/test/test_non_intrusive.cpp:152:    BOOST_CHECK_MESSAGE(pa1 == &a, "Copy of pointer not correctly restored");

And the Boost.Test docs is not an exception:

BOOST_WARN_MESSAGE( res > 1, "sin(45){" << res << "} is <= 1. Hmm.. Strange. " );

As you can see, the message users provide usually describes the failure ("X failed"), not the test itself ("testing X"). So when they use --log_level=success they are very confused when they see that something successfully failed (as a message from UTF "check 'X failed' passed" means that the failure X was expected).

In my case, I used the message to report the position where two strings mismatched using simple "std::mismatch-begin" (naturally expecting it to be called only in a case of failure), so with --log_level=success it printed that all strings mismatched in their end with funny and ambiguous messages like "info: check 'Strings "asd" and "asd" mismatch at position 3' passed" (does it mean that they were expected to mismatch at pos 3?), while they worked perfectly when the check failed: "error in "test": Strings "asd" and "asx" mismatch at position 2"

I believe if you run Boost regression test suite with --log_level=success you will see a lot of similar absurd messages.

From the first glance, the fix for both problems seems obvious (haven't tried yet, though) - in case of BOOST_<level>_MESSAGE the following version of BOOST_CHECK_IMPL should be used:

#define BOOST_CHECK_IMPL( P, check_descr, TL, CT )                         \
do {                                                                       \
    BOOST_TEST_PASSPOINT();                                                \
    const bool result = P;                                                 \
    if (result)                                                            \
      BOOST_TEST_TOOL_IMPL( check_impl, result, BOOST_TEST_STRINGIZE( P ), TL, CT ), 0 );         \
    else                                                                   \
      BOOST_TEST_TOOL_IMPL( check_impl, result, check_descr, TL, CT ), 0 );\
} while( ::boost::test_tools::dummy_cond )                                 \
/**/
                       

[Gennadiy Rozental]

We can't do what you suggest for 2 reasons:

1. P is not always generate bool, sometimes it's predicate_result.
2. In some cases error message with level all is indeed funny. But I'd argue that this is very rarely the issue and in most cases this level is not intended to be human readable. Also in general case we can't be sure your message is better that what it is now.

As for the solution to your problems I see at least two:

1. Lazy message.

The check description message can be made "lazy". In other words you can implement structure which perform actual message generation only in the point when it's requested (operator<<(std::ostream&, YourStruct). I guess I can implement some kind of helpers for you, but it's unclear yet how general this problem is. Try this route and let me know if you see something worth putting in the library.

2. predicate_result

Instead of using BOOST_CHECK_MESSAGE, you cna use BOOST_CHECK and implement custom predicate function returning predicate_result structure. You can see more details here: http://www.boost.org/doc/libs/1_40_0/libs/test/doc/html/utf/testing-tools/custom-predicate.html This predicate function can produce different messages in case if check succeeded or failed.

Let me know if you still having troubles.