Support multiple calls to framework::init() allowing wrappers to support running tests using test tools in full systems

[Jamie Allsop <ja11sop@…>]

Boost.Test is designed on the assumption that a special main() function will be provided to ensure the test framework is initialised properly and that the function needed to add test suites for execution is properly provided. This works well for simple test programs.

However another important use case we have and I'm sure others too is that we also wish to run our production systems 'as-if' in production but have tests execute when run in a 'testing' mode.

This has the benefit of allowing full use of all the Boost.Test tool facilities and makes test points easily recognisable. Additionally it also means the test out analysis tools that we use for Boost.Test in our unit testing can be used without modification. Hence on a coding level we use the same test code as for unit tests and we then benefit from being able to directly interpret the results, just as we do for normal unit tests.

In order to support this within the Boost.Test framework (we use the unit-test framework for this as we simply require access to the tools and reporting facilities) we must make it possible to write a test runner wrapper than can ultimately call boost::unit_test::framework::init() more than once.

A minimal test runner class might look like this (taken from a real code but stripped down so will not work as is):

namespace test {

class runner
{
public:

    using master_test_suite_t  = boost::unit_test::master_test_suite_t;
    using test_suite_t         = boost::unit_test::test_suite;
    using call_add_tests_t     = boost::function<void( master_test_suite_t& )>;

public:

    static
    void store_properties( /* properties from commandline or config file */ )
    {
        auto& Runner = instance();
        Runner.store_arguments( /* properties */ );
    }

    static
    int run( const std::string& Title, const call_add_tests_t& Callback )
    {
        auto& Runner = instance();

        Runner.set_title( Title );
        Runner.set_add_tests_callback( Callback );

#ifdef BOOST_TEST_ALTERNATIVE_INIT_API
        boost::unit_test::init_unit_test_func init_func = &runner::init_unit_test;
#else
        boost::unit_test::init_unit_test_func init_func = &runner::init_unit_test_suite;
#endif
        return boost::unit_test::unit_test_main( init_func, Runner.argc(), Runner.argv() );
    }

private:

    runner()
    {
    }

    static runner& instance()
    {
        static runner Runner;
        return Runner;
    }

private:

    static
    bool init_unit_test()
    {
        init_unit_test_suite( 0, 0 );
        return true;
    }

    static
    boost::unit_test::test_suite* init_unit_test_suite( int argc, char* argv[] )
    {
        auto AddTests = runner::instance().get_add_tests_callback();
        if( AddTests )
        {
            AddTests( boost::unit_test::framework::master_test_suite() );
        }
        return 0;
    }

private:

    void store_arguments( /* properties */  )
    {
        ArgumentStrings_ = /* copy of Boost.Test relevant properties */
        Arguments_ = /* pointers to actual strings in ArgumentStrings_ */
    }

private:

    int argc() const
    {
        return ArgumentStrings_.size();
    }

    char** argv() const
    {
        return const_cast<char**>( &Arguments_[0] );
    }

    void set_title( const std::string& Title )
    {
        if( ArgumentStrings_.empty() )
        {
            ArgumentStrings_.push_back( Title );
            Arguments_.push_back( ArgumentStrings_[0].c_str() );
            Arguments_.push_back( 0 );
        }
        else
        {
            ArgumentStrings_[0] = Title;
            Arguments_[0] = ArgumentStrings_[0].c_str();
        }
    }

    void set_add_tests_callback( const call_add_tests_t& Callback )
    {
        Callback_ = Callback;
    }

    const call_add_tests_t& get_add_tests_callback()
    {
        return Callback_;
    }

private:

    call_add_tests_t            Callback_;
    std::vector<std::string>    ArgumentStrings_;
    std::vector<const char*>    Arguments_;
};

} // test

We would use a runner like this as follows:

    // Implement a function to add tests to the Master Test Suite
    // Note this can be a member function and simply call
    // another member to collate the test suites
    void add_tests( test::runner::master_test_suite_t& MasterSuite )
    {
        MasterSuite.add( scenario_test_suite() );
    }

    // A
    test::runner::test_suite_t* scenario_test_suite()
    {
        auto* test_suite = BOOST_TEST_SUITE( Name_.c_str() );

        test_suite->add( BOOST_TEST_CASE( boost::bind( &self_t::validate_incoming_messages, this->shared_from_this() ) ) );
        test_suite->add( BOOST_TEST_CASE( boost::bind( &self_t::validate_outgoing_messages, this->shared_from_this() ) ) );
        test_suite->add( BOOST_TEST_CASE( boost::bind( &self_t::validate_data_messages,     this->shared_from_this() ) ) );

        return test_suite;
    }

    // Calling test() actually causes the tests to be executed
    // We assume in this example that the object that
    // test() belongs to is a shared_ptr
    void test()
    {
        test::runner::run( Name_, boost::bind( &self_t::add_tests, this->shared_from_this(), _1 ) );
    }

The attached patch makes the above code possible by doing 2 things:

1. If the macro BOOST_TEST_USE_QUALIFIED_COMMANDLINE_ARGUMENTS is defined then the Boost.Test command line arguments are scoped using boost.test.* as a prefix. This means commandline arguments used to control tests will not interfere with command line arguments used to control the code being tested. --log_level is an excellent example of why you might need to do that. --boost.test.log_level is much clearer and will not clash.

2. Calls to boost::unit_test::framework::init() will occur each time run() is called. However with the patch each call resets the framework to a consistent initial state and the tests proceed as expected as if the framework had been called once from main().

We are using this patch with boost 1.48 right now (and so it is very slightly different due to the changes between 1.48 and trunk) and it allows us to run our main processes in a 'test' mode generating full test result output with timings that are then sent to our build server. It works very well, has no impact on normal unit test usage and is a very small change for a very big gain.

This patch along with the patches attached to these tickets:

#7397

Boost.Test, since boost 1.48 is using the deprecated Boost.Timer class

#7410

Test Units (Cases and Suites) in Boost.Test do not capture __FILE__ and __LINE__ at declaration point making it impossible to provide source file linking using external test management tools

#7417

Detailed test status is not available in the Boost.Test log (status, assertions, passed) and so live test case status cannot be tracked

...combine to significantly improve the value of Boost.Test in terms of integration with thirdparty analysis tools and integration with other code libraries.

Given the small changes that are made I'd like to see them hopefully make it into the next boost release.

[Gennadiy Rozental]

Please open new ticket against trunk version of Boost.Test with detailed description of what you are trying to achieve