https://svn.boost.org/trac10/ticket/7419 <p> Boost.Test is designed on the assumption that a special <code>main()</code> function will be provided to ensure the test framework is initialised properly and that the function needed to add test suites for execution is properly provided. This works well for simple test programs. </p> <p> However another important use case we have and I'm sure others too is that we also wish to run our production systems 'as-if' in production but have tests execute when run in a 'testing' mode. </p> <p> This has the benefit of allowing full use of all the Boost.Test tool facilities and makes test points easily recognisable. Additionally it also means the test out analysis tools that we use for Boost.Test in our unit testing can be used without modification. Hence on a coding level we use the same test code as for unit tests and we then benefit from being able to directly interpret the results, just as we do for normal unit tests. </p> <p> In order to support this within the Boost.Test framework (we use the unit-test framework for this as we simply require access to the tools and reporting facilities) we must make it possible to write a test runner wrapper than can ultimately call <code>boost::unit_test::framework::init()</code> more than once. </p> <p> A minimal test runner class might look like this (taken from a real code but stripped down so will not work as is): </p> <div class="wiki-code"><div class="code"><pre> <span class="k">namespace</span> <span class="n">test</span> <span class="p">{</span> <span class="k">class</span> <span class="nc">runner</span> <span class="p">{</span> <span class="k">public</span><span class="o">:</span> <span class="k">using</span> <span class="n">master_test_suite_t</span> <span class="o">=</span> <span class="n">boost</span><span class="o">::</span><span class="n">unit_test</span><span class="o">::</span><span class="n">master_test_suite_t</span><span class="p">;</span> <span class="k">using</span> <span class="n">test_suite_t</span> <span class="o">=</span> <span class="n">boost</span><span class="o">::</span><span class="n">unit_test</span><span class="o">::</span><span class="n">test_suite</span><span class="p">;</span> <span class="k">using</span> <span class="n">call_add_tests_t</span> <span class="o">=</span> <span class="n">boost</span><span class="o">::</span><span class="n">function</span><span class="o">&lt;</span><span class="kt">void</span><span class="p">(</span> <span class="n">master_test_suite_t</span><span class="o">&amp;</span> <span class="p">)</span><span class="o">&gt;</span><span class="p">;</span> <span class="k">public</span><span class="o">:</span> <span class="k">static</span> <span class="kt">void</span> <span class="n">store_properties</span><span class="p">(</span> <span class="cm">/* properties from commandline or config file */</span> <span class="p">)</span> <span class="p">{</span> <span class="k">auto</span><span class="o">&amp;</span> <span class="n">Runner</span> <span class="o">=</span> <span class="n">instance</span><span class="p">();</span> <span class="n">Runner</span><span class="p">.</span><span class="n">store_arguments</span><span class="p">(</span> <span class="cm">/* properties */</span> <span class="p">);</span> <span class="p">}</span> <span class="k">static</span> <span class="kt">int</span> <span class="n">run</span><span class="p">(</span> <span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">&amp;</span> <span class="n">Title</span><span class="p">,</span> <span class="k">const</span> <span class="n">call_add_tests_t</span><span class="o">&amp;</span> <span class="n">Callback</span> <span class="p">)</span> <span class="p">{</span> <span class="k">auto</span><span class="o">&amp;</span> <span class="n">Runner</span> <span class="o">=</span> <span class="n">instance</span><span class="p">();</span> <span class="n">Runner</span><span class="p">.</span><span class="n">set_title</span><span class="p">(</span> <span class="n">Title</span> <span class="p">);</span> <span class="n">Runner</span><span class="p">.</span><span class="n">set_add_tests_callback</span><span class="p">(</span> <span class="n">Callback</span> <span class="p">);</span> <span class="cp">#ifdef BOOST_TEST_ALTERNATIVE_INIT_API</span> <span class="n">boost</span><span class="o">::</span><span class="n">unit_test</span><span class="o">::</span><span class="n">init_unit_test_func</span> <span class="n">init_func</span> <span class="o">=</span> <span class="o">&amp;</span><span class="n">runner</span><span class="o">::</span><span class="n">init_unit_test</span><span class="p">;</span> <span class="cp">#else</span> <span class="n">boost</span><span class="o">::</span><span class="n">unit_test</span><span class="o">::</span><span class="n">init_unit_test_func</span> <span class="n">init_func</span> <span class="o">=</span> <span class="o">&amp;</span><span class="n">runner</span><span class="o">::</span><span class="n">init_unit_test_suite</span><span class="p">;</span> <span class="cp">#endif</span> <span class="k">return</span> <span class="n">boost</span><span class="o">::</span><span class="n">unit_test</span><span class="o">::</span><span class="n">unit_test_main</span><span class="p">(</span> <span class="n">init_func</span><span class="p">,</span> <span class="n">Runner</span><span class="p">.</span><span class="n">argc</span><span class="p">(),</span> <span class="n">Runner</span><span class="p">.</span><span class="n">argv</span><span class="p">()</span> <span class="p">);</span> <span class="p">}</span> <span class="k">private</span><span class="o">:</span> <span class="n">runner</span><span class="p">()</span> <span class="p">{</span> <span class="p">}</span> <span class="k">static</span> <span class="n">runner</span><span class="o">&amp;</span> <span class="n">instance</span><span class="p">()</span> <span class="p">{</span> <span class="k">static</span> <span class="n">runner</span> <span class="n">Runner</span><span class="p">;</span> <span class="k">return</span> <span class="n">Runner</span><span class="p">;</span> <span class="p">}</span> <span class="k">private</span><span class="o">:</span> <span class="k">static</span> <span class="kt">bool</span> <span class="n">init_unit_test</span><span class="p">()</span> <span class="p">{</span> <span class="n">init_unit_test_suite</span><span class="p">(</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span> <span class="p">);</span> <span class="k">return</span> <span class="nb">true</span><span class="p">;</span> <span class="p">}</span> <span class="k">static</span> <span class="n">boost</span><span class="o">::</span><span class="n">unit_test</span><span class="o">::</span><span class="n">test_suite</span><span class="o">*</span> <span class="n">init_unit_test_suite</span><span class="p">(</span> <span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span><span class="o">*</span> <span class="n">argv</span><span class="p">[]</span> <span class="p">)</span> <span class="p">{</span> <span class="k">auto</span> <span class="n">AddTests</span> <span class="o">=</span> <span class="n">runner</span><span class="o">::</span><span class="n">instance</span><span class="p">().</span><span class="n">get_add_tests_callback</span><span class="p">();</span> <span class="k">if</span><span class="p">(</span> <span class="n">AddTests</span> <span class="p">)</span> <span class="p">{</span> <span class="n">AddTests</span><span class="p">(</span> <span class="n">boost</span><span class="o">::</span><span class="n">unit_test</span><span class="o">::</span><span class="n">framework</span><span class="o">::</span><span class="n">master_test_suite</span><span class="p">()</span> <span class="p">);</span> <span class="p">}</span> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span> <span class="p">}</span> <span class="k">private</span><span class="o">:</span> <span class="kt">void</span> <span class="n">store_arguments</span><span class="p">(</span> <span class="cm">/* properties */</span> <span class="p">)</span> <span class="p">{</span> <span class="n">ArgumentStrings_</span> <span class="o">=</span> <span class="cm">/* copy of Boost.Test relevant properties */</span> <span class="n">Arguments_</span> <span class="o">=</span> <span class="cm">/* pointers to actual strings in ArgumentStrings_ */</span> <span class="p">}</span> <span class="k">private</span><span class="o">:</span> <span class="kt">int</span> <span class="n">argc</span><span class="p">()</span> <span class="k">const</span> <span class="p">{</span> <span class="k">return</span> <span class="n">ArgumentStrings_</span><span class="p">.</span><span class="n">size</span><span class="p">();</span> <span class="p">}</span> <span class="kt">char</span><span class="o">**</span> <span class="n">argv</span><span class="p">()</span> <span class="k">const</span> <span class="p">{</span> <span class="k">return</span> <span class="k">const_cast</span><span class="o">&lt;</span><span class="kt">char</span><span class="o">**&gt;</span><span class="p">(</span> <span class="o">&amp;</span><span class="n">Arguments_</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="p">);</span> <span class="p">}</span> <span class="kt">void</span> <span class="n">set_title</span><span class="p">(</span> <span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">&amp;</span> <span class="n">Title</span> <span class="p">)</span> <span class="p">{</span> <span class="k">if</span><span class="p">(</span> <span class="n">ArgumentStrings_</span><span class="p">.</span><span class="n">empty</span><span class="p">()</span> <span class="p">)</span> <span class="p">{</span> <span class="n">ArgumentStrings_</span><span class="p">.</span><span class="n">push_back</span><span class="p">(</span> <span class="n">Title</span> <span class="p">);</span> <span class="n">Arguments_</span><span class="p">.</span><span class="n">push_back</span><span class="p">(</span> <span class="n">ArgumentStrings_</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">c_str</span><span class="p">()</span> <span class="p">);</span> <span class="n">Arguments_</span><span class="p">.</span><span class="n">push_back</span><span class="p">(</span> <span class="mi">0</span> <span class="p">);</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="n">ArgumentStrings_</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">Title</span><span class="p">;</span> <span class="n">Arguments_</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">ArgumentStrings_</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">c_str</span><span class="p">();</span> <span class="p">}</span> <span class="p">}</span> <span class="kt">void</span> <span class="n">set_add_tests_callback</span><span class="p">(</span> <span class="k">const</span> <span class="n">call_add_tests_t</span><span class="o">&amp;</span> <span class="n">Callback</span> <span class="p">)</span> <span class="p">{</span> <span class="n">Callback_</span> <span class="o">=</span> <span class="n">Callback</span><span class="p">;</span> <span class="p">}</span> <span class="k">const</span> <span class="n">call_add_tests_t</span><span class="o">&amp;</span> <span class="n">get_add_tests_callback</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="n">Callback_</span><span class="p">;</span> <span class="p">}</span> <span class="k">private</span><span class="o">:</span> <span class="n">call_add_tests_t</span> <span class="n">Callback_</span><span class="p">;</span> <span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o">&lt;</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">&gt;</span> <span class="n">ArgumentStrings_</span><span class="p">;</span> <span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o">&lt;</span><span class="k">const</span> <span class="kt">char</span><span class="o">*&gt;</span> <span class="n">Arguments_</span><span class="p">;</span> <span class="p">};</span> <span class="p">}</span> <span class="c1">// test</span> </pre></div></div><p> We would use a runner like this as follows: </p> <div class="wiki-code"><div class="code"><pre> <span class="c1">// Implement a function to add tests to the Master Test Suite</span> <span class="c1">// Note this can be a member function and simply call</span> <span class="c1">// another member to collate the test suites</span> <span class="kt">void</span> <span class="nf">add_tests</span><span class="p">(</span> <span class="n">test</span><span class="o">::</span><span class="n">runner</span><span class="o">::</span><span class="n">master_test_suite_t</span><span class="o">&amp;</span> <span class="n">MasterSuite</span> <span class="p">)</span> <span class="p">{</span> <span class="n">MasterSuite</span><span class="p">.</span><span class="n">add</span><span class="p">(</span> <span class="n">scenario_test_suite</span><span class="p">()</span> <span class="p">);</span> <span class="p">}</span> <span class="c1">// A</span> <span class="n">test</span><span class="o">::</span><span class="n">runner</span><span class="o">::</span><span class="n">test_suite_t</span><span class="o">*</span> <span class="n">scenario_test_suite</span><span class="p">()</span> <span class="p">{</span> <span class="k">auto</span><span class="o">*</span> <span class="n">test_suite</span> <span class="o">=</span> <span class="n">BOOST_TEST_SUITE</span><span class="p">(</span> <span class="n">Name_</span><span class="p">.</span><span class="n">c_str</span><span class="p">()</span> <span class="p">);</span> <span class="n">test_suite</span><span class="o">-&gt;</span><span class="n">add</span><span class="p">(</span> <span class="n">BOOST_TEST_CASE</span><span class="p">(</span> <span class="n">boost</span><span class="o">::</span><span class="n">bind</span><span class="p">(</span> <span class="o">&amp;</span><span class="n">self_t</span><span class="o">::</span><span class="n">validate_incoming_messages</span><span class="p">,</span> <span class="k">this</span><span class="o">-&gt;</span><span class="n">shared_from_this</span><span class="p">()</span> <span class="p">)</span> <span class="p">)</span> <span class="p">);</span> <span class="n">test_suite</span><span class="o">-&gt;</span><span class="n">add</span><span class="p">(</span> <span class="n">BOOST_TEST_CASE</span><span class="p">(</span> <span class="n">boost</span><span class="o">::</span><span class="n">bind</span><span class="p">(</span> <span class="o">&amp;</span><span class="n">self_t</span><span class="o">::</span><span class="n">validate_outgoing_messages</span><span class="p">,</span> <span class="k">this</span><span class="o">-&gt;</span><span class="n">shared_from_this</span><span class="p">()</span> <span class="p">)</span> <span class="p">)</span> <span class="p">);</span> <span class="n">test_suite</span><span class="o">-&gt;</span><span class="n">add</span><span class="p">(</span> <span class="n">BOOST_TEST_CASE</span><span class="p">(</span> <span class="n">boost</span><span class="o">::</span><span class="n">bind</span><span class="p">(</span> <span class="o">&amp;</span><span class="n">self_t</span><span class="o">::</span><span class="n">validate_data_messages</span><span class="p">,</span> <span class="k">this</span><span class="o">-&gt;</span><span class="n">shared_from_this</span><span class="p">()</span> <span class="p">)</span> <span class="p">)</span> <span class="p">);</span> <span class="k">return</span> <span class="n">test_suite</span><span class="p">;</span> <span class="p">}</span> <span class="c1">// Calling test() actually causes the tests to be executed</span> <span class="c1">// We assume in this example that the object that</span> <span class="c1">// test() belongs to is a shared_ptr</span> <span class="kt">void</span> <span class="n">test</span><span class="p">()</span> <span class="p">{</span> <span class="n">test</span><span class="o">::</span><span class="n">runner</span><span class="o">::</span><span class="n">run</span><span class="p">(</span> <span class="n">Name_</span><span class="p">,</span> <span class="n">boost</span><span class="o">::</span><span class="n">bind</span><span class="p">(</span> <span class="o">&amp;</span><span class="n">self_t</span><span class="o">::</span><span class="n">add_tests</span><span class="p">,</span> <span class="k">this</span><span class="o">-&gt;</span><span class="n">shared_from_this</span><span class="p">(),</span> <span class="n">_1</span> <span class="p">)</span> <span class="p">);</span> <span class="p">}</span> </pre></div></div><p> The attached patch makes the above code possible by doing 2 things: </p> <ol><li>If the macro <code>BOOST_TEST_USE_QUALIFIED_COMMANDLINE_ARGUMENTS</code> is defined then the Boost.Test command line arguments are scoped using <code>boost.test.*</code> as a prefix. This means commandline arguments used to control tests will not interfere with command line arguments used to control the code being tested. <code>--log_level</code> is an excellent example of why you might need to do that. <code>--boost.test.log_level</code> is much clearer and will not clash. </li></ol><ol start="2"><li>Calls to <code>boost::unit_test::framework::init()</code> will occur each time <code>run()</code> is called. However with the patch each call resets the framework to a consistent initial state and the tests proceed as expected as if the framework had been called once from <code>main()</code>. </li></ol><p> We are using this patch with boost 1.48 right now (and so it is very slightly different due to the changes between 1.48 and trunk) and it allows us to run our main processes in a 'test' mode generating full test result output with timings that are then sent to our build server. It works very well, has no impact on normal unit test usage and is a very small change for a very big gain. </p> <p> This patch along with the patches attached to these tickets: </p> <p> </p><div><dl class="wiki compact"><dt><a class="closed" href="/trac10/ticket/7397" title="Boost.Test, since boost 1.48 is using the deprecated Boost.Timer class">#7397</a></dt><dd>Boost.Test, since boost 1.48 is using the deprecated Boost.Timer class</dd><dt><a class="closed" href="/trac10/ticket/7410" title="Test Units (Cases and Suites) in Boost.Test do not capture __FILE__ ...">#7410</a></dt><dd>Test Units (Cases and Suites) in Boost.Test do not capture __FILE__ and __LINE__ at declaration point making it impossible to provide source file linking using external test management tools</dd><dt><a class="new" href="/trac10/ticket/7417" title="Detailed test status is not available in the Boost.Test log (status, ...">#7417</a></dt><dd>Detailed test status is not available in the Boost.Test log (status, assertions, passed) and so live test case status cannot be tracked</dd></dl></div><p> </p> <p> ...combine to significantly improve the value of Boost.Test in terms of integration with thirdparty analysis tools and integration with other code libraries. </p> <p> Given the small changes that are made I'd like to see them hopefully make it into the next boost release. </p> en-us /htdocs/site/boost.png https://svn.boost.org/trac10/ticket/7419 Trac 1.4.3 Jamie Allsop <ja11sop@…> Mon, 24 Sep 2012 23:56:31 GMT https://svn.boost.org/trac10/ticket/7419 https://svn.boost.org/trac10/ticket/7419 <ul> <li><strong>attachment</strong> → <span class="trac-field-new">boost_test_support_multiple_test_runs.diff</span> </li> </ul> Ticket Gennadiy Rozental Sat, 18 Jul 2015 07:33:06 GMT https://svn.boost.org/trac10/ticket/7419#comment:1 https://svn.boost.org/trac10/ticket/7419#comment:1 <ul> <li><strong>status</strong> <span class="trac-field-old">new</span> → <span class="trac-field-new">closed</span> </li> <li><strong>resolution</strong> → <span class="trac-field-new">obsolete</span> </li> <li><strong>milestone</strong> <span class="trac-field-old">To Be Determined</span> → <span class="trac-field-new">Boost 1.59.0</span> </li> </ul> <p> Please open new ticket against trunk version of Boost.Test with detailed description of what you are trying to achieve </p> Ticket