Boost C++ Libraries: Ticket #12902: boost test labels

Raffi Enficiaud — Sat, 18 Mar 2017 13:30:24 GMT

Hi,

Do you mean you want at the same time:

only a test suite
exclude one specific label

? From this it says that label and paths are mutually exclusive, but to be honest, I do not know.

For preventing a specific label to run, the "relative_spec" in the above link should work (using "!").

Thu, 23 Mar 2017 10:23:49 GMT

I think we don't know how to actually combine these 2 filters. I tried something like 'Suite1!label2' but it does not work.

The documentation http://www.boost.org/doc/libs/1_63_0/libs/test/doc/html/boost_test/runtime_config/test_unit_filtering.html#boost_test.runtime_config.test_unit_filtering.command_line_control specifies "3. Shrink the statically defined set by disabling some of the enabled test cases." But this "shrinking" is never explained.

Our use case is "all tests in Suite1 except these marked with label2".

Also, can you create an unit test that is disabled by default and will be enabled only if a particular label will be provided, not just test suite path?

Sun, 07 May 2017 11:13:25 GMT

I also run into a problem with filtering test cases when labels are involved. For instance I have multiple test suites where each represents a specific feature that will be tested in the suite. Because some of the test cases actually take a long time to run I also added labels "quick" and "slow". The idea was to be able to run only the fast tests, or the slow tests, or all together.

My test suite structure looks something like this:

F1*
    test1@slow
    test2@quick
F2*
    test1@quick
    test2@slow

Selecting based on labels alone works well by specifying --run_test '@slow' etc. It would run the following tests:

F1*
    test1@slow
F2*
    test2@slow

It also works if I want to exclude, for example, the slow tests for a specific feature (or a selected subset of the features): --run_test 'F1:!@slow'

F1*
    test1@quick

However, if I only want to run the fast tests but exclude one feature (--run_test '@quick:!F1') it doesn't work. It seems to be impossible to exclude test cases that have been enabled by a label. Though, according to the documentation "the disabler takes the precedence over the enabler."

F1*
    test2@quick
F2*
    test1@quick

Additionally it would be nice to express something like "Run test suite F1 but only the tests labelled quick". I think currently that can only be achieved by negating all labels except for @quick. But if there are many labels it quickly becomes impractical.

owner, status, milestone changed

Raffi Enficiaud — Thu, 04 Jan 2018 10:01:13 GMT

owner changed from Gennadiy Rozental to Raffi Enficiaud
status new → assigned
milestone To Be Determined → Boost 1.66.0

I think there is an easy way to combine labels and path:

suite1/suite2/@labelX,case3 enabling case3 and all test cases that have @labelX, both within suite1/suite2
suite1/suite2/!@labelX,case3 enabling case3 and disabling all test cases that have @labelX, both within suite1/suite2
disabling should have precedence over enabling

For the case of --run_test '@quick:!F1' I need to investigate further, and this is actually a bug.

Concerning this:

Also, can you create an unit test that is disabled by default and will be enabled only if a particular label will be provided, not just test suite path?

I do not know, I need to check. Stay tuned!

milestone changed

Raffi Enficiaud — Thu, 21 Jun 2018 05:13:35 GMT

milestone Boost 1.66.0 → Boost 1.68.0