Adding Test Groups to BOOST.TEST

[sumedhghaisas@…]

Hello, My name is Sumedh. Lately I am developing a C++ library which needs extensive benchmarking and testing. For this, and for my own curiosity I created a C++ benchmarking and testing library called MANAK (​https://github.com/Manak-org/Manak). In this library I created a useful feature(or so I think :P) which I call 'test group'. 'Test group' is a set of tests which can be instantiated multiple times with different parameters and templates. The most basic use case of this feature occurs when you are trying to test a machine learning library where a same objective is satisfied by different algorithms and they all have same API thus creating similar test structure. In this scenario one has to add all the tests multiple times for each algorithm assuming you are templating the function. The solution I implemented in MANAK is following -

template<size_t index>
MANAK_GROUP(TestGroup);

std::string c_name;

GINIT(const std::string& str)
{
  c_name = str;
}

MANAK_AUTO_GROUP_TEST_CASE("Test1_with_" + c_name)
{
  
}

MANAK_AUTO_GROUP_TEST_CASE("Test2_with_" + c_name)
{
  
} 

MANAK_GROUP_END();

MANAK_ADD_GROUP(TestGroup<Class1>, "Class1");
MANAK_ADD_GROUP(TestGroup<Class2>, "Class2");

Here the test group 'TestGroup' is instantiated twice with classes 'Class1' and 'Class2'. The tests inside a test group share common variables which can be set with GINIT as shown above(GINIT can also be overloaded.). This helps us to name the test differently when using with different settings. Test groups works fluently with suites and fixtures and depending on the implementation of them the rules are set. I have analyzed the implementation of BOOST.TEST and realized that my implementation can easily be ported to BOOST.TEST. I have many decorators to this concepts which adds more functionality to it but the basic concept is this. I am a regular contributor to ML library MLPACK (​http://mlpack.org/) which uses BOOST.TEST as the primary testing framework and we there feel the need of this feature.

Tell me your opinion about this feature and all the comments/suggestions are welcome. I have already looked at the plan of BOOST.TEST and I think test group can be implemented in BOOST.TEST. If you like the idea I will try porting the basic code on which we can develop. :)

[Gennadiy Rozental]

The feature you are describing looks very similar to data driven testing we introduced in Boost.Test v3 (boost 1.59). Please give it a shot and let us know if anything you need is missing.

[sumedhghaisas@…]

Okay I am entirely sure but I think they are different concepts, but I might be wrong. I will explain my reasons in following points -

Data driven tests in Boost.Test applies only to a single test test where as test group lets you initialize multiple tests with parameters. For example - Lets say you have class A and Class B. They are have equivalent API. I want to run 30 tests on each one. With current framework. I have to write 30 tests and initialize them individually for both the classes. Like I have to initialize every test for every class where as test group will combine those tests in a group and lets you initialize them on a class with one single line. So if you have 10 such classes all you need are 10 lines to initialize those tests where as in current framework one has to initialize 300 test cases individually.

The concept of Data driven tests is to let the user flexibility to run single test on various parameters where as the main concept of test group is to let user flexibility to create a group of tests which can be initialized on parameters.

template<size_t index>
MANAK_GROUP(TestGroup);

std::string c_name;

GINIT(const std::string& str)
{
  c_name = str;
}

MANAK_AUTO_GROUP_TEST_CASE("Test1_with_" + c_name)
{
  
}

MANAK_AUTO_GROUP_TEST_CASE("Test2_with_" + c_name)
{
  
} 

MANAK_GROUP_END();

MANAK_ADD_GROUP(TestGroup<Class1>, "Class1");
MANAK_ADD_GROUP(TestGroup<Class2>, "Class2");
MANAK_ADD_GROUP(TestGroup<Class3>, "Class3");

Like the code above will be written like below in current framework

template<typename T>
void fun1() {}

template<typename T>
void fun2() {}

template<typename T>
void fun3() {}

register fun1 with class 1
register fun1 with class 2
register fun1 with class 3

register fun2 with class 1
register fun2 with class 2
register fun2 with class 3

register fun3 with class 1
register fun3 with class 2
register fun3 with class 3

In this case if user has to add a new test he has to add that test for each class, or if user needs to add new class he has to add all the functions to this class. Test group makes this situation more constructive and abstracts the construction of test from the class its is being applied to.

I am extremely sorry if I am missing something and would happy to learn if such functionality already exists which we can use in our project. In such case please provide a very short sample case. Thanking you in advance.

[sumedhghaisas@…]

In the last comment the first sentence is "Okay I am not entirely sure but I think they are different concepts, but I might be wrong".

[Gennadiy Rozental]

Am I right and what you want is essentially data driven test suites? Does API like this look right:

std::vector params{"a", "b", "c", "d" };

BOOST_DATA_TEST_SUITE( s1, params )

BOOST_DATA_TEST_CASE(test1) {

sample is one of the string here

}

BOOST_DATA_TEST_CASE(test2) {

sample is one of the string here

}

BOOST_TEST_SUITE_END()

[sumedhghaisas@…]

Okay I think you are right... The extra thing that I would love to have is to pass multiple classes instead of multiple params... sort of like...

typedef std::tuple<Class A, Class B, Class C> t_params;

BOOST_DATA_TEST_SUITE( s1, t_params )

BOOST_DATA_TEST_CASE(test1) {

    T test_object(...)... // T is templated class

}

BOOST_DATA_TEST_CASE(test2) {

     T test_object(...)... // T is templated class

}

BOOST_TEST_SUITE_END()

Basically the tests will run for each class mentioned. I agree this way is much cleaner than my way. But I am not sure about the implementation of such thing, that I have to check. This kind of feature is helpful especially when many classes implement the same thing different ways with similar APIs.

[Raffi Enficiaud]

In the last case, if t_params does not hold data and is a type, you have the template test cases:

http://www.boost.org/doc/libs/1_59_0/libs/test/doc/html/boost_test/tests_organization/test_cases/test_organization_templates.html

You just write the body of test1/test2 and then you instanciate those tests with t_params.