Changes between Version 7 and Version 8 of BestPracticeHandbook

Timestamp:

May 5, 2015, 1:47:10 PM (7 years ago)

Author:

Niall Douglas

Comment:

--

BestPracticeHandbook

@@ -224 +224 @@
 
 
-== 4. Consider making it possible to use an XML outputting unit testing framework, even if not enabled by default ==
+== 4. Strongly consider using free CI per-commit testing, even if you have a private CI ==
+
+Despite that [https://travis-ci.org/ Travis] provides free of cost per-commit continuous integration testing for Linux and OS X, and that [http://www.appveyor.com/ Appveyor] provides the same for Microsoft Windows, there were still libraries in those reviewed which made use of neither and furthermore had no per-commit CI testing whatsoever.
+
+I'll be blunt: not having per-commit CI testing is '''unacceptable''' in this modern day and age and is an excellent sign of a library author not committed to software quality. Especially when such CI services are free of cost, and it's purely your laziness that you haven't configured them yet.
+
+So first things first, if your C++ 11/14 library is not using any form of per-commit testing yet, '''go add Travis and Appveyor right support now'''. Configuration is extremely easy if your project lives on Github, simply login into both services using your github account and enable your project. Next add a suitable .travis.yml and an .appveyor.yml file to the root of your project, and push the commit. Watch Travis and Appveyor build and run your CI suitable unit test suite, and report back on Github (especially if it's a pull request) whether the build and tests passed or failed. From now on when someone issues a pull request fixing a bug, you'll instantly know if that pull request compiles and passes all unit tests on Linux, OS X and Windows, and much more importantly, ''so will the pull request submitter'' and they usually will go fix problems themselves so you the maintainer never need find out a pull request is defective on some build target.
+
+Example travis.yml's to clone from:
+
+* https://github.com/krzysztof-jusiak/di/blob/cpp14/.travis.yml
+* https://github.com/BoostGSoC13/boost.afio/blob/master/.travis.yml
+
+Example appveyor.yml's to clone from:
+
+* https://github.com/krzysztof-jusiak/di/blob/cpp14/.appveyor.yml
+* https://github.com/BoostGSoC13/boost.afio/blob/master/appveyor.yml
+
+Both Travis and Appveyor are excellent for getting an 90% confidence signal that some commit did not break something. For a free service with little configuration effort that's fantastic. However if you want a 98% confidence signal you will need to spend a few months of your life configuring your own [https://jenkins-ci.org/ Jenkins CI installation], most of that time will be learning how not to configure Jenkins as Jenkins is a black, black art indeed - but again great for being free of cost given the return on investment. Once mastered, Jenkins can do almost anything from per-commit testing to soak testing to input fuzz testing to automating a long list of tasks for you (e.g. diffing and synchronising two forks of a repo for you by bisecting commit histories against unit testing), but it '''will''' take many dedicated months to acquire the skills to configure a maintainable and stable Jenkins install.
+
+Should you add Travis and Appveyor CI support if you already are using your own private Jenkins CI?
+
+I think the answer is uncategorically yes. The reasons are these:
+
+1. Having Travis + Appveyor badges (see https://raw.githubusercontent.com/krzysztof-jusiak/di/cpp14/README.md for example Markdown for badges) on your open source project is a universally recognised signal of attention to quality.
+2. Other free tooling such as [https://coveralls.io/ Coveralls.io] have built in integration for github and travis. Hooking Jenkins into Coveralls isn't hard, but it "just works" with Travis instead and that's a similar pattern with most free tooling which consumes CI results.
+3. Future tooling by Boost which dashboards Boost libraries and/or ranks libraries by a quality score will almost certainly automate on Travis and Appveyor being queryable by their RESTful APIs. In other words, placing your library in Github and adding Travis and Appveyor CI support has the highest chance of working immediately with any future Boost tooling with minimal additional effort by you.
+
+
+== 5. Strongly consider compiling your code with static analysis tools ==
+
+== 5. Strongly consider running a pass of your unit tests under valgrind and the runtime sanitisers ==
+
+== 5. Consider making it possible to use an XML outputting unit testing framework, even if not enabled by default ==
 
 A very noticeable trend in the libraries above is that around half use good old C assert() and static_assert() instead of a unit testing framework.
@@ -303 +336 @@
 * https://github.com/ned14/Boost.APIBind
 
-== 5. Consider not doing compiler feature detection yourself ==
-
-TODO
+
+== 6. Consider breaking up your testing into per-commit CI testing, 24 hour soak testing, and input fuzz testing ==
+
+When a library is small, you can generally get away with running all tests per commit, and as that is easier that is usually what one does.
+
+However as a library grows and matures, you should really start thinking about categorising your tests into quick ones suitable for per-commit testing, long ones suitable for 24 hour soak testing, and adding fuzz testing whereby an AST analysis tool will try executing your functions with input deliberately designed to exercise least followed code path combinations ("coverage informed fuzz testing"). I haven't mentioned the distinction between http://stackoverflow.com/questions/4904096/whats-the-difference-between-unit-functional-acceptance-and-integration-test unit testing and functional testing and integration testing] here as I personally think that distinction not useful for libraries mostly developed in a person's free time (due to lack of resources, and the fact we all prefer to develop instead of test, one tends to fold unit and functional and integration testing into a single amorphous set of tests which don't strictly delineate as really we should, and instead of proper unit testing one tends to substitute automated fuzz testing, which really isn't the same thing but it does end up covering similar enough ground to make do).
+
+There are two main techniques to categorising tests, and each has substantial pros and cons.
+
+The first technique is that you tag tests in your test suite with keyword tags, so "ci-quick", "ci-slow", "soak-test" and so on. The unit test framework then lets you select at execution time which set of tags you want. This sounds great, but there are two big drawbacks. The first is that each test framework has its own way of doing tags, and these are invariably not compatible so if you have a switchable Boost.Test/CATCH/Google Test generic test code setup then you'll have a problem with the tagging. One nasty but portable workaround I use is magic test naming and then using a regex test selector string, this is why I have categorised slashes in the test names exampled in the section above so I can select tests by category via their name. The second drawback is that you will find that tests often end up calling some generic implementation with different parameters, and you have to go spell out many sets of parameters in individual test cases when one's gut feeling is that those parameters really should be fuzz variables directly controlled by the test runner. Most test frameworks support passing variables into tests from the command line, but again this varies strongly across test frameworks in a way hard to write generic test code.
+
+The second technique is a hack, but a very effective one. One simply parameterises tests with environment variables, and then code calling the unit test program can configure special behaviour by setting environment variables before each test iteration. This technique is especially valuable for converting per-commit tests into soak tests because you simply configure an environment variable which means ITERATIONS to something much larger, and now the same per-commit tests are magically transformed into soak tests. The big drawback here is that just iterating per commit tests a lot more does not a proper soak test suite make, and one can fool oneself into believing your code is highly stable and reliable when it is really only highly stable and reliable at running per commit tests, which obviously it will always be because you run those exact same patterns per commit so those are always the use patterns which will behave the best. Boost.AFIO is 24 hour soak tested on its per-commit tests, and yet I have been more than once surprised at segfaults caused by someone simply doing operations in a different order than the tests did them :(
+
+Regarding fuzz testing, there are a number of tools available for C++, though all are not quick and require lots of sustained CPU time to calculate and execute all possible code path variations. One of the most promising going into the long term is LLVM's fuzz testing facilities which are summarised at http://llvm.org/docs/LibFuzzer.html as they make excellent use of the clang sanitisers to find the bad code paths. I haven't played with it yet with Boost.AFIO, though it is very high on my todo list as I have very little unit testing in AFIO (only functional and integration testing), and fuzz testing of my internal routines would be an excellent way of implementing comprehensive exception safety testing which I am also missing (and feel highly unmotivated to implement by hand).
+
+
+== 7. Consider not doing compiler feature detection yourself ==
+
+Something extremely noticeable about nearly all the reviewed C++ 11/14 libraries is that they manually do compiler feature detection in their config.hpp, usually via old fashioned compiler version checking. This tendency is not unsurprising as the number of potential C++ compilers your code usually needs to handle has essentially shrunk to three, and the chances are very high that three compiler will be upper bound going into the long term future. This makes compiler version checking a lot more tractable than say fifteen years ago.
+
+However, C++ 1z is expected to provide a number of feature detection macros via the work of SG-10, and GCC and clang already partially support these, especially in very recent compiler releases. To fill in the gaps in older editions of GCC and clang, and indeed MSVC at all, you might consider making use of the header file at https://github.com/ned14/Boost.APIBind/blob/master/include/cpp_feature.h which provides the following SG-10 feature detection macros on all versions of GCC, clang and MSVC:
+
+ __cpp_exceptions:: Whether C++ exceptions are available
+ __cpp_rtti:: Whether C++ RTTI is available
+
+ __cpp_alias_templates::
+ __cpp_alignas::
+ __cpp_decltype::
+ __cpp_default_function_template_args::
+ __cpp_defaulted_functions::
+ __cpp_delegated_constructors::
+ __cpp_deleted_functions::
+ __cpp_explicit_conversions::
+ __cpp_generalized_initializers::
+ __cpp_implicit_moves::
+ __cpp_inheriting_constructors::
+ __cpp_inline_namespaces::
+ __cpp_lambdas::
+ __cpp_local_type_template_args::
+ __cpp_noexcept::
+ __cpp_nonstatic_member_init::
+ __cpp_nullptr::
+ __cpp_override_control::
+ __cpp_reference_qualified_functions::
+ __cpp_range_for::
+ __cpp_raw_strings::
+ __cpp_rvalue_references::
+ __cpp_static_assert::
+ __cpp_thread_local::
+ __cpp_auto_type::
+ __cpp_strong_enums::
+ __cpp_trailing_return::
+ __cpp_unicode_literals::
+ __cpp_unrestricted_unions::
+ __cpp_user_defined_literals::
+ __cpp_variadic_templates::
+
+ __cpp_contextual_conversions::
+ __cpp_decltype_auto::
+ __cpp_aggregate_nsdmi::
+ __cpp_digit_separators::
+ __cpp_init_captures::
+ __cpp_generic_lambdas::
+ __cpp_relaxed_constexpr::
+ __cpp_return_type_deduction::
+ __cpp_runtime_arrays::
+ __cpp_variable_templates::
+
+The advantage of using these SG-10 macros in C++ 11/14 code is threefold:
+
+1. It should be future proof.
+2. It's a lot nicer than testing compiler versions.
+3. It expands better if a fourth C++ compiler suddenly turned up.
+
+Why use the https://github.com/ned14/Boost.APIBind/blob/master/include/cpp_feature.h header file instead of doing it by hand?
+
+1. Complete compiler support for GCC, clang and MSVC all versions.
+2. Updates in compiler support will get reflected into cpp_feature.h for you.
+3. You benefit from any extra compilers added automatically.
+4. If you're using Boost.APIBind you automatically get cpp_feature.h included for you as soon as you include any APIBind header file.
+
+Incidentally Boost.APIBind wraps these macros into Boost.Config compatible macros in https://github.com/ned14/Boost.APIBind/blob/master/include/boost/config.hpp which would be included, as with Boost, using "boost/config.hpp".