wiki:BestPracticeHandbook

Context Navigation

Version 10 (modified by Niall Douglas, 7 years ago) ( diff )
--

Links To Examples Of Best Practice For C++ 11/14 Libraries

originally written by Niall Douglas May 2015

As part of preparing my C++ Now 2015 presentation "A review of C++ 11/14 only Boost libraries" I examined ten C++ 11/14 mandatory libraries heading towards Boost which were:

Name	Authors	Min C++	Boost headers required	Entered Boost peer review queue	Description	Repository
Boost.Fiber	Oliver Kowalke	14	Context, Config	Conditionally accepted	a framework for micro-/userland-threads (fibers) scheduled cooperatively	https://github.com/olk/boost-fiber
Boost.AFIO	Niall Douglas, Paul Kirth	11	none	2013-10	strongly ordered portable asynchronous filesystem and file i/o extending ASIO	https://github.com/BoostGSoC13/boost.afio
Boost.DI	Krzysztof Jusiak	14	none	2015-01	provides compile time, macro free constructor dependency injection	https://github.com/krzysztof-jusiak/di
Boost.Hana	Louis Dionne	14	none	2015-04	functional programming in C++ based around transformations of heterogeneous collections of types.	https://github.com/ldionne/hana
Boost.Http	Vinícius dos Santos Oliveira	11	asio, filesystem, system, datetime, utility	2015-04	create applications that need to expose services through HTTP	https://github.com/BoostGSoC14/boost.http
Boost.APIBind	Niall Douglas	11	none	not submitted yet	toolkit for modularizing Boost libraries, managing versioned API and ABI dependency binds, makes it easy for Boost libraries to switch between use of the C++ 11 STL and the Boost STL	https://github.com/ned14/Boost.BindLib
Boost.Expected	Pierre Talbot, Vicente J. Botet Escriba	11	none	not submitted yet	category and functional programming in C++ based around the monadic expected<T, E>	https://github.com/ptal/expected
Boost.Tick	Paul Fultz II	11	none	not submitted yet	trait introspection and concept creator for C++11	https://github.com/pfultz2/Tick
Boost.Fit	Paul Fultz II	11	none	not submitted yet	provides utilities for functions and function objects	https://github.com/pfultz2/Fit
Boost.Sqlpp11	Roland Bock	11	none	not submitted yet	A type safe embedded domain specific language for SQL queries and results in C++	https://github.com/rbock/sqlpp11

There was very significant divergence in best practices between these libraries at the time of examination, and moreover the divergences were uneven possibly reflecting an uneven understanding of what best practice might be in C++ 11/14. Thence from my examination I have prepared the list below of what I felt were best practices in C++ 11/14 mandatory libraries with hyperlinks to the relevant source code files in the relevant libraries above. As what a "best practice" is or is not may vary according to opinion, I have attempted to provide a rationale for each of the suggestions below. It is up to library authors to weigh each suggestion and to decide whether to apply it, or not, to their library.

One of the most strikingly consistent features of these new libraries is the lack of dependence on Boost, even to the extent of not using the boost namespace as well as no Boost at all. The big advantage of this is modularity, so almost all of these libraries can be used standalone which is an oft requested feature by Boost end users. However this is likely not sustainable as more C++ 11/14 libraries become useful to other C++ 11/14 libraries and coupling therefore increases. I therefore dedicate significant effort below into how to most flexibly couple your library to other libraries to leave options open, the techniques for which have very significantly diversified in C++ 11.

One will note that the list below is much more C++ 11/14 focused than Boost focused. This is because it is derived from the first crop of C++ 11/14 mandatory Boost libraries. This is not a handbook for writing Boost libraries or even C++ 11/14 Boost libraries, if you want that first start reading here (note some of the guidelines on that page don't really apply to C++ 11/14 libraries) and then read here and here.

I have tried to keep these points generic to all C++ 11/14 libraries in the hope that they are useful far outside Boost. I have also ordered them with what I consider the most important ("strongly consider") first and not as important ("consider") later.

Links To Examples Of Best Practice For C++ 11/14 Libraries

1. Strongly consider using git and GitHub to host a copy of your library and its documentation

There are many source code control systems, with subversion and CVS being the two most recently popular of yesteryear. Probably the current most popular source code control system is Git, and despite its (Mingw i.e. non-native) port on Microsoft Windows being a bit flaky, it is very useful once mastered.

There is less widespread consensus about where to host your git repositories, with the most popular by far being github which is a proprietary service run by a profit making company. Nevertheless, one often hears strong arguments in favour of gitlab, bitbucket and many other alternatives.

All the Boost libraries are on github, as are all the libraries I reviewed. The huge advantage of github over all others is that the free tooling exampled below integrates easily with github. Choosing github therefore makes your life much easier. Note that as git is a distributed source code control system, you can keep a canonical master copy anywhere and write a script which autorefreshes the github copy, thus triggering any of the free tooling you have integrated there. In other words, don't necessarily place all your eggs in the github basket, and consider making github simply a medium for conveniently triggering the free tooling.

Github also provides free website hosting for HTML. Have a script automatically generate documentation and commit it to the gh-pages branch in your normal repo. This should present a copy of your HTML at http:// username.github.io/repository.

This is the script which generates the documentation for proposed Boost.AFIO, and indeed you can see the exact output generated by this script at http://boostgsoc13.github.io/boost.afio/. You may find it useful.

cd boost-local
rm -rf libs/afio/doc/doxy/doxygen_output/html
mkdir -p libs/afio/doc/doxy/doxygen_output/html
cd doc
../b2 -a ../libs/afio/doc
cd ../..
if [ ! -e publish ]; then
git clone -b gh-pages git@github.com:BoostGSoC13/boost.afio.git publish
fi
cd publish
git reset --hard b1414e11be50ff81124e2e1583f1bbb734ad9ead
cd ..
rm -rf publish/*
mkdir -p publish/doc/html
cp -a boost-local/doc/html/afio* publish/doc/html/
cp -a doc/src publish/doc/
cp -a doc/src/images/boost.png publish/
cp -af boost-local/doc/src publish/doc/
mkdir -p publish/libs/afio/doc
cp -a doc/* publish/libs/afio/doc/
cd boost-local/libs/afio/doc/doxy
doxygen
cd ../../../../../publish
cp -a ../boost-local/libs/afio/doc/doxy/doxygen_output/html .
cp -a ../Readme.md .
cp -a ../Readme.md Readme.html
echo '<html><head><title>Boost.AFIO documentation</title><meta http-equiv="refresh" content="300"/><body>
<h1>Boost.AFIO documentation</h1>
<p><a href="doc/html/afio.html">BoostBook format documentation</a></p>
<p><a href="html/index.html">Doxygen format documentation</a></p>
<p><a href="afio-stable.tar.bz2">Ready to go stable AFIO distribution with all git submodules (from master branch)</a></p>
<p></p>' > index.html
cat Readme.md | tail -n +4 >> index.html
echo '
</body></html>' >> index.html
git add .
git commit -a -m 'SHA stamp by Jenkins as building correctly' || true
git push -f

Some may wonder what the hard git reset to a SHA is for. This prevents the gh-pages branch continuously growing in storage by breaking history for the branch, and therefore making git clone times grow excessively. As the branch is purely for HTML publishing, breaking history like this is safe.

Other examples of libraries which use github for their documentation:

2. Strongly consider versioning your library's namespace using inline namespaces and requesting users to alias a versioned namespace instead of using it directly

C++ 11 adds a new feature called inline namespaces which are far more powerful than they first appear:

namespace boost { namespace afio { inline namespace v1 { /* stuff */ } } }
// Stuff is generated at the ABI link layer in boost::afio::v1
// But to the compiler everything boost::afio::v1::* appears identically in boost::afio::*
// INCLUDING for ADL and overload resolution
// In other words you can declare your code in boost::afio::v1, and use it as if declared in boost::afio.

// The other important new C++ 11 feature is namespace aliasing, so
namespace local { namespace afio = boost::afio; /* use afio::* and it all works */ }

The reason this pattern is so useful is because it greatly eases the lives of your end users and you the library maintainer in years to come when you need to break API compatibility. Let's take a case example: imagine the situation typical in 03 C++ libraries where library Boost.Foo uses dependent library Boost.AFIO:

namespace boost { namespace afio {
  struct foo {
    static void api(int i, double f);
  };
} }
...
namespace boost { namespace foo {
  boost::afio::api(1, 2);
} }

Imagine that you now release an API breaking refactor of Boost.AFIO, which would look like this:

namespace boost { namespace afio {
  struct foo {
    static void api(double f, int i);  // Oh dear, f and i have been swapped!
  };
} }
...
namespace boost { namespace foo {
  boost::afio::api(1, 2);  // This is probably now a bug!
} }

The users of Boost.Foo which uses boost::afio::foo::api() now finds that their library no longer passes its unit testing because foo::api() has changed from before. They will quite rightly throw up a fuss, and under Boost's present rules you will be asked to roll back your refactor until Boost.Foo has also been refactored to match your refactor. This causes inconvenience for you the maintainer of Boost.AFIO, the maintainer of Boost.Foo, and is a general pain for users. It also breaks modularity, increases coupling between libraries in a way which saddles you the maintainer of Boost.AFIO with the lack of maintenance or failure of timely maintenance of libraries dependent on AFIO, and I can't strongly enough recommend you don't blindly copy the 03 idiom of suggesting client code use your library directly using fully qualified namespacing.

The good news is we can make all this go away with inline namespaces and namespace aliasing, so consider this pattern instead:

namespace boost { namespace afio { inline namespace v1 {
  struct foo {
    static void api(int i, double f);
  };
} } }
...
namespace boost { namespace foo {
  // Probably somewhere in this library's config.hpp
  namespace afio = boost::afio;  // This is the key use change which needs to be strongly recommended to your library's users
  ...
  // In implementation code after config.hpp
  afio::api(1, 2);  // Note the no longer fully qualified use of afio. The local namespace alias is used to "symlink" to "the latest" version of Boost.AFIO
} }

Now imagine your refactor occurs as before:

namespace boost { namespace afio {
  // Probably defined by boost/afio.hpp which in turn includes boost/afio_v2.hpp
  inline namespace v2 {
    struct foo {
      static void api(double f, int i);  // new implementation
    };
  }
  // Probably defined by boost/afio_v1.hpp
  namespace v1 {
    struct foo {
      static void api(int i, double f);  // old implementation
    };
  }
} }
...
namespace boost { namespace foo {
  // Probably somewhere in this library's config.hpp
  namespace afio = boost::afio::v1;  // By changing this one single line we "fix" the problem. Earlier we included <boost/afio_v1.hpp> instead of <boost/afio.hpp>.
  ...
  // In implementation code after config.hpp
  afio::api(1, 2);  // And this finds the v1 AFIO implementation, not the v2 implementation
} }

What have we just achieved?

Library Boost.Foo dependent on Boost.AFIO no longer requires lots of refactoring work if Boost.AFIO is refactored. Just two lines changed in its config.hpp, something easy for the release managers to do.
Library Boost.AFIO can now be evolved far quicker than before, and simply keep shipping entire copies of legacy versions without problems with colliding namespaces. As end users get round to upgrading, legacy versions can be removed from the distro after a period of warning.

What are the problems with this technique?

You now need to ship multiple copies of your library, maintain multiple copies of your library, and make sure simultaneous use of multiple library versions in the same executable doesn't conflict. I suspect this cost is worth it for the added flexibility to evolve breaking changes for most library maintainers. You probably want to employ a per-commit run of http://ispras.linuxbase.org/index.php/ABI_compliance_checker to make sure you don't accidentally break the API (or ABI where appropriate) of a specific API version of your library. Also don't forget that git lets you submodule yourself on a different branch, so if you do ship multiple versions you can mount specific branches of yourself on that version's tracking branch within yourself.
The above technique alone is insufficient for header only end users where multiple versions of your library must coexist within the same translation unit. With some additional extra work, it is possible to allow multiple header only library versions to also coexist in the same translation unit, but this is covered in a separate recommendation below.
Many end users are not used to locally aliasing a library namespace in order to use it, and may continue to directly qualify it using the 03 idiom. You may consider defaulting to not using an inline namespace for the version to make sure users don't end up doing this in ways which hurt themselves, but that approach has both pros and cons.

Some fun extra things this technique enables:

Something not so obvious above is that you can also stub out fake copies of dependencies where that dependency is missing in the current config. For example, imagine optional compression support where your config.hpp either namespace aliases to boost::foo::compression either a real compression library, or an internal stub copy which actually does nothing. Your code is then written to assume a compression library aliased at boost::foo::compression and need not consider if it's actually there or not. The advantages here for reducing coupling are very obvious.
This technique is highly extensible to allow dependency injection of STL11 vs Boost on a per-feature basis e.g. your user wants Boost.Thread instead of STL11 thread but only for threading, so your library can be so modular as to allow both options to end users. This is covered in a separate recommendation below.

Examples of libraries which use versioned namespaces and aliasing to bind a namespace locally:

https://boostgsoc13.github.io/boost.afio/

3. Strongly consider trying your library on Microsoft Visual Studio 2015

More than half the libraries reviewed had no support for Microsoft Visual Studio, and only supported GCC and clang. When the authors were asked why, in many cases it was simply assumed that MSVC didn't implement much C++ 11/14 and the authors hadn't attempted MSVC support.

This is in fact untrue. Here is a complete list of C++ 11/14 features which VS2015 does NOT support (everything else you can assume it supports, including a full and complete C++ 14 STL):

Expression SFINAE (there are workarounds. Note the STL "turns on" magic Expression SFINAE support for those parts of the STL requiring it, so any Expression SFINAE done by the STL for you works as expected).
Any serious constexpr use (try "#define constexpr" i.e. disable it completely. Most of the time your code will compile and work. Consider using a BOOST_CONSTEXPR macro thereafter). It is claimed by Microsoft that full C++ 11 constexpr conformance is present, but to be honest in my own code I don't find anything less than C++ 14 constexpr useful in practice.
No two phase lookup. Reordering decls to make the compiler look up in a way which doesn't produce non-conforming outcomes is a straightforward, if annoying, workaround.
MSVC's C99 support is still less than complete, but it's significantly more complete than before.
MSVC's preprocessor is still non-conforming, but it's less broken than it has ever been.
Variable templates.
Non-static data member initialisers for aggregates.

VS2015 is a very highly conforming C++ 11/14 compiler. It meets or exceeds clang 3.3 on every C++ feature, so if your library can be compiled by clang 3.3 then it is highly likely it should compile, without too much work, on VS2015. VS2015 even has some support for C++ 1z (C++ 17) matching about what clang 3.5 provides with the main showstoppers being lack of relaxed constexpr and no variable templates. See http://blogs.msdn.com/b/vcblog/archive/2015/04/29/c-11-14-17-features-in-vs-2015-rc.aspx.

I am not claiming that you won't get surprises when you try getting MSVC to compile your code which you thought was standards compliant. MSVC is not an AST based compiler and uses heuristics to trigger partial AST compilation, and therefore has a unique processing model which exposes your assumptions about what you think or thought is valid C++. This is exactly why it is worthwhile getting your C++ 11/14 library working on MSVC because you will get a better quality, more standards conforming C++ library out of it.

A good example of just how much C++ 14 support VS2015 provides is in Boost.DI. When I first contacted the author about the lack of VS2015 support, he proceeded to port his entirely C++ 14 codebase to VS2015 successfully, though he had to do a bit of refactoring to make the port work. Interestingly because he didn't push constexpr too far in DI, VS2015's constexpr support was enough for DI to work as expected on VS2015 for the most part.

https://krzysztof-jusiak.github.io/di/boost/libs/di/doc/html/

4. Strongly consider using free CI per-commit testing, even if you have a private CI

Despite that Travis provides free of cost per-commit continuous integration testing for Linux and OS X, and that Appveyor provides the same for Microsoft Windows, there were still libraries in those reviewed which made use of neither and furthermore had no per-commit CI testing whatsoever.

I'll be blunt: not having per-commit CI testing is unacceptable in this modern day and age and is an excellent sign of a library author not committed to software quality. Especially when such CI services are free of cost, and it's purely your laziness that you haven't configured them yet.

So first things first, if your C++ 11/14 library is not using any form of per-commit testing yet, go add Travis and Appveyor right support now. Configuration is extremely easy if your project lives on Github, simply login into both services using your github account and enable your project. Next add a suitable .travis.yml and an .appveyor.yml file to the root of your project, and push the commit. Watch Travis and Appveyor build and run your CI suitable unit test suite, and report back on Github (especially if it's a pull request) whether the build and tests passed or failed. From now on when someone issues a pull request fixing a bug, you'll instantly know if that pull request compiles and passes all unit tests on Linux, OS X and Windows, and much more importantly, so will the pull request submitter and they usually will go fix problems themselves so you the maintainer never need find out a pull request is defective on some build target.

Example travis.yml's to clone from:

Example appveyor.yml's to clone from:

Both Travis and Appveyor are excellent for getting an 90% confidence signal that some commit did not break something. For a free service with little configuration effort that's fantastic. However if you want a 98% confidence signal you will need to spend a few months of your life configuring your own Jenkins CI installation, most of that time will be learning how not to configure Jenkins as Jenkins is a black, black art indeed - but again great for being free of cost given the return on investment. Once mastered, Jenkins can do almost anything from per-commit testing to soak testing to input fuzz testing to automating a long list of tasks for you (e.g. diffing and synchronising two forks of a repo for you by bisecting commit histories against unit testing), but it will take many dedicated months to acquire the skills to configure a maintainable and stable Jenkins install.

Should you add Travis and Appveyor CI support if you already are using your own private Jenkins CI?

I think the answer is uncategorically yes. The reasons are these:

Having Travis + Appveyor badges (see https://raw.githubusercontent.com/krzysztof-jusiak/di/cpp14/README.md for example Markdown for badges) on your open source project is a universally recognised signal of attention to quality.
Other free tooling such as Coveralls.io have built in integration for github and travis. Hooking Jenkins into Coveralls isn't hard, but it "just works" with Travis instead and that's a similar pattern with most free tooling which consumes CI results.
Future tooling by Boost which dashboards Boost libraries and/or ranks libraries by a quality score will almost certainly automate on Travis and Appveyor being queryable by their RESTful APIs. In other words, placing your library in Github and adding Travis and Appveyor CI support has the highest chance of working immediately with any future Boost tooling with minimal additional effort by you.

5. Strongly consider per-commit compiling your code with static analysis tools

In Travis and Appveyor it is easy to configure a special build job which uses the clang static analyser on Linux/OS X and the MSVC static analyser on Windows. These perform lengthy additional static AST analysis tests to detect when your code is doing something stupid and the use of these is an excellent sign that the developer cares about code quality. Static analysis is perfectly suited to be run by a CI as it takes an inordinate amount of time to compile your program, so a CI can trundle off and do the lengthy work itself while you get on with other work.

Enabling Microsoft's static analyser is easy, simply add /analyze to the compiler command line. Your compile will take ten times longer and new warnings will appear. Note though that the MSVC static analyser is quite prone to false positives like miscounting array entries consumed. You can suppress those using the standard #pragma warning(disable: XXX) system around the offending code.

Enabling clang's static analyser is slightly harder. You'll need to replace the call of the clang++ tool with the tool set into the CXX environment variable by the scan-build tool. See http://clang-analyzer.llvm.org/scan-build.html. For Boost projects, I found this script to work well:

MYPWD=`pwd`
REPORTS="$MYPWD/clangScanBuildReports" 
rm -rf "$REPORTS"
git submodule update --init --recursive
cd boost-local
/usr/share/clang/scan-build-3.4/scan-build --use-analyzer=/usr/bin/clang-3.4 -o "$REPORTS" ./b2 toolset=gcc-4.7 libs/afio/test -a --test=test_all.cpp --link-test

Note that my b2 has a user-config.jam which resets the compiler used to the value of $CXX from the environment.

scan-build will generate a HTML report of the issues found with a pretty graphical display of the logic followed by the analyser. Jenkins has a plugin which can publish this HTML report for you per build, for other CIs you'll need to copy the generated files onto a website somewhere e.g. committing them to your repo under gh-pages and pushing them back to github.

6. Strongly consider running a per-commit pass of your unit tests under both valgrind and the runtime sanitisers

In Travis it is highly worth adding a special build job which runs your unit tests under:

valgrind memcheck (Linux only): This detects illegal reads and writes, use of uninit values, use of unaddressable memory, illegal/double frees, and memory leaks. This tool is highly recommended, with its only downsides being a severe performance penalty (one can detect if running in valgrind inside your tests and treble timeouts. Look into the RUNNING_ON_VALGRIND macro in valgrind.h which by the way compiles just fine on MSVC too. You can also markup your code with valgrind instrumentation (also compatible with MSVC) and simply leave the instrumentation permanently in your binaries) and the fact it can't test Windows code.

Some will argue that their library is a pure C++ compiler library and does no memory allocation, and therefore running valgrind makes no sense for their library. Ah, but remember that valgrind isn't just testing your code, it is testing the code produced by the compiler. If you are doing cutting edge C++ 14 programming you may trigger code generation bugs in compilers past or future, or bugs in the STL caused by how your code uses it. A valgrind pass on your unit tests will catch bad code generation bugs, and potentially one day save you hours maybe days of frustrating debugging of weird segfaults!

Running your unit tests under valgrind is easy, simply prepend valgrind when calling your (preferably optimised though with debug info) test executable. You may find special compilation options will greatly improve the usefulness of error output, try -fno-omit-frame-pointer -fno-optimize-sibling-calls -fno-inline, though note disabling inlining may hide your bug.

Undefined behaviour sanitiser (GCC and clang only): Turned on using -fsanitize=undefined, this detects when your code does undefined behaviour, and is sufficiently lightweight you should consider shipping release binaries with this permanently turned on along with stack smashing detection if using GCC 4.9 or later (-fstack-protector-strong). I personally have the ubsan always on for all builds of any code of mine capable of accepting untrusted input. At the time of writing, turning on the ubsan will prevent these things happening: use of misaligned pointer or reference, load of bool not 0 nor 1, out of bounds array indexing, bad casting, bad derived cast, bad cast of void* to type, bad or wrong vptr use, use of impossible enum value, divide by zero, bad function pointer call, use of null ptr, use of bytes not in object, exiting a value returning function without a return value, returning null from a function not allowed to return null, illegal shifts, signed integer overflow, reaching unreachable code, negative variable length array use.

As you can see, these tests make buffer overflow ROP chain exploits very hard, and therefore your code much, much harder to exploit from a security perspective. I think any library author whose library can accept untrusted input who doesn't always turn ubsan on is being irresponsible. Let me put this another way, if OpenSSL had been compiled with ubsan enabled in release builds, Heartbleed would have caused a process to fatal exit instead of leaking secrets, and Heartbleed would not have been successful except as a denial of service attack.

Thread sanitiser (GCC and clang only): If your library is capable of threaded use or your unit testing creates threads, you definitely should soak execute your unit tests with the thread sanitiser (-fsanitize=thread) which provides a good quality check of the correct use of the C11/C++11 atomic memory model e.g. are all your atomic acquires matched with atomic releases in the right order? Did you read a memory location which was written concurrently without an acquire-release serialisation lock? Sadly the tool can't detect use of memory fences which substantially reduces your flexibility when writing with atomics, so do bear that in mind.

Some may note I didn't recommend the address sanitiser (GCC and clang only). This is because you need to recompile your STL and libc with the address sanitiser to achieve perfect coverage, plus valgrind detects far more problems, however if valgrind is just far too slow for your testing then employing the address sanitiser can be a useful substitute for valgrind for certain tests only. Note that the address sanitiser is perfect for untrusted input fuzz testing as it is much faster than valgrind, so I recommend the address sanitiser in the next section.

7. Strongly consider a nightly or weekly input fuzz automated test if your library is able to accept untrusted input

If your library can consume any form of serialisation or parameters supplied from a network or file or query, including any regular expressions or any type of string even if you don't process it yourself and hand it off to another library, then you need to be doing input fuzz testing at least weekly. Even with ubsan enabled in release builds (see previous section) and therefore use of untrusted input to subvert your security is hard, one can use missing code path verification logic to cause programs to delete or replace user data or write into secure data without introducing undefined behaviour.

The classic tool for fuzz testing data inputs is American Fuzzy Lop (afl). This is a mature, very well understood tool. You should use it in combination with the runtime sanitisers described above, so ideally with valgrind + ubsan, but if valgrind is too slow then with the address sanitiser + ubsan. You may also wish to consider additionally fuzz testing the parameters of every API in your library, see below for tooling to help with that.

One of the most promising new input fuzz testing tools going into the long term is LLVM's fuzz testing facilities which are summarised at http://llvm.org/docs/LibFuzzer.html as they make use of the clang sanitiser coverage recording facility to additionally find the code paths least covered, plus the tool is very fast compared to afl.

8. Consider making it possible to use an XML outputting unit testing framework, even if not enabled by default

A very noticeable trend in the libraries above is that around half use good old C assert() and static_assert() instead of a unit testing framework.

There are many very good reasons not to use a unit testing framework by default, but there are few good reasons to not be able to use a unit testing framework at all. A big problem for the Boost release managers when your library cannot output XML indicating exactly which tests pass and which fail (including the static ones) is that all they get instead is failure to compile or failure to execute. This forces them to dig into compiler error diagnostics and unit test diagnostics respectively. It also makes what may be a very minor problem easily delegated appear as serious as the most serious possible problem because there is no way to quickly disambiguate without diving into potentially a debugger, so all these are good reasons to support some XML outputting unit testing framework which reports an XML entry one per test for each test case in every test suite in your library.

Let me give you an example with Boost.AFIO which executes about a hundred thousand tests for about 70 test platforms and configurations per commit. I once committed a change and noticed in the test matrix that only statically linked libraries were failing. The cause was immediately obvious to me: I had leaked ABI in a way that the unit tests which deliberately build mismatching versions of AFIO to ensure namespace version changes don't conflict had tripped, and without even having to investigate the error itself I knew to revisit my commit for ABI leakage. For someone less familiar with the library, a quick look into the failing test would have revealed the name of the failing test case and instantly realise it was an ABI leakage problem. This sort of extra information is a big win for anyone trying to get a release out the door.

There are big advantages for unit test stability analysis tooling as well. Jenkins CI can record the unit tests for thousands of builds, and if you have a test that regularly but rarely fails then Jenkins can flag such unstable tests. Atlassian tooling free for open source can display unit test aggregate statistics on a dashboard, and free web service tooling able to do ever more sophisticated statistical analysis which you once had to pay for is becoming ever more common.

Finally, specifically for Boost libraries we have an automated regression testing system which works by various end users uploading XML results generated by Boost.Test to an FTP site where a cron script regularly runs to generate static HTML tables of passing and failing tests. Needless to say, if your library was as useful as possible to that system everybody wins, and your library is not as useful to that system if it uses assert() and even static_assert() because the XML uploaded is a compiler error console log or an assert failure diagnostic instead of a detailed list of which tests passed and which failed.

Hopefully by now I have persuaded you to use an XML outputting unit test framework. If you are a Boost library, the obvious choice is to use Boost.Test. Despite its many problems, being slow to develop against and lack of maintenance in its release branch, Boost.Test is still a very competitive choice, and if you ignore the overly dense documentation and simply lift the pattern from this quick sample you'll be up and running very quickly:

#include "boost/test/unit_test.hpp"  // Note the lack of angle brackets

BOOST_AUTO_TEST_SUITE(all)  // Can actually be any name you like

BOOST_AUTO_TEST_CASE(works/spinlock, "Tests that the spinlock works as intended")  // Note the forward slashes in the test name
{
  boost::spinlock::spinlock<bool> lock;
  BOOST_REQUIRE(lock.try_lock());
  BOOST_REQUIRE(!lock.try_lock());
  lock.unlock();
  
  std::lock_guard<decltype(lock)> h(lock);
  BOOST_REQUIRE(!lock.try_lock());
}
// More BOOST_AUTO_TEST_CASE(), as many as is wanted

BOOST_AUTO_TEST_SUITE_END()

Already those familiar with Boost.Test will notice some unusual choices, but I'll come back to why shortly. For reference there are additional common tests in addition to BOOST_REQUIRE:

BOOST_CHECK(expr): Check if expr is true, continuing the test case anyway if false.
BOOST_CHECK_THROWS(expr): Check if expr throws an exception, continuing the test case anyway if false.
BOOST_CHECK_THROW(expr, type): Check if expr throws an exception of a specific type, continuing the test case anyway if false.
BOOST_CHECK_NO_THROW(expr): Check if expr does not throw an exception, continuing the test case anyway if false.

BOOST_REQUIRE(expr): Check if expr is true, immediately exiting the test case if false.
BOOST_REQUIRE_THROWS(expr): Check if expr throws an exception, immediately exiting the test case if false.
BOOST_REQUIRE_THROW(expr, type): Check if expr throws an exception of a specific type, immediately exiting the test case if false.
BOOST_REQUIRE_NO_THROW(expr): Check if expr does not throw an exception, immediately exiting the test case if false.

BOOST_TEST_MESSAGE(msg): Log a message with the XML output.
BOOST_CHECK_MESSAGE(pred, msg): If pred is false, log a message with the XML output.
BOOST_WARN_MESSAGE(pred, msg): If pred is false, log a warning message with the XML output.
BOOST_FAIL(msg): Immediately exit this test case with a message.

Boost.Test provides an enormous amount of extra stuff (especially in its unstable branch), but for 99% of C++ code the above is all you will ever need as more complex checks can be synthesised from the primitives above. There is also a very specific reason I chose this exact subset of Boost.Test's functionality to suggest using here, because Boost.APIBind's lightweight header only Boost.Test emulation defines just the above subset and usefully does so into a header inside APIBind called "boost/test/unit_test.hpp", so if you include just that header you get compatibility with APIBind and Boost.Test. In other words, by using the pattern just suggested you can:

With a macro switch turn on full fat Boost.Test.
For the default use Boost.APIBind's thin wrap of the CATCH header only unit testing library which I have forked with added thread safety support. CATCH is very convenient to develop against, provides pretty coloured console unit test output and useful diagnostics, and on request on the command line can also output JUnit format XML ready for consumption by almost every unit test XML consuming tool out there. Boost.Test theoretically can be used header only, but you'll find it's very slow, whereas CATCH is always header only and has a minimal effect on compile time. CATCH also comes as a single kitchen sink header file, and APIBind includes a copy for you.
For those so motivated that they really want assert() and nothing more, simply wrap the above macros with calls to assert(). Your single unit test code base can now target up to three separate ways of reporting unit test fails.

Note if CATCH doesn't have enough features and Boost.Test is too flaky, another popular choice with tons of bells and whistles is Google Test.

What are the problems with replacing asserts with a unit test framework?

Asserts are fast and don't synchronise threads. Unit test frameworks almost always must grab a mutex for every single check, even if that check passes, which can profoundly damage the effectiveness of your testing. The obvious workaround is to prepend an if statement of the test before every check, so if(!expr) BOOST_CHECK(expr); but be aware now only failures will be output into XML, and many CI parsers will consider zero XML test results in a test case to be a fatal error (workaround: always do a BOOST_CHECK(true) at the very end of the test).

Getting static_asserts to decay cleanly into a BOOST_CHECK without #ifdef-ing is not always obvious. The obvious beginning is:

#ifndef AM_USING_BOOST_TEST_FOR_STATIC_ASSERTS
#define BOOST_CHECK_MESSAGE(pred, msg) static_assert(pred, msg)
#endif

... and now use BOOST_CHECK_MESSAGE instead of static_assert directly. If your static assert is inside library implementation code, consider a macro which the unit tests override when being built with a unit test framework, but otherwise defaults to static_assert.

Asserts have no effect when NDEBUG is defined. Your test code may assume this for optimised builds, and a simple regex find and replace may not be sufficient.

Libraries implementing XML outputting unit testing with the Boost.Test macro API:

9. Consider breaking up your testing into per-commit CI testing, 24 hour soak testing, and parameter fuzz testing

When a library is small, you can generally get away with running all tests per commit, and as that is easier that is usually what one does.

However as a library grows and matures, you should really start thinking about categorising your tests into quick ones suitable for per-commit testing, long ones suitable for 24 hour soak testing, and parameter fuzz testing whereby an AST analysis tool will try executing your functions with input deliberately designed to exercise unusual code path combinations. The order of these categories generally reflects the maturity of a library, so if a library's API is still undergoing heavy refactoring the second and third categories aren't so cost effective. I haven't mentioned the distinction between http://stackoverflow.com/questions/4904096/whats-the-difference-between-unit-functional-acceptance-and-integration-test unit testing and functional testing and integration testing] here as I personally think that distinction not useful for libraries mostly developed in a person's free time (due to lack of resources, and the fact we all prefer to develop instead of test, one tends to fold unit and functional and integration testing into a single amorphous set of tests which don't strictly delineate as really we should, and instead of proper unit testing one tends to substitute automated parameter fuzz testing, which really isn't the same thing but it does end up covering similar enough ground to make do).

There are two main techniques to categorising tests, and each has substantial pros and cons.

The first technique is that you tag tests in your test suite with keyword tags, so "ci-quick", "ci-slow", "soak-test" and so on. The unit test framework then lets you select at execution time which set of tags you want. This sounds great, but there are two big drawbacks. The first is that each test framework has its own way of doing tags, and these are invariably not compatible so if you have a switchable Boost.Test/CATCH/Google Test generic test code setup then you'll have a problem with the tagging. One nasty but portable workaround I use is magic test naming and then using a regex test selector string, this is why I have categorised slashes in the test names exampled in the section above so I can select tests by category via their name. The second drawback is that you will find that tests often end up calling some generic implementation with different parameters, and you have to go spell out many sets of parameters in individual test cases when one's gut feeling is that those parameters really should be fuzz variables directly controlled by the test runner. Most test frameworks support passing variables into tests from the command line, but again this varies strongly across test frameworks in a way hard to write generic test code.

The second technique is a hack, but a very effective one. One simply parameterises tests with environment variables, and then code calling the unit test program can configure special behaviour by setting environment variables before each test iteration. This technique is especially valuable for converting per-commit tests into soak tests because you simply configure an environment variable which means ITERATIONS to something much larger, and now the same per-commit tests are magically transformed into soak tests. The big drawback here is that just iterating per commit tests a lot more does not a proper soak test suite make, and one can fool oneself into believing your code is highly stable and reliable when it is really only highly stable and reliable at running per commit tests, which obviously it will always be because you run those exact same patterns per commit so those are always the use patterns which will behave the best. Boost.AFIO is 24 hour soak tested on its per-commit tests, and yet I have been more than once surprised at segfaults caused by someone simply doing operations in a different order than the tests did them :(

Regarding parameter fuzz testing, there are a number of tools available for C++, though all are not quick and require lots of sustained CPU time to calculate and execute all possible code path variations. The classic is of course http://ispras.linuxbase.org/index.php/API_Sanity_Autotest, though you'll need their ABI Compliance Checker working properly first which has become much easier for C++ 11 code since they recently added GCC 4.8 support. You should combine this with an executable built with, as a minimum, the address and undefined behaviour sanitisers. I haven't played with this tool yet with Boost.AFIO, though it is very high on my todo list as I have very little unit testing in AFIO (only functional and integration testing), and fuzz testing of my internal routines would be an excellent way of implementing comprehensive exception safety testing which I am also missing (and feel highly unmotivated to implement by hand).

10. Consider not doing compiler feature detection yourself

Something extremely noticeable about nearly all the reviewed C++ 11/14 libraries is that they manually do compiler feature detection in their config.hpp, usually via old fashioned compiler version checking. This tendency is not unsurprising as the number of potential C++ compilers your code usually needs to handle has essentially shrunk to three, and the chances are very high that three compiler will be upper bound going into the long term future. This makes compiler version checking a lot more tractable than say fifteen years ago.

However, C++ 1z is expected to provide a number of feature detection macros via the work of SG-10, and GCC and clang already partially support these, especially in very recent compiler releases. To fill in the gaps in older editions of GCC and clang, and indeed MSVC at all, you might consider making use of the header file at https://github.com/ned14/Boost.APIBind/blob/master/include/cpp_feature.h which provides the following SG-10 feature detection macros on all versions of GCC, clang and MSVC:

cpp_exceptions: Whether C++ exceptions are available
cpp_rtti: Whether C++ RTTI is available

cpp_alias_templates
cpp_alignas
cpp_decltype
cpp_default_function_template_args
cpp_defaulted_functions
cpp_delegated_constructors
cpp_deleted_functions
cpp_explicit_conversions
cpp_generalized_initializers
cpp_implicit_moves
cpp_inheriting_constructors
cpp_inline_namespaces
cpp_lambdas
cpp_local_type_template_args
cpp_noexcept
cpp_nonstatic_member_init
cpp_nullptr
cpp_override_control
cpp_reference_qualified_functions
cpp_range_for
cpp_raw_strings
cpp_rvalue_references
cpp_static_assert
cpp_thread_local
cpp_auto_type
cpp_strong_enums
cpp_trailing_return
cpp_unicode_literals
cpp_unrestricted_unions
cpp_user_defined_literals
cpp_variadic_templates

cpp_contextual_conversions
cpp_decltype_auto
cpp_aggregate_nsdmi
cpp_digit_separators
cpp_init_captures
cpp_generic_lambdas
cpp_relaxed_constexpr
cpp_return_type_deduction
cpp_runtime_arrays
cpp_variable_templates

The advantage of using these SG-10 macros in C++ 11/14 code is threefold:

It should be future proof.
It's a lot nicer than testing compiler versions.
It expands better if a fourth C++ compiler suddenly turned up.

Why use the https://github.com/ned14/Boost.APIBind/blob/master/include/cpp_feature.h header file instead of doing it by hand?

Complete compiler support for GCC, clang and MSVC all versions.
Updates in compiler support will get reflected into cpp_feature.h for you.
You benefit from any extra compilers added automatically.
If you're using Boost.APIBind you automatically get cpp_feature.h included for you as soon as you include any APIBind header file.

Incidentally Boost.APIBind wraps these macros into Boost.Config compatible macros in https://github.com/ned14/Boost.APIBind/blob/master/include/boost/config.hpp which would be included, as with Boost, using "boost/config.hpp".

Consider having Travis send your unit test code coverage results to Coveralls.io

Note: See TracWiki for help on using the wiki.

Download in other formats:

Plain Text