| 189 | | 1. You now need to ship multiple copies of your library, maintain multiple copies of your library, and make sure simultaneous use of multiple library versions in the same executable doesn't conflict. I suspect this cost is worth it for the added flexibility to evolve breaking changes for most library maintainers. You probably want to employ a per-commit run of http://ispras.linuxbase.org/index.php/ABI_compliance_checker to make sure you don't accidentally break the API (or ABI where appropriate) of a specific API version of your library. Also don't forget that git lets you submodule yourself on a different branch, so if you do ship multiple versions you can mount specific branches of yourself on that version's tracking branch within yourself. |
| 190 | | 2. The above technique alone is insufficient for header only end users where multiple versions of your library must coexist within the same translation unit. With some additional extra work, it is possible to allow multiple header only library versions to also coexist in the same translation unit, but this is covered in a separate recommendation below. |
| | 189 | 1. You now need to ship multiple copies of your library, maintain multiple copies of your library, and make sure simultaneous use of multiple library versions in the same executable doesn't conflict. I suspect this cost is worth it for the added flexibility to evolve breaking changes for most library maintainers. You probably want to employ a per-commit run of http://ispras.linuxbase.org/index.php/ABI_compliance_checker to make sure you don't accidentally break the API (or ABI where appropriate) of a specific API version of your library, so in your custom build run you might check out an original SHA for your library separate to your latest commit, build both and use the ABI compliance checker tool to determine if anything has broken. Similarly, the same toolset (ABIDump) could be used to detect where ABIs collide by having some shell script error out if any ABI overlaps between two libraries, perhaps using the diff tool. |
| | 190 | |
| | 191 | Also don't forget that git lets you recursively submodule yourself but pinned to a different branch by [https://git-scm.herokuapp.com/docs/gitmodules adding the `submodule.name.branch` stanza to .gitmodules], so if you do ship multiple versions you can mount specific version tracking branches of yourself within yourself such that a recursive submodule update checks out all the versions of yourself into a given checkout. |
| | 192 | 2. The above technique alone is insufficient for header only end users where multiple versions of your library must coexist within the same translation unit. With some additional extra work using the preprocessor, it is possible to allow multiple header only library versions to also coexist in the same translation unit, but this is covered in a separate recommendation below. |
| 217 | | VS2015 is a very highly conforming C++ 11/14 compiler. It meets or exceeds clang 3.3 on every C++ feature, so if your library can be compiled by clang 3.3 then it is highly likely it should compile, without too much work, on VS2015. VS2015 even has some support for C++ 1z (C++ 17) matching about what clang 3.5 provides with the main showstoppers being lack of relaxed constexpr and no variable templates. See http://blogs.msdn.com/b/vcblog/archive/2015/04/29/c-11-14-17-features-in-vs-2015-rc.aspx. |
| 218 | | |
| 219 | | I am not claiming that you won't get surprises when you try getting MSVC to compile your code which you thought was standards compliant. MSVC is not an AST based compiler and uses heuristics to trigger partial |
| 220 | | |
| 221 | | A good example of just how much C++ 14 support VS2015 provides is in Boost.DI. When I first contacted the author about the lack of VS2015 support, he proceeded to port his entirely C++ 14 codebase to VS2015 successfully, though he had to do a bit of refactoring to make the port work. Interestingly because he didn't push constexpr too far in DI, VS2015's constexpr support was enough for DI to work as expected on VS2015 for the most part. |
| | 219 | VS2015 is a very highly conforming C++ 11/14 compiler. It meets or exceeds clang 3.3 on every C++ 11/14 feature, so if your library can be compiled by clang 3.3 then it is highly likely it should compile, without too much work, on VS2015 assuming the missing items above are not showstoppers. VS2015 even has some support for C++ 1z (C++ 17) matching about what clang 3.5 provides minus C++ 14 relaxed constexpr and C++ 14 variable templates. See http://blogs.msdn.com/b/vcblog/archive/2015/04/29/c-11-14-17-features-in-vs-2015-rc.aspx. |
| | 220 | |
| | 221 | I am not claiming that you won't get surprises when you try getting MSVC to compile your code which you thought was standards compliant. MSVC is not an AST based compiler and uses heuristics to trigger partial local AST compilation, and therefore has a unique processing model which exposes your assumptions about what you think or thought is valid C++. This is exactly why it is worthwhile getting your C++ 11/14 library working on MSVC because you will get a better quality, more standards conforming C++ library out of it. |
| | 222 | |
| | 223 | A good example of just how much C++ 14 support VS2015 provides is in Boost.DI. When I first contacted the author about the lack of VS2015 support, he proceeded to port his entirely C++ 14 codebase to VS2015 successfully, though he had to do a bit of refactoring to make the port work. Interestingly because he didn't push constexpr use past C++ 11 capacity in DI, VS2015's constexpr support was enough for DI to work as expected on VS2015 for the most part. |
| 272 | | Note that my b2 has a user-config.jam which resets the compiler used to the value of $CXX from the environment. |
| 273 | | |
| 274 | | scan-build will generate a HTML report of the issues found with a pretty graphical display of the logic followed by the analyser. Jenkins has a plugin which can publish this HTML report for you per build, for other CIs you'll need to copy the generated files onto a website somewhere e.g. committing them to your repo under gh-pages and pushing them back to github. |
| 275 | | |
| 276 | | == 6. Strongly consider running a per-commit pass of your unit tests under both valgrind and the runtime sanitisers == |
| | 274 | Note that my b2 has a $HOME/user-config.jam which resets the compiler used to the value of $CXX from the environment: |
| | 275 | |
| | 276 | {{{ |
| | 277 | import os ; |
| | 278 | using gcc : : [ os.environ CXX ] ; |
| | 279 | }}} |
| | 280 | |
| | 281 | scan-build will generate a HTML report of the issues found with a pretty graphical display of the logic followed by the analyser into the $REPORTS directory. Jenkins has a plugin which can publish this HTML report for you per build, for other CIs you'll need to copy the generated files onto a website somewhere e.g. committing them to your repo under gh-pages and pushing them back to github. |
| | 282 | |
| | 283 | |
| | 284 | == 6. QUALITY/SAFETY: Strongly consider running a per-commit pass of your unit tests under both valgrind and the runtime sanitisers == |
| 282 | | Some will argue that their library is a pure C++ compiler library and does no memory allocation, and therefore running valgrind makes no sense for their library. Ah, but remember that valgrind isn't just testing your code, ''it is testing the code produced by the compiler''. If you are doing cutting edge C++ 14 programming you may trigger code generation bugs in compilers past or future, or bugs in the STL caused by how your code uses it. A valgrind pass on your unit tests will catch bad code generation bugs, and potentially one day save you hours maybe days of frustrating debugging of weird segfaults! |
| | 290 | Some will argue that their library is a pure constexpr metaprogramming library and does no memory allocation, and therefore running valgrind makes no sense for their library. Ah, but remember that valgrind isn't just testing your code, ''it is testing the code produced by the compiler''. If you are doing cutting edge C++ 14 programming you may trigger code generation bugs in compilers past or future, or bugs in the STL caused by how your code uses it. A valgrind pass on your unit tests will catch bad code generation bugs, and potentially one day save you hours maybe days of frustrating debugging of weird segfaults! |
| 286 | 294 | Undefined behaviour sanitiser (GCC and clang only):: Turned on using `-fsanitize=undefined`, this detects when your code does undefined behaviour, and is sufficiently lightweight you should consider shipping release binaries with this permanently turned on along with stack smashing detection if using GCC 4.9 or later (`-fstack-protector-strong`). I personally have the ubsan always on for all builds of any code of mine capable of accepting untrusted input. At the time of writing, turning on the ubsan will prevent these things happening: use of misaligned pointer or reference, load of bool not 0 nor 1, out of bounds array indexing, bad casting, bad derived cast, bad cast of void* to type, bad or wrong vptr use, use of impossible enum value, divide by zero, bad function pointer call, use of null ptr, use of bytes not in object, exiting a value returning function without a return value, returning null from a function not allowed to return null, illegal shifts, signed integer overflow, reaching unreachable code, negative variable length array use. |
| 287 | 295 | |
| 288 | | As you can see, these tests make buffer overflow ROP chain exploits very hard, and therefore your code much, much harder to exploit from a security perspective. I think any library author whose library can accept untrusted input who doesn't always turn ubsan on is being '''irresponsible'''. Let me put this another way, if OpenSSL had been compiled with ubsan enabled in release builds, [http://en.wikipedia.org/wiki/Heartbleed Heartbleed] would have caused a process to fatal exit instead of leaking secrets, and Heartbleed would not have been successful except as a denial of service attack. |
| 289 | | |
| 290 | | Thread sanitiser (GCC and clang only):: If your library is capable of threaded use or your unit testing creates threads, you definitely should soak execute your unit tests with the thread sanitiser (`-fsanitize=thread`) |
| 291 | | |
| 292 | | Some may note I didn't recommend the address sanitiser (GCC and clang only). This is because you need to recompile your STL and libc with the address sanitiser to achieve perfect coverage, plus valgrind detects far more problems, however if valgrind is just far too slow for your testing then employing the address sanitiser can be a useful substitute for valgrind for certain tests only. Note that the address sanitiser is perfect for untrusted input fuzz testing as it is much faster than valgrind, so I recommend the address sanitiser in the next section. |
| 293 | | |
| 294 | | |
| 295 | | == 7. S |
| 296 | | |
| 297 | | If your library can consume any form of serialisation or parameters supplied from a network or file or query, including any regular expressions or any type of string ''even if you don't process it yourself'' and hand it off to another library, then you '''need''' to be doing input fuzz testing at least weekly. Even with ubsan enabled in release builds (see previous section) and therefore use of untrusted input to subvert your security is hard, one can use missing code path verification logic to cause programs to delete or replace user data or write into secure data without introducing undefined behaviour. |
| | 296 | As you can see, these tests make buffer overflow [https://en.wikipedia.org/wiki/Return-oriented_programming ROP chain exploits] very hard, and therefore your code much, much harder to exploit from a security perspective. I think any library author whose library can accept untrusted input who doesn't always turn ubsan on is being '''irresponsible'''. |
| | 297 | |
| | 298 | Thread sanitiser (GCC and clang only):: If your library is capable of threaded use or your unit testing creates threads, you definitely should soak execute your unit tests with the thread sanitiser (`-fsanitize=thread`) for a few hours per week which provides a good quality check of the correct use of the C11/C++11 atomic memory model e.g. are all your atomic acquires matched with atomic releases in the right order? Did you read a memory location which was written concurrently without an acquire-release serialisation lock? Sadly the tool can't detect use of memory fences which substantially reduces your flexibility when writing with atomics, so do bear that in mind. |
| | 299 | |
| | 300 | Some may note I didn't recommend the address sanitiser (GCC and clang only). This is because you need to recompile your STL and libc with the address sanitiser to achieve perfect coverage, plus valgrind detects far more problems and valgrind detects bad code generated by the compiler and memory corruption by third party libraries. However if valgrind is just far too slow for your testing then employing the address sanitiser can be a useful substitute for valgrind for certain tests only. Note that the address sanitiser is perfect for untrusted input fuzz testing as it is much faster than valgrind, so I recommend the address sanitiser in the next section. |
| | 301 | |
| | 302 | |
| | 303 | == 7. SAFETY: Strongly consider a nightly or weekly input fuzz automated test if your library is able to accept untrusted input == |
| | 304 | |
| | 305 | If your library can consume any form of serialisation or parameters supplied from a network or file or query, including any regular expressions or any type of string ''even if you don't process it yourself'' and hand it off to another library, then you '''need''' to be doing input fuzz testing for a few hours weekly. Even with ubsan enabled in release builds (see previous section) and therefore use of untrusted input to subvert your security is harder, one can use missing code path verification logic to cause programs to delete or replace user data or write into secure data without introducing undefined behaviour. |
| 355 | | Boost.Test provides an enormous amount of extra stuff (especially in its unstable branch), but for 99% of C++ code the above is all you will ever need as more complex checks can be synthesised from the primitives above. There is also a very specific reason I chose this exact subset of Boost.Test's functionality to suggest using here, because [https://github.com/ned14/Boost.APIBind/blob/master/include/boost/test/unit_test.hpp Boost.APIBind]'s lightweight header only Boost.Test emulation defines just the above subset and usefully does so into a header inside APIBind called "boost/test/unit_test.hpp", so if you include just that header you get compatibility with APIBind and Boost.Test. In other words, by using the pattern just suggested you can: |
| | 363 | Boost.Test provides an enormous amount of extra stuff (especially in its unstable branch) for all sorts of advanced testing scenarios, but for most software being developed in a person's free time most of those advanced testing facilities don't provide sufficient benefit for the significant added cost of implementation. Hence, for personally developed open source the above primitive checks, or a combination thereof into more complex solutions, is likely sufficient for 99% of C++ code. There is also a very specific reason I chose this exact subset of Boost.Test's functionality to suggest using here, because [https://github.com/ned14/Boost.APIBind/blob/master/include/boost/test/unit_test.hpp Boost.APIBind]'s lightweight header only Boost.Test emulation defines just the above subset and usefully does so into a header inside APIBind called "boost/test/unit_test.hpp" which is identical to the Boost.Test header path, so if you include just that header you get compatibility with APIBind and Boost.Test. In other words, by using the pattern just suggested you can: |
| 358 | | 2. For the default use Boost.APIBind's thin wrap of [https://github.com/philsquared/Catch the CATCH header only unit testing library] which I have forked with added thread safety support. CATCH is very convenient to develop against, provides pretty coloured console unit test output and useful diagnostics, and on request on the command line can also output JUnit format XML ready for consumption by almost every unit test XML consuming tool out there. Boost.Test theoretically can be used header only, but you'll find it's very slow, whereas CATCH is always header only and has a minimal effect on compile time. CATCH also comes as a single kitchen sink header file, and APIBind includes a copy for you. |
| | 366 | 2. For the default use Boost.APIBind's thin wrap of [https://github.com/philsquared/Catch the CATCH header only unit testing library] which I have forked with added thread safety support. CATCH is very convenient to develop against, provides pretty coloured console unit test output and useful diagnostics, and on request on the command line can also output JUnit format XML ready for consumption by almost every unit test XML consuming tool out there. Boost.Test theoretically can be used header only, but you'll find it's very hard on your compile times, whereas CATCH is always header only and has a minimal effect on compile time. CATCH also comes as a single kitchen sink header file, and APIBind includes a copy for you. |
| 388 | | However as a library grows and matures, you should really start thinking about categorising your tests into quick ones suitable for per-commit testing, long ones suitable for 24 hour soak testing, and parameter fuzz testing whereby an AST analysis tool will try executing your functions with input deliberately designed to exercise unusual code path combinations. The order of these categories generally reflects the maturity of a library, so if a library's API is still undergoing heavy refactoring the second and third categories aren't so cost effective. I haven't mentioned the distinction between http://stackoverflow.com/questions/4904096/whats-the-difference-between-unit-functional-acceptance-and-integration-test unit testing and functional testing and integration testing] here as I personally think that distinction not useful for libraries mostly developed in a person's free time (due to lack of resources, and the fact we all prefer to develop instead of test, one tends to fold unit and functional and integration testing into a single amorphous set of tests which don't strictly delineate as really we should, and instead of proper unit testing one tends to substitute automated parameter fuzz testing, which really isn't the same thing but it does end up covering similar enough ground to make do). |
| | 396 | However as a library grows and matures, you should really start thinking about categorising your tests into quick ones suitable for per-commit testing, long ones suitable for 24 hour soak testing, and parameter fuzz testing whereby a fuzz tool will try executing your functions with input deliberately designed to exercise unusual code path combinations. The order of these categories generally reflects the maturity of a library, so if a library's API is still undergoing heavy refactoring the second and third categories aren't so cost effective. I haven't mentioned the distinction between [http://stackoverflow.com/questions/4904096/whats-the-difference-between-unit-functional-acceptance-and-integration-test unit testing and functional testing and integration testing] here as I personally think that distinction not useful for libraries mostly developed in a person's free time (due to lack of resources, and the fact we all prefer to develop instead of test, one tends to fold unit and functional and integration testing into a single amorphous set of tests which don't strictly delineate as really we should, and instead of proper unit testing one tends to substitute automated parameter fuzz testing, which really isn't the same thing but it does end up covering similar enough ground to make do). |
| 392 | | The first technique is that you tag tests in your test suite with keyword tags, so "ci-quick", "ci-slow", "soak-test" and so on. The unit test framework then lets you select at execution time which set of tags you want. This sounds great, but there are two big drawbacks. The first is that each test framework has its own way of doing tags, and these are invariably not compatible so if you have a switchable Boost.Test/CATCH/Google Test generic test code setup then you'll have a problem with the tagging. One nasty but portable workaround I use is magic test naming and then using a regex test selector string, this is why I have categorised slashes in the test names exampled in the section above so I can select tests by category via their name. The second drawback is that you will find that tests often end up calling some generic implementation with different parameters, and you have to go spell out many sets of parameters in individual test cases when one's gut feeling is that those parameters really should be fuzz variables directly controlled by the test runner. Most test frameworks support passing variables into tests from the command line, but again this varies strongly across test frameworks in a way hard to write generic test code. |
| 393 | | |
| 394 | | The second technique is a hack, but a very effective one. One simply parameterises tests with environment variables, and then code calling the unit test program can configure special behaviour by setting environment variables before each test iteration. This technique is especially valuable for converting per-commit tests into soak tests because you simply configure an environment variable which means ITERATIONS to something much larger, and now the same per-commit tests are magically transformed into soak tests. The big drawback here is that just iterating per commit tests a lot more does not a proper soak test suite make, and one can fool oneself into believing your code is highly stable and reliable when it is really only highly stable and reliable at running per commit tests, which obviously it will always be because you run those exact same patterns per commit so those are always the use patterns which will behave the best. Boost.AFIO is 24 hour soak tested on its per-commit tests, and yet I have been more than once surprised at segfaults caused by someone simply doing operations in a different order than the tests did them :( |
| 395 | | |
| 396 | | Regarding parameter fuzz testing, there are a number of tools available for C++, though all are not quick and require lots of sustained CPU time to calculate and execute all possible code path variations. The classic is of course http://ispras.linuxbase.org/index.php/API_Sanity_Autotest, though you'll need [http://ispras.linuxbase.org/index.php/ABI_Compliance_Checker their ABI Compliance Checker] working properly first which has become much easier for C++ 11 code since they recently added GCC 4.8 support. You should combine this with an executable built with, as a minimum, [http://clang.llvm.org/docs/UsersManual.html#controlling-code-generation the address and undefined behaviour sanitisers]. I haven't played with this tool yet with Boost.AFIO, though it is very high on my todo list as I have very little unit testing in AFIO (only functional and integration testing), and fuzz testing of my internal routines would be an excellent way of implementing comprehensive exception safety testing which I am also missing (and feel highly unmotivated to implement by hand). |
| 397 | | |
| 398 | | |
| 399 | | == 10. |
| 400 | | |
| 401 | | Something extremely noticeable about nearly all the reviewed C++ 11/14 libraries is that they manually do compiler feature detection in their config.hpp, usually via old fashioned compiler version checking. This tendency is not unsurprising as the number of potential C++ compilers your code usually needs to handle has essentially shrunk to three, and the chances are very high that three compiler will be upper bound going into the long term future. This makes compiler version checking a lot more tractable than say fifteen years ago. |
| | 400 | The first technique is that you tag tests in your test suite with keyword tags, so "ci-quick", "ci-slow", "soak-test" and so on. The unit test framework then lets you select at execution time which set of tags you want. This sounds great, but there are two big drawbacks. The first is that each test framework has its own way of doing tags, and these are invariably not compatible so if you have a switchable Boost.Test/CATCH/Google Test generic test code setup then you'll have a problem with the tagging. One nasty but portable workaround I use is to include the tag into the test name and then using a regex test selector string on the command line, this is why I have categorised slashes in the test names exampled in the section above so I can select tests by category via their name. The second drawback is that you will find that tests often end up internally calling some generic implementation with different parameters, and you have to go spell out many sets of parameters in individual test cases when one's gut feeling is that those parameters really should be fuzz variables directly controlled by the test runner. Most test frameworks support passing variables into tests from the command line, but again this varies strongly across test frameworks in a way hard to write generic test code, so you end up hard coding various sets of variables one per test case. |
| | 401 | |
| | 402 | The second technique is a hack, but a very effective one. One simply parameterises tests with environment variables, and then code calling the unit test program can configure special behaviour by setting environment variables before each test iteration. This technique is especially valuable for converting per-commit tests into soak tests because you simply configure an environment variable which means ITERATIONS to something much larger, and now the same per-commit tests are magically transformed into soak tests. Another major use case is to reduce iterations for when you are running under valgrind, or even just a very slow ARM dev board. The big drawback here is the self deception that just iterating per commit tests a lot more does not a proper soak test suite make, and one can fool oneself into believing your code is highly stable and reliable when it is really only highly stable and reliable at running per commit tests, which obviously it will always be because you run those exact same patterns per commit so those are always the use patterns which will behave the best. Boost.AFIO is 24 hour soak tested on its per-commit tests, and yet I have been more than once surprised at segfaults caused by someone simply doing operations in a different order than the tests did them :( |
| | 403 | |
| | 404 | Regarding parameter fuzz testing, there are a number of tools available for C++, some better or more appropriate to your use case than others. The classic is of course http://ispras.linuxbase.org/index.php/API_Sanity_Autotest, though you'll need [http://ispras.linuxbase.org/index.php/ABI_Compliance_Checker their ABI Compliance Checker] working properly first which has become much easier for C++ 11 code since they recently added GCC 4.8 support (note that GCC 4.8 still has incomplete C++ 14 support). You should combine this with an executable built with, as a minimum, [http://clang.llvm.org/docs/UsersManual.html#controlling-code-generation the address and undefined behaviour sanitisers]. I haven't played with this tool yet with Boost.AFIO, though it is very high on my todo list as I have very little unit testing in AFIO (only functional and integration testing), and fuzz testing of my internal routines would be an excellent way of implementing comprehensive exception safety testing which I am also missing (and feel highly unmotivated to implement by hand). |
| | 405 | |
| | 406 | |
| | 407 | == 10. PORTABILITY: Consider not doing compiler feature detection yourself == |
| | 408 | |
| | 409 | Something extremely noticeable about nearly all the reviewed C++ 11/14 libraries is that they manually do compiler feature detection in their config.hpp, usually via old fashioned compiler version checking. This tendency is not unsurprising as the number of potential C++ compilers your code usually needs to handle has essentially shrunk to three unlike the dozen common compilers implementing the 1998 C++ standard, and the chances are very high that three compilers will be upper bound going into the long term future. This makes compiler version checking a lot more tractable than say fifteen years ago. |
| 464 | | Incidentally Boost.APIBind wraps these macros into Boost.Config compatible macros in https://github.com/ned14/Boost.APIBind/blob/master/include/boost/config.hpp which would be included, as with Boost, using "boost/config.hpp". |
| 465 | | |
| 466 | | 11. Consider having Travis send your unit test code coverage results to Coveralls.io |
| 467 | | |
| | 472 | Problems with cpp_feature.h: |
| | 473 | |
| | 474 | 1. No support for detecting STL library feature availability. One can do this somewhat with GCC as it always pairs to a libstdc++ version, and of course one can do this for MSVC. However clang pairs to whatever is the latest STL on the system, plus GCC combined with libc++ is becoming increasingly common on Linux. In short you are on your own for STL library feature detection as I am unaware of any easy way to abstract this without the SG-10 library feature detection facilities built into the compiler. |
| | 475 | |
| | 476 | Incidentally Boost.APIBind wraps these SG-10 feature macros into Boost.Config compatible macros in https://github.com/ned14/Boost.APIBind/blob/master/include/boost/config.hpp which would be included, as with Boost, using "boost/config.hpp". You can therefore if you really want use the Boost feature detection macros instead, even without Boost being present. |
| | 477 | |
| | 478 | |
| | 479 | == 11. QUALITY: Consider having Travis send your unit test code coverage results to Coveralls.io == |
| | 480 | |
| | 481 | == 12. QUALITY: Consider creating a status dashboard for your library with everything you need to know showed in one place == |
| | 482 | |
| | 483 | == 13. DESIGN: Consider enabling multiple versions of your library to coexist within the same translation unit == |
| | 484 | |
| | 485 | == 14. DESIGN: Consider making (more) use of namespace composure as a design pattern == |
| | 486 | |
| | 487 | |
