| 278 | | == 7. Strongly consider a nightly or weekly input fuzz automated test if your library is able to accept untrusted input (any form of serialisation or parameters supplied from a network or file or query, including any regular expressions or any type of string even if you don't process it yourself) == |
| 279 | | |
| 280 | | |
| 281 | | One of the most promising going into the long term is LLVM's fuzz testing facilities which are summarised at http://llvm.org/docs/LibFuzzer.html as they make excellent use of the clang sanitisers to find the bad code paths, and the tool is very fast. |
| | 278 | In Travis it is highly worth adding a special build job which runs your unit tests under: |
| | 279 | |
| | 280 | valgrind memcheck (Linux only):: This detects illegal reads and writes, use of uninit values, use of unaddressable memory, illegal/double frees, and memory leaks. This tool is '''highly''' recommended, with its only downsides being a severe performance penalty (one can detect if running in valgrind inside your tests and treble timeouts. Look into the `RUNNING_ON_VALGRIND` macro in valgrind.h which by the way compiles just fine on MSVC too. You can also markup your code with valgrind instrumentation (also compatible with MSVC) and simply leave the instrumentation permanently in your binaries) and the fact it can't test Windows code. |
| | 281 | |
| | 282 | Some will argue that their library is a pure C++ compiler library and does no memory allocation, and therefore running valgrind makes no sense for their library. Ah, but remember that valgrind isn't just testing your code, ''it is testing the code produced by the compiler''. If you are doing cutting edge C++ 14 programming you may trigger code generation bugs in compilers past or future, or bugs in the STL caused by how your code uses it. A valgrind pass on your unit tests will catch bad code generation bugs, and potentially one day save you hours maybe days of frustrating debugging of weird segfaults! |
| | 283 | |
| | 284 | Running your unit tests under valgrind is easy, simply prepend valgrind when calling your (preferably optimised though with debug info) test executable. You may find special compilation options will greatly improve the usefulness of error output, try `-fno-omit-frame-pointer -fno-optimize-sibling-calls -fno-inline`, though note disabling inlining may hide your bug. |
| | 285 | |
| | 286 | Undefined behaviour sanitiser (GCC and clang only):: Turned on using `-fsanitize=undefined`, this detects when your code does undefined behaviour, and is sufficiently lightweight you should consider shipping release binaries with this permanently turned on along with stack smashing detection if using GCC 4.9 or later (`-fstack-protector-strong`). I personally have the ubsan always on for all builds of any code of mine capable of accepting untrusted input. At the time of writing, turning on the ubsan will prevent these things happening: use of misaligned pointer or reference, load of bool not 0 nor 1, out of bounds array indexing, bad casting, bad derived cast, bad cast of void* to type, bad or wrong vptr use, use of impossible enum value, divide by zero, bad function pointer call, use of null ptr, use of bytes not in object, exiting a value returning function without a return value, returning null from a function not allowed to return null, illegal shifts, signed integer overflow, reaching unreachable code, negative variable length array use. |
| | 287 | |
| | 288 | As you can see, these tests make buffer overflow ROP chain exploits very hard, and therefore your code much, much harder to exploit from a security perspective. I think any library author whose library can accept untrusted input who doesn't always turn ubsan on is being '''irresponsible'''. Let me put this another way, if OpenSSL had been compiled with ubsan enabled in release builds, [http://en.wikipedia.org/wiki/Heartbleed Heartbleed] would have caused a process to fatal exit instead of leaking secrets, and Heartbleed would not have been successful except as a denial of service attack. |
| | 289 | |
| | 290 | Thread sanitiser (GCC and clang only):: If your library is capable of threaded use or your unit testing creates threads, you definitely should soak execute your unit tests with the thread sanitiser (`-fsanitize=thread`) which provides a good quality check of the correct use of the C11/C++11 atomic memory model e.g. are all your atomic acquires matched with atomic releases in the right order? Did you read a memory location which was written concurrently without an acquire-release serialisation lock? Sadly the tool can't detect use of memory fences which substantially reduces your flexibility when writing with atomics, so do bear that in mind. |
| | 291 | |
| | 292 | Some may note I didn't recommend the address sanitiser (GCC and clang only). This is because you need to recompile your STL and libc with the address sanitiser to achieve perfect coverage, plus valgrind detects far more problems, however if valgrind is just far too slow for your testing then employing the address sanitiser can be a useful substitute for valgrind for certain tests only. Note that the address sanitiser is perfect for untrusted input fuzz testing as it is much faster than valgrind, so I recommend the address sanitiser in the next section. |
| | 293 | |
| | 294 | |
| | 295 | == 7. Strongly consider a nightly or weekly input fuzz automated test if your library is able to accept untrusted input == |
| | 296 | |
| | 297 | If your library can consume any form of serialisation or parameters supplied from a network or file or query, including any regular expressions or any type of string ''even if you don't process it yourself'' and hand it off to another library, then you '''need''' to be doing input fuzz testing at least weekly. Even with ubsan enabled in release builds (see previous section) and therefore use of untrusted input to subvert your security is hard, one can use missing code path verification logic to cause programs to delete or replace user data or write into secure data without introducing undefined behaviour. |
| | 298 | |
| | 299 | The classic tool for fuzz testing data inputs is [http://lcamtuf.coredump.cx/afl/ American Fuzzy Lop (afl)]. This is a mature, very well understood tool. You should use it in combination with the runtime sanitisers described above, so ideally with valgrind + ubsan, but if valgrind is too slow then with the address sanitiser + ubsan. You may also wish to consider additionally fuzz testing the parameters of every API in your library, see below for tooling to help with that. |
| | 300 | |
| | 301 | One of the most promising new input fuzz testing tools going into the long term is LLVM's fuzz testing facilities which are summarised at http://llvm.org/docs/LibFuzzer.html as they make use of the clang sanitiser coverage recording facility to additionally find the code paths least covered, plus the tool is very fast compared to afl. |
