Interprocess mutex: race condition crash (when using compiler optimization)

[Robert Ayrapetyan <robert.ayrapetyan@…>]

Environment:

OS: FreeBSD 11.0-RELEASE-p2 #0: Mon Oct 24 06:55:27 UTC 2016     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
Boost: boost-libs-1.63.0_1
Clang: FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)

Description: program creates few named maps (boost::interprocess::map) in a shared memory (one map of custom structs and one map of sets (boost::interprocess::set)). Then multiple processes start to add elements into both maps concurrently. Each element has "time to live" value and "purge_expired" function is being called on each insertion. Named inter-process mutexes are being used for synchronizing access to each map.

Steps to reproduce:

1. Unpack attached archive to some folder (e.g. /tmp/testzero)
2. mkdir /tmp/testzero/build; cd /tmp/testzero/build; cmake ..; make
3. Run testzero (note: requires at least 4GB or RAM)

Expected: program should pass all cycles and exit successfully
Actual: program crashes after several seconds of execution (depends on CPU) When additional logging is enabled, we can often see two different processes pass "mutex lock" and changing same map simultaneously which leads to an exception.

The issue is reproducible only when flags -O2 or -O3 are being used otherwise code works as expected.

A stack dump:

#0  boost::intrusive::rbtree_algorithms<boost::intrusive::rbtree_node_traits<boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, true> >::rebalance_after_insertion (
    header=<value optimized out>, p=<value optimized out>) at offset_ptr.hpp:728
728	      n.priv_offset() = tmp;
[New Thread 801a16000 (LWP 100862/<unknown>)]
(gdb) bt
#0  boost::intrusive::rbtree_algorithms<boost::intrusive::rbtree_node_traits<boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, true> >::rebalance_after_insertion (
    header=<value optimized out>, p=<value optimized out>) at offset_ptr.hpp:728
#1  0x0000000000404c62 in boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, 0ul>::priv_deallocate (
    this=<value optimized out>, addr=<value optimized out>) at rbtree_algorithms.hpp:388
#2  0x00000000004045ad in boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, 0ul>::deallocate (
    this=0x801c00010, addr=0x8028e32e1) at rbtree_best_fit.hpp:1321
#3  0x00000000004044df in boost::container::container_detail::private_node_pool_impl<boost::interprocess::segment_manager_base<boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, 0ul> > >::purge_blocks (this=0x8028e31f0) at segment_manager.hpp:227
#4  0x00000000004117b9 in boost::container::container_detail::node_alloc_holder<boost::interprocess::private_node_allocator<unsigned long, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, 0ul>, boost::interprocess::iset_index>, 64ul>, boost::intrusive::rbtree_impl<boost::intrusive::bhtraits<boost::container::container_detail::tree_node<unsigned long, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, (boost::container::tree_type_enum)0, true>, boost::intrusive::rbtree_node_traits<boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, true>, (boost::intrusive::link_mode_type)0, boost::intrusive::dft_tag, 3u>, void, boost::container::value_to_node_compare<boost::container::container_detail::tree_node<unsigned long, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, (boost::container::tree_type_enum)0, true>, boost::intrusive::tree_value_compare<boost::interprocess::offset_ptr<unsigned long, long, unsigned long, 0ul>, std::__1::less<unsigned long>, boost::move_detail::identity<unsigned long>, true> >, unsigned long, true, void> >::~node_alloc_holder (this=0x8028e31f0) at node_pool_impl.hpp:88
#5  0x000000000041190f in boost::intrusive::bstree_impl<boost::intrusive::bhtraits<boost::container::container_detail::tree_node<std::__1::pair<unsigned long const, shared_memory::MappedValType<boost::container::set<unsigned long, std::__1::less<unsigned long>, boost::interprocess::private_node_allocator<unsigned long, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, 0ul>, boost::interprocess::iset_index>, 64ul>, boost::container::tree_opt<(boost::container::tree_type_enum)0, true> > > >, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, (boost::container::tree_type_enum)0, true>, boost::intrusive::rbtree_node_traits<boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, true>, (boost::intrusive::link_mode_type)0, boost::intrusive::dft_tag, 3u>, void, boost::container::value_to_node_compare<boost::container::container_detail::tree_node<std::__1::pair<unsigned long const, shared_memory::MappedValType<boost::container::set<unsigned long, std::__1::less<unsigned long>, boost::interprocess::private_node_allocator<unsigned long, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, 0ul>, boost::interprocess::iset_index>, 64ul>, boost::container::tree_opt<(boost::container::tree_type_enum)0, true> > > >, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, (boost::container::tree_type_enum)0, true>, boost::intrusive::tree_value_compare<boost::interprocess::offset_ptr<std::__1::pair<unsigned long const, shared_memory::MappedValType<boost::container::set<unsigned long, std::__1::less<unsigned long>, boost::interprocess::private_node_allocator<unsigned long, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, 0ul>, boost::interprocess::iset_index>, 64ul>, boost::container::tree_opt<(boost::container::tree_type_enum)0, true> > > >, long, unsigned long, 0ul>, std::__1::less<unsigned long>, boost::container::container_detail::select1st<unsigned long>, false> >, unsigned long, true, (boost::intrusive::algo_types)5, void>::erase_and_dispose<boost::container::container_detail::allocator_destroyer<boost::interprocess::allocator<boost::container::container_detail::tree_node<std::__1::pair<unsigned long const, shared_memory::MappedValType<boost::container::set<unsigned long, std::__1::less<unsigned long>, boost::interprocess::private_node_allocator<unsigned long, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, 0ul>, boost::interprocess::iset_index>, 64ul>, boost::container::tree_opt<(boost::container::tree_type_enum)0, true> > > >, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, (boost::container::tree_type_enum)0, true>, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, 0ul>, boost::interprocess::iset_index> > > > (this=<value optimized out>, i=<value optimized out>)
    at /tmp/testzero/main.cpp:64
#6  0x000000000040ee97 in shared_memory::Map<unsigned long, boost::container::set<unsigned long, std::__1::less<unsigned long>, boost::interprocess::private_node_allocator<unsigned long, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, 0ul>, boost::interprocess::iset_index>, 64ul>, boost::container::tree_opt<(boost::container::tree_type_enum)0, true> > >::purge_expired (this=0x7fffffffea18, it=<value optimized out>, count=2)
    at tree.hpp:1101
#7  0x0000000000403a0d in shared_memory::MapSet<unsigned long, unsigned long>::add (this=0x7fffffffea18, k=@0x7fffffffe8c0, val=@0x7fffffffe8b8, expires=<value optimized out>)
    at /tmp/testzero/main.cpp:197
#8  0x0000000000402a4f in upd1 (map_foo=<value optimized out>, map_of_sets=@0x7fffffffea18, fstruct=@0x7fffffffe910) at /tmp/testzero/main.cpp:250
#9  0x0000000000402f0e in main (argc=<value optimized out>, argv=<value optimized out>) at /tmp/testzero/main.cpp:298

[robert.ayrapetyan@…]

Cannot reproduce in latest FreeBSD (11.1-RELEASE-p10) with clang 4.0 and boost 1.66. Please close this issue.

[Ion Gaztañaga]

Thanks for the report for both boost 1.63 and 1.66, closing.