attachment set

Thu, 07 May 2015 13:03:38 GMT

attachment → future.hpp

Thu, 07 May 2015 13:07:31 GMT

Replying to Konrad Zemek <konrad.zemek@…>:

Basically, the future's destructor waits on lock that was already locked earlier in the callstack:

Just to avoid any miscommunication, I meant a mutex, not a lock. :)

viboes — Thu, 14 May 2015 15:56:48 GMT

Please, could you try the develop branch in order to see if this is already fixed? I have done a lot of changes in it.

owner, status changed

viboes — Thu, 14 May 2015 15:58:15 GMT

owner changed from Anthony Williams to viboes
status new → assigned

Sat, 16 May 2015 18:56:14 GMT

The same deadlock happens in develop (e39703f).

I "solved" it for testing purposes by reordering variable initialization in do_continuation, but it's obviously the wrong way: if the future's destructor happens-before relocker's destructor, then relocker locks an already destroyed mutex:

diff --git a/include/boost/thread/future.hpp b/include/boost/thread/future.hpp
index e2dcb43..16b6270 100644
--- a/include/boost/thread/future.hpp
+++ b/include/boost/thread/future.hpp
@@ -230,9 +230,9 @@ namespace boost
             void do_continuation(boost::unique_lock<boost::mutex>& lock)
             {
                 if (! continuations.empty()) {
+                  relocker rlk(lock);
                   continuations_type the_continuations = continuations;
                   continuations.clear();
-                  relocker rlk(lock);
                   for (continuations_type::iterator it = the_continuations.begin(); it != the_continuations.end(); ++it) {
                     (*it)->launch_continuation();
                   }

Following is the stack trace from the application using Boost.Thread from develop:

#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f9a9553856a in _L_lock_913 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f9a95538390 in __GI___pthread_mutex_lock (mutex=0x7f9a8c000ed0) at ../nptl/pthread_mutex_lock.c:79
#3  0x00000000004bb872 in pthread_mutex_lock (m=0x7f9a8c000ed0) at /usr/include/boost/thread/pthread/mutex.hpp:62
#4  boost::mutex::lock (this=0x7f9a8c000ed0) at /usr/include/boost/thread/pthread/mutex.hpp:116
#5  0x00000000004c4b83 in boost::unique_lock<boost::mutex>::lock (this=0x7f9a93139d30)
    at /usr/include/boost/thread/lock_types.hpp:346
#6  0x00000000004c4ceb in boost::unique_lock<boost::mutex>::unique_lock (this=0x7f9a93139d30, m_=...)
    at /usr/include/boost/thread/lock_types.hpp:124
#7  0x00000000004be583 in boost::detail::shared_state_base::dec (this=0x7f9a8c000e90)
    at /usr/include/boost/thread/future.hpp:197
#8  0x00000000004ce1ae in boost::detail::basic_future<void>::~basic_future (this=0x1144a68,
    __in_chrg=<optimized out>) at /usr/include/boost/thread/future.hpp:1203
#9  0x00000000004c602e in boost::future<void>::~future (this=0x1144a68, __in_chrg=<optimized out>)
    at /usr/include/boost/thread/future.hpp:1556
#10 0x00000000004de3c6 in boost::detail::future_unwrap_shared_state<boost::future<boost::future<void> >, void>::~future_unwrap_shared_state (this=0x1144950, __in_chrg=<optimized out>) at /usr/include/boost/thread/future.hpp:5065
#11 0x00000000004de414 in boost::detail::future_unwrap_shared_state<boost::future<boost::future<void> >, void>::~future_unwrap_shared_state (this=0x1144950, __in_chrg=<optimized out>) at /usr/include/boost/thread/future.hpp:5065
#12 0x00000000004de44c in boost::checked_delete<boost::detail::future_unwrap_shared_state<boost::future<boost::future<void> >, void> > (x=0x1144950) at /usr/include/boost/core/checked_delete.hpp:34
#13 0x00000000004e01e2 in boost::detail::sp_counted_impl_p<boost::detail::future_unwrap_shared_state<boost::future<boost::future<void> >, void> >::dispose (this=0x1144150) at /usr/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
#14 0x00000000004bab3c in boost::detail::sp_counted_base::release (this=0x1144150)
    at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146
#15 0x00000000004babff in boost::detail::shared_count::~shared_count (this=0x7f9a8c000988,
    __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:443
#16 0x00000000004becd6 in boost::shared_ptr<boost::detail::shared_state_base>::~shared_ptr (this=0x7f9a8c000980,
    __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:323
#17 0x00000000004d98fa in std::_Destroy<boost::shared_ptr<boost::detail::shared_state_base> > (
    __pointer=0x7f9a8c000980) at /usr/include/c++/4.9/bits/stl_construct.h:93
#18 0x00000000004d5d29 in std::_Destroy_aux<false>::__destroy<boost::shared_ptr<boost::detail::shared_state_base>*>
    (__first=0x7f9a8c000980, __last=0x7f9a8c000990) at /usr/include/c++/4.9/bits/stl_construct.h:103
#19 0x00000000004d070d in std::_Destroy<boost::shared_ptr<boost::detail::shared_state_base>*> (
    __first=0x7f9a8c000980, __last=0x7f9a8c000990) at /usr/include/c++/4.9/bits/stl_construct.h:126
#20 0x00000000004c8587 in std::_Destroy<boost::shared_ptr<boost::detail::shared_state_base>*, boost::shared_ptr<boost::detail::shared_state_base> > (__first=0x7f9a8c000980, __last=0x7f9a8c000990)
    at /usr/include/c++/4.9/bits/stl_construct.h:151
#21 0x00000000004c255f in std::vector<boost::shared_ptr<boost::detail::shared_state_base>, std::allocator<boost::shared_ptr<boost::detail::shared_state_base> > >::~vector (this=0x7f9a93139f80, __in_chrg=<optimized out>)
    at /usr/include/c++/4.9/bits/stl_vector.h:424
#22 0x00000000004be6f0 in boost::detail::shared_state_base::do_continuation (this=0x7f9a8c000e90, lock=...)
    at /usr/include/boost/thread/future.hpp:238
#23 0x00000000004be860 in boost::detail::shared_state_base::mark_finished_internal (this=0x7f9a8c000e90, lock=...)
    at /usr/include/boost/thread/future.hpp:264
#24 0x00000000004beea5 in boost::detail::shared_state<void>::mark_finished_with_result_internal (
    this=0x7f9a8c000e90, lock=...) at /usr/include/boost/thread/future.hpp:716
#25 0x00000000004e0761 in boost::detail::future_unwrap_shared_state<boost::future<boost::future<void> >, void>::launch_continuation (this=0x7f9a8c000e90) at /usr/include/boost/thread/future.hpp:5096
#26 0x00000000004be6a5 in boost::detail::shared_state_base::do_continuation (this=0x7f9a8c0012e0, lock=...)
    at /usr/include/boost/thread/future.hpp:237
#27 0x00000000004be860 in boost::detail::shared_state_base::mark_finished_internal (this=0x7f9a8c0012e0, lock=...)
    at /usr/include/boost/thread/future.hpp:264
#28 0x00000000004beea5 in boost::detail::shared_state<void>::mark_finished_with_result_internal (
    this=0x7f9a8c0012e0, lock=...) at /usr/include/boost/thread/future.hpp:716
#29 0x00000000004e0761 in boost::detail::future_unwrap_shared_state<boost::future<boost::future<void> >, void>::launch_continuation (this=0x7f9a8c0012e0) at /usr/include/boost/thread/future.hpp:5096
#30 0x00000000004be6a5 in boost::detail::shared_state_base::do_continuation (this=0x7f9a8c000a20, lock=...)
    at /usr/include/boost/thread/future.hpp:237
#31 0x00000000004be785 in boost::detail::shared_state_base::set_continuation_ptr (this=0x7f9a8c000a20,
    continuation=..., lock=...) at /usr/include/boost/thread/future.hpp:251
#32 0x00000000004e0675 in boost::detail::future_unwrap_shared_state<boost::future<boost::future<void> >, void>::launch_continuation (this=0x7f9a8c0012e0) at /usr/include/boost/thread/future.hpp:5087
#33 0x00000000004be6a5 in boost::detail::shared_state_base::do_continuation (this=0x7f9a8c0010d0, lock=...)
    at /usr/include/boost/thread/future.hpp:237
#34 0x00000000004be860 in boost::detail::shared_state_base::mark_finished_internal (this=0x7f9a8c0010d0, lock=...)
    at /usr/include/boost/thread/future.hpp:264
#35 0x00000000004e83ef in boost::detail::shared_state<boost::future<void> >::mark_finished_with_result_internal(boost::future<void>&&, boost::unique_lock<boost::mutex>&) (this=0x7f9a8c0010d0,
    result_=<unknown type in /home/konrad/plgrid/helpers/debug/test/unit/retrier_test, CU 0xa555, DIE 0x37494>,
    lock=...) at /usr/include/boost/thread/future.hpp:525
#36 0x00000000004e8361 in boost::detail::shared_state<boost::future<void> >::mark_finished_with_result(boost::future<void>&&) (this=0x7f9a8c0010d0,
    result_=<unknown type in /home/konrad/plgrid/helpers/debug/test/unit/retrier_test, CU 0xa555, DIE 0x37494>)
    at /usr/include/boost/thread/future.hpp:553
#37 0x00000000004e8280 in _ZN5boost6detail41future_executor_continuation_shared_stateINS_9executors17basic_thread_poolENS_6futureIvEES5_ZN3one13communication6layers7RetrierI10LowerLayerE4sendESsiEUlT_E_E3runENS_10shared_ptrINS0_17shared_state_baseEEE (that_=...) at /usr/include/boost/thread/future.hpp:4284
#38 0x00000000004e80fa in _ZN5boost6detail6run_itINS0_41future_executor_continuation_shared_stateINS_9executors17basic_thread_poolENS_6futureIvEES6_ZN3one13communication6layers7RetrierI10LowerLayerE4sendESsiEUlT_E_EEEclEv (
    this=0x7f9a8c001268) at /usr/include/boost/thread/future.hpp:4256
#39 0x00000000004e7f7c in _ZN5boost6detail16nullary_functionIFvvEE9impl_typeINS0_6run_itINS0_41future_executor_continuation_shared_stateINS_9executors17basic_thread_poolENS_6futureIvEESA_ZN3one13communication6layers7RetrierI10LowerLayerE4sendESsiEUlT_E_EEEEE4callEv (this=0x7f9a8c001260) at /usr/include/boost/thread/detail/nullary_function.hpp:52
#40 0x00000000004bf6af in boost::detail::nullary_function<void ()>::operator()() (this=0x7f9a9313a690)
    at /usr/include/boost/thread/detail/nullary_function.hpp:125
#41 0x00000000004bf70d in boost::executors::basic_thread_pool::worker_thread (this=0x1145ac0)
    at /usr/include/boost/thread/executors/basic_thread_pool.hpp:91
#42 0x00000000004e53d7 in boost::detail::invoke<void (boost::executors::basic_thread_pool::*)(), boost::executors::basic_thread_pool<>*>(void (boost::executors::basic_thread_pool::*&&)(), boost::executors::basic_thread_pool<>*&&, (boost::executors::basic_thread_pool<>*&&)...) (
    f=<unknown type in /home/konrad/plgrid/helpers/debug/test/unit/retrier_test, CU 0xa555, DIE 0x75e98>,
    a0=<unknown type in /home/konrad/plgrid/helpers/debug/test/unit/retrier_test, CU 0xa555, DIE 0x75e9d>)
    at /usr/include/boost/thread/detail/invoke.hpp:77
#43 0x00000000004e3687 in boost::detail::thread_data<void (boost::executors::basic_thread_pool::*)(), boost::executors::basic_thread_pool*>::run2<1ul> (this=0x1145ca0) at /usr/include/boost/thread/detail/thread.hpp:75
#44 0x00000000004e0ad0 in boost::detail::thread_data<void (boost::executors::basic_thread_pool::*)(), boost::executors::basic_thread_pool*>::run (this=0x1145ca0) at /usr/include/boost/thread/detail/thread.hpp:81
#45 0x00007f9a9575cd8a in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
#46 0x00007f9a955360a5 in start_thread (arg=0x7f9a9313d700) at pthread_create.c:309
#47 0x00007f9a94d3dcfd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

viboes — Mon, 01 Jun 2015 17:29:57 GMT

Replying to Konrad Zemek <konrad.zemek@…>:

The same deadlock happens in develop (e39703f).

I "solved" it for testing purposes by reordering variable initialization in do_continuation, but it's obviously the wrong way: if the future's destructor happens-before relocker's destructor, then relocker locks an already destroyed mutex:

I need to check if this situation is possible. Thanks for signaling it.

viboes — Sat, 05 Sep 2015 14:54:23 GMT

The code in develop (or >=boost.1.58) contains now

#if defined BOOST_THREAD_PROVIDES_FUTURE_CONTINUATION
            void do_continuation(boost::unique_lock<boost::mutex>& lock)
            {
                if (! continuations.empty()) {
                  continuations_type the_continuations = continuations;
                  continuations.clear();
                  relocker rlk(lock);
                  for (continuations_type::iterator it = the_continuations.begin(); it != the_continuations.end(); ++it) {
                    (*it)->launch_continuation();
                  }
                }
            }
#else
            void do_continuation(boost::unique_lock<boost::mutex>&)
            {
            }
#endif

Please could you give a try?

viboes — Tue, 08 Sep 2015 17:22:39 GMT

Replying to viboes:

The code in develop (or >=boost.1.58) contains now

#if defined BOOST_THREAD_PROVIDES_FUTURE_CONTINUATION
            void do_continuation(boost::unique_lock<boost::mutex>& lock)
            {
                if (! continuations.empty()) {
                  continuations_type the_continuations = continuations;
                  continuations.clear();
                  relocker rlk(lock);
                  for (continuations_type::iterator it = the_continuations.begin(); it != the_continuations.end(); ++it) {
                    (*it)->launch_continuation();
                  }
                }
            }
#else
            void do_continuation(boost::unique_lock<boost::mutex>&)
            {
            }
#endif

Please could you give a try?

Forget this message :(

viboes — Sun, 15 Nov 2015 00:02:38 GMT

Hi,

we have fixed some issues with in future.hpp (https://github.com/boostorg/thread/commit/60a8cb9b5c69423ee0bea403cacc971606575f01).

Please, could you check with the develop branch?

type changed

viboes — Sat, 05 Mar 2016 09:39:53 GMT

type Bugs → Support Requests

Moved to support until we have some news.

status changed; resolution set; milestone deleted

viboes — Sat, 03 Sep 2016 13:16:42 GMT

status assigned → closed
resolution → worksforme
milestone To Be Determined

I believe this issue has already been fixed.

Please reopen it if it is not the case.

Boost C++ Libraries: Ticket #11275: Deadlock when destroying continuation future

attachment set

owner, status changed

type changed

status changed; resolution set; milestone deleted