Boost C++ Libraries: Ticket #11562: (condition_variable_any::wait_until + recursive_mutex + steady_clock) timer expires after computer time is set forward on Ubuntu 64-bit

severity changed

anonymous — Fri, 21 Aug 2015 08:01:50 GMT

severity Problem → Showstopper

status, component changed

viboes — Thu, 03 Sep 2015 22:16:04 GMT

status new → assigned
component chrono → thread

I've a patch that uses pthread_condattr_setclock. I have no platform providing it (I'm using MacOS). The user will need to define BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC until I find a way to detect it.

Added to develop branch https://github.com/boostorg/thread/commit/9f883f6ad7377b88b79fa70cc5de68cbfb0213e4

Can you test it?

status changed

viboes — Sat, 05 Sep 2015 14:21:30 GMT

status assigned → new

boriss@… — Tue, 08 Sep 2015 09:50:52 GMT

We have tested and wait_until is now working OK. We have found out that sleep function is NOT working anymore after this fix:

boost::this_thread::sleep(boost::posix_time::milliseconds(timeout));

It seems that it stays in the endless loop.

Please add this changes also to the boost::condition not only to the boost::condition_variable

boriss@… — Tue, 08 Sep 2015 10:56:35 GMT

Function sleep is deprecated from 1.56 version. But sleep_for also hangs after your fix:

boost::this_thread::sleep_for(boost::chrono::milliseconds(timeout));

boriss@… — Tue, 08 Sep 2015 11:09:56 GMT

It hangs also if system time is not changed

viboes — Tue, 08 Sep 2015 17:18:28 GMT

I've added a patch that uses pthread_condattr_setclock. I have no platform providing it (I'm using MacOS). The user will need to define BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC until I find a way to detect it.

Add to develop branch https://github.com/boostorg/thread/commit/9f883f6ad7377b88b79fa70cc5de68cbfb0213e4

Can you test it?

status changed

viboes — Tue, 08 Sep 2015 17:18:43 GMT

status new → assigned

boriss@… — Wed, 09 Sep 2015 10:55:14 GMT

Like i described above, we have compiled boost 1.59.0. We did the testing and found out that wait_until is NOT working but sleep/sleep_for is working OK. Then we have add your fix (we have replaced condition_variable_fwd.hpp) and compile boost again. After that wait_until is working OK , but sleep/sleep_for is NOT working anymore. I seems that it stays in endless loop. We have tested also with boost version 1.55.0 and it is the same behavior.

viboes — Thu, 17 Sep 2015 17:55:19 GMT

Have you added the define BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC?

viboes — Thu, 17 Sep 2015 18:02:56 GMT

Replying to viboes:

Have you added the define BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC?

Curiously the patch works for another tester #11377.

boriss@… — Fri, 18 Sep 2015 05:54:34 GMT

We have added the define BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC. But after inserting the patch sleep/sleep_for is NOT working anymore. I seems that it stays in endless loop.

anonymous — Fri, 18 Sep 2015 06:00:11 GMT

Replying to boriss@…:

We have added the define BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC. But after inserting the patch sleep/sleep_for is NOT working anymore. I seems that it stays in endless loop.

Will this changes also be added to the boost::condition?

viboes — Fri, 18 Sep 2015 21:46:16 GMT

Replying to anonymous:

Replying to boriss@…:

We have added the define BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC. But after inserting the patch sleep/sleep_for is NOT working anymore. I seems that it stays in endless loop.

Will this changes also be added to the boost::condition?

Do you mean boost::condition_variable_any? if yes, the answer is yes if it works.

anonymous — Mon, 21 Sep 2015 05:51:42 GMT

Replying to viboes:

Replying to anonymous:

Replying to boriss@…:

We have added the define BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC. But after inserting the patch sleep/sleep_for is NOT working anymore. I seems that it stays in endless loop.

Will this changes also be added to the boost::condition?

Do you mean boost::condition_variable_any? if yes, the answer is yes if it works.

Yes i mean boost::condition_variable_any. But please check sleep and sleep_for functions which are not working after your patch. Tester #11377 was testing only sleep_until function.

viboes — Wed, 23 Sep 2015 22:19:49 GMT

There are a lot of issues with this time issue. If

boost::chrono::time_point<boost::chrono::steady_clock> untilTime(boost::chrono::steady_clock::now() + boost::chrono::milliseconds(dueTime));
boost::condition_variable  m_condition;
(void)m_condition.wait_until(guard, untilTime);

works now I will propose to close this issue. I will not fix the sleep issue as deprecated. Please create an issue for sleep_for if it doesn't exist yet.

I believe that there is already one #7665.

viboes — Wed, 23 Sep 2015 23:00:48 GMT

Please could you try to replace in thread/v2/thread.hpp line #94

#ifdef BOOST_THREAD_SLEEP_FOR_IS_STEADY

#if defined BOOST_THREAD_SLEEP_FOR_IS_STEADY && ! defined BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC

boriss@… — Thu, 24 Sep 2015 10:46:50 GMT

Replying to viboes:

Please could you try to replace in thread/v2/thread.hpp line #94
#ifdef BOOST_THREAD_SLEEP_FOR_IS_STEADY
by
#if defined BOOST_THREAD_SLEEP_FOR_IS_STEADY && ! defined BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC

After this change also sleep_for is working OK. Please add changes from boost::condition_variable also to boost::condition_variable_any

Can we expect the patch for this or will be in the next boost release? It is already known when 1.60 version will be released?

viboes — Thu, 24 Sep 2015 16:15:24 GMT

Replying to boriss@…:

Replying to viboes:
Please could you try to replace in thread/v2/thread.hpp line #94
#ifdef BOOST_THREAD_SLEEP_FOR_IS_STEADY
by
#if defined BOOST_THREAD_SLEEP_FOR_IS_STEADY && ! defined BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC
After this change also sleep_for is working OK. Please add changes from boost::condition_variable also to boost::condition_variable_any

Can we expect the patch for this or will be in the next boost release? It is already known when 1.60 version will be released?

Now that we know that it works for sleep_for also, I will start the adaptation of boost::condition_variable_any.

Thanks for your test and patience.

milestone changed

viboes — Fri, 25 Sep 2015 05:37:52 GMT

milestone To Be Determined → Boost 1.60.0

Please, could you try the develop branch

https://github.com/boostorg/thread/commit/8f5de1d7c34ff7248261874f273498a622bf76d4

status changed; resolution set

viboes — Sun, 27 Sep 2015 13:42:34 GMT

status assigned → closed
resolution → fixed

https://github.com/boostorg/thread/commit/730cb550e6d969520c3f39e0ddf5d80351bddf3a

boriss@… — Tue, 13 Oct 2015 06:47:53 GMT

We are using 1.59 release sources. After adding changes from boost::condition_variable_any (file condition_variable.hpp) we get an Segmentation fault error:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe5bad700 (LWP 3978)]
0x00007ffff75e859a in boost::chrono::detail::duration_cast<boost::chrono::duration<long, boost::ratio<1l, 1000000000l> >, boost::chrono::duration<long, boost::ratio<1l, 1000000000l> > >::operator() (this=0x0, fd=...)
    at /usr/local/include/boost/chrono/duration.hpp:313
313	        BOOST_CONSTEXPR ToDuration operator()(const FromDuration& fd) const

Using gdb debugger we got in backtrace all the time the following print out:

at /usr/local/include/boost/thread/pthread/condition_variable.hpp:311
#1460 0x00007ffff75e0731 in boost::condition_variable_any::wait_until<boost::unique_lock<boost::recursive_mutex>, boost::chrono::duration<long, boost::ratio<1l, 1000000000l> > > (this=0x91b6d0, lock=..., t=...)

After updating also latest boost::chrono from development branch we got the same error.

viboes — Tue, 13 Oct 2015 07:24:44 GMT

Please, is it possible for you to try with the develop branch?

status changed; resolution deleted

viboes — Tue, 13 Oct 2015 07:25:05 GMT

status closed → reopened
resolution fixed

boriss@… — Tue, 13 Oct 2015 07:30:45 GMT

We have cloned develop branch with:

git clone --recursive https://github.com/boostorg/boost.git

and after that we got some errors during compiling of boost.

Can you give me instructions how to checkout/clone on Ubuntu Linux develop branch and how to compile?

viboes — Sat, 17 Oct 2015 06:30:52 GMT

Maybe this can helps https://svn.boost.org/trac/boost/wiki/TryModBoost

Please, can you send me what error are you getting?

boriss@… — Tue, 20 Oct 2015 10:09:10 GMT

We have managed to compile develop branch and we get the same segmentation error like described above. It seems that the cause is in condition_variable_any (file condition_variable.hpp).

viboes — Tue, 20 Oct 2015 15:55:44 GMT

Please, could you tell me with which program are you having this crash? is a Boost.Thread test?

Could you show the part related to

boost::condition_variable_any::wait_until<boost::unique_lock<boost::recursive_mutex>, boost::chrono::duration<long, boost::ratio<1l, 1000000000l> > >

I see that there is a boost::recursive_mutex. Could you tell me if you can change it to a mutex?

boriss@… — Wed, 21 Oct 2015 06:15:53 GMT

We are testing with our application that we are developing. We cannot change to a mutex since we need recursive_mutex.

boriss@… — Wed, 21 Oct 2015 08:40:42 GMT

Our code is following:

boost::recursive_mutex m_CsQueuedItems;
boost::recursive_mutex::scoped_lock _lock1(m_CsQueuedItems);
boost::condition_variable_any m_EvQueuedItems;
boost::chrono::time_point<boost::chrono::steady_clock> untilTime(boost::chrono::steady_clock::now() + boost::chrono::milliseconds(500));
m_EvQueuedItems.wait_until(_lock1, untilTime);

We are calling wait_until from the new thread.

If we use mutex instead of recursive_mutex then we don't get segmentation fault. But like explained above we need recursive mutex. Hope this helps.

viboes — Wed, 21 Oct 2015 22:28:59 GMT

I will do my best, trying to reproduce the issue on my PC. Have you reached to isolate the bug in a simple example?

milestone changed

viboes — Wed, 21 Oct 2015 22:29:07 GMT

milestone Boost 1.60.0 → To Be Determined

viboes — Wed, 21 Oct 2015 22:42:02 GMT

I don't reach to reproduce myself the issue. I need to know if the issue is with condition_variable_any or with recursive_mutex.

Please could you try to isolate the bug in a simple example?

summary changed

viboes — Wed, 21 Oct 2015 22:46:07 GMT

summary Timer (using steady_clock) expires after computer time is set forward on Ubuntu 64-bit → (condition_variable_any::wait_until + recursive_mutex + steady_clock) timer expires after computer time is set forward on Ubuntu 64-bit

viboes — Wed, 21 Oct 2015 22:48:16 GMT

Oh I missed that you have already tested with mutex and it works.

viboes — Wed, 21 Oct 2015 22:52:05 GMT

Could you check if BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC is defined in your platform?

viboes — Wed, 21 Oct 2015 23:14:42 GMT

Humm, please, could you take a look at http://stackoverflow.com/questions/11752155/behavior-of-condition-variable-any-when-used-with-a-recursive-mutex.

Is this related to your issue?

Another pointer http://lists.boost.org/threads-devel/2010/09/0594.php

boriss@… — Thu, 22 Oct 2015 05:42:30 GMT

We didn't changed our code. And before your latest changes in file condition_variable.hpp we didn't have this problem.

boriss@… — Fri, 23 Oct 2015 11:03:33 GMT

Replying to boriss@…:

We didn't changed our code. And before your latest changes in file condition_variable.hpp we didn't have this problem.

We have defined BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC. And it works if we use changes only from condition_variable

viboes — Sun, 25 Oct 2015 23:41:44 GMT

Replying to boriss@…:

We didn't changed our code. And before your latest changes in file condition_variable.hpp we didn't have this problem.

This is not enough, If you code was not well formed. Can you check if you are in this case a linked?

viboes — Sun, 25 Oct 2015 23:42:10 GMT

Replying to boriss@…:

Replying to boriss@…:

We didn't changed our code. And before your latest changes in file condition_variable.hpp we didn't have this problem.

We have defined BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC. And it works if we use changes only from condition_variable

Does it means that I can close the ticket?

status changed; resolution set

viboes — Sun, 15 Nov 2015 00:05:32 GMT

status reopened → closed
resolution → fixed

milestone changed

viboes — Sun, 15 Nov 2015 00:05:41 GMT

milestone To Be Determined → Boost 1.60.0

status changed; resolution deleted

boriss@… — Fri, 20 Nov 2015 08:13:18 GMT

status closed → reopened
resolution fixed

Sorry, i missed your latest post. I believe that our code is well formed. We used/tested our code with different boost versions and we didn't have this problems. We get segmentation fault only after including your changes from file condition_variable.hpp

status changed; resolution set

viboes — Tue, 08 Dec 2015 21:34:20 GMT

status reopened → closed
resolution → fixed

status changed; resolution deleted

boriss@… — Wed, 09 Dec 2015 12:57:11 GMT

status closed → reopened
resolution fixed

We have prepared small example where we have problem with condition_variable_any: .h file:

class ThreadsTest:public CppUnit::TestFixture
{
  CPPUNIT_TEST_SUITE(ThreadsTest);
  CPPUNIT_TEST(testThreads);
  CPPUNIT_TEST_SUITE_END();
public:
  void setUp();
  void tearDown();
  void testThreads();
  void runThread();
  boost::recursive_mutex m_CsQueuedItems;
  boost::condition_variable_any m_EvQueuedItems;
  bool mIsInterrupted;
};

.cpp file:

void ThreadsTest::runThread()
{
  while (!mIsInterrupted)
  {
    {
      boost::recursive_mutex::scoped_lock _lock1(m_CsQueuedItems);
      boost::chrono::time_point<boost::chrono::steady_clock> untilTime(boost::chrono::steady_clock::now() + boost::chrono::milliseconds(500));
      m_EvQueuedItems.wait_until(_lock1, untilTime);
    }
  }
}
void ThreadsTest::testThreads()
{
  mIsInterrupted=false;
  {
    boost::recursive_mutex::scoped_lock _lock1(m_CsQueuedItems);
    boost::thread newThread(boost::bind(&ThreadsTest::runThread, this));
  }
  ThreadBase::Sleep(1000);
  mIsInterrupted=true;
  ThreadBase::Sleep(1000);
}

We used boost1.60.0 beta1, rc4 with added definition BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC.

We got segmentation fault error. testThreads() function is part of unit test and is called first.

Can you please check the example and let us know if you see any problem with this code. If there is no problem with our example, please try to fix the problem in boost library.

viboes — Wed, 09 Dec 2015 17:17:22 GMT

Do you use version 2, 3, or 4 (BOOST_THREAD_VERSION)?

viboes — Wed, 09 Dec 2015 18:03:44 GMT

Could you post a complete example with the back trace of the segmentation fault?

boriss@… — Thu, 10 Dec 2015 07:20:15 GMT

Replying to viboes:

Do you use version 2, 3, or 4 (BOOST_THREAD_VERSION)?

We use version 2

attachment set

boriss@… — Thu, 10 Dec 2015 07:59:02 GMT

attachment → ThreadsTest.cpp

attachment set

boriss@… — Thu, 10 Dec 2015 07:59:43 GMT

attachment → gdb.bt

boriss@… — Thu, 10 Dec 2015 08:03:20 GMT

Replying to viboes:

Could you post a complete example with the back trace of the segmentation fault?

Please find attached example (ThreadsTest.cpp) and back trace (gdb.bt)

viboes — Sun, 28 Feb 2016 18:36:49 GMT

Replying to boriss@…:

Replying to viboes:

Do you use version 2, 3, or 4 (BOOST_THREAD_VERSION)?

We use version 2

Sorry, I missed your reply.

Could you use version 4?

viboes — Sun, 28 Feb 2016 19:02:09 GMT

Replying to boriss@…:

Replying to viboes:

Could you post a complete example with the back trace of the segmentation fault?

Please find attached example (ThreadsTest.cpp) and back trace (gdb.bt)

The backtrace is pointing to lines that let me think that BOOST_THREAD_HAS_CONDATTR_SET_CLOCK_MONOTONIC is not defined.

#4  0x00000000004339de in boost::condition_variable_any::wait_until<boost::unique_lock<boost::recursive_mutex>, boost::chrono::duration<long, boost::ratio<1l, 1000000000l> > > (this=0x7fffffffe108, lock=..., t=...) at /usr/local/include/boost/thread/pthread/condition_variable.hpp:311

Am I missing something?

Anyway, the bt indicates a loop on line 311. The code in line #311 is

        template <class lock_type, class Duration>
        cv_status
        wait_until(
            lock_type& lock,
            const chrono::time_point<chrono::steady_clock, Duration>& t)
        {
            using namespace chrono;
            typedef time_point<steady_clock, nanoseconds> nano_sys_tmpt;
            wait_until(lock,
                        nano_sys_tmpt(ceil<nanoseconds>(t.time_since_epoch())));
            return steady_clock::now() < t ? cv_status::no_timeout :
                                             cv_status::timeout;
        }

The problem is in line #339

        inline cv_status wait_until(
            unique_lock<mutex>& lk,
            chrono::time_point<chrono::steady_clock, chrono::nanoseconds> tp)
        {
            using namespace chrono;
            nanoseconds d = tp.time_since_epoch();
            timespec ts = boost::detail::to_timespec(d);
            if (do_wait_until(lk, ts)) return cv_status::no_timeout;
            else return cv_status::timeout;
        }

Instead of using

            lock_type& lock,

uses

            unique_lock<mutex>& lk,

Hrr. I will fix this very quickly.

milestone changed

viboes — Sun, 28 Feb 2016 22:52:36 GMT

milestone Boost 1.60.0 → Boost 1.61.0

https://github.com/boostorg/thread/commit/4fb88b29fa19abe3d495ac95b57c21f7786a5e0d

status changed; resolution set

viboes — Sat, 05 Mar 2016 09:24:12 GMT

status reopened → closed
resolution → fixed

status changed; resolution deleted

boriss@… — Wed, 16 Mar 2016 10:36:31 GMT

status closed → reopened
resolution fixed

With Boost 1.60 + above mentioned patch we get compile error durring compilaton of our application:

/usr/local/include/boost/thread/pthread/condition_variable.hpp: In destructor ‘boost::thread_cv_detail::lock_on_exit<MutexType>::~lock_on_exit() [with MutexType = int]’:
/usr/local/include/boost/thread/pthread/condition_variable.hpp:400:57:   instantiated from ‘bool boost::condition_variable_any::do_wait_until(lock_type&, const timespec&) [with lock_type = int]’
/usr/local/include/boost/thread/pthread/condition_variable.hpp:346:37:   instantiated from here
/usr/local/include/boost/thread/pthread/condition_variable.hpp:52:21: error: request for member ‘lock’ in ‘*((boost::thread_cv_detail::lock_on_exit<int>*)this)->boost::thread_cv_detail::lock_on_exit<int>::m’, which is of non-class type ‘int’
/usr/local/include/boost/thread/pthread/condition_variable.hpp: In member function ‘void boost::thread_cv_detail::lock_on_exit<MutexType>::activate(MutexType&) [with MutexType = int]’:
/usr/local/include/boost/thread/pthread/condition_variable.hpp:406:15:   instantiated from ‘bool boost::condition_variable_any::do_wait_until(lock_type&, const timespec&) [with lock_type = int]’
/usr/local/include/boost/thread/pthread/condition_variable.hpp:346:37:   instantiated from here
/usr/local/include/boost/thread/pthread/condition_variable.hpp:45:17: error: request for member ‘unlock’ in ‘m_’, which is of non-class type ‘int’

We see also that you changed only condition_variable.hpp

viboes — Wed, 23 Mar 2016 22:44:02 GMT

Please, could you give me an example?

viboes — Wed, 23 Mar 2016 22:49:46 GMT

Please could you add

        template <class lock_type>

just before

        inline bool do_wait_until(
          lock_type& m,
          struct timespec const &timeout)
        {

and tell me if this fixes the compilation error?

anonymous — Thu, 24 Mar 2016 09:50:13 GMT

This line "template <class lock_type>" is already there, even in the version at: https://github.com/boostorg/thread/blob/develop/include/boost/thread/pthread/condition_variable.hpp

see line 393

viboes — Thu, 24 Mar 2016 17:44:21 GMT

The issue is in line 339

https://github.com/boostorg/thread/blob/develop/include/boost/thread/pthread/condition_variable.hpp#L339

Sorry it was before this line

inline cv_status wait_until

viboes — Thu, 24 Mar 2016 23:02:50 GMT

fixed by https://github.com/boostorg/thread/commit/9db70b803d8f45632cd92cd61ab019c1ff3600b2

boriss@… — Thu, 31 Mar 2016 08:15:14 GMT

After your latest fix our application compile successfully. Also changing computer time doesn't cause problems anymore. Thanks for your support.

status changed; resolution set

viboes — Thu, 31 Mar 2016 22:17:16 GMT

status reopened → closed
resolution → fixed