Boost C++ Libraries: Ticket #3670: endless loop in dev_poll_reactor on Solaris https://svn.boost.org/trac10/ticket/3670 <p> The following example-program exposes high CPU-usage on Solaris 10, despite not doing anything useful while sleeping. Top reports a CPU-usage between 10 and 25% on a 4 CPU-box. </p> <p> In order to reproduce, a TCP-server has to be listening on the configured host-port. </p> <pre class="wiki">#include &lt;boost/asio.hpp&gt; #include &lt;boost/thread/thread.hpp&gt; #include &lt;time.h&gt; #include &lt;vector&gt; #include &lt;iostream&gt; // g++ -pthread -g2 -I/opt/include/boost-1_41 -L/opt/lib -lboost_system-gcc43-mt-1_41 -lboost_thread-gcc43-mt-1_41 -lsocket -lnsl f.cc // this program exposes a high CPU usage when run on solaris 10 using the dev_poll_reactor (the default). const char HOSTNAME[]="localhost"; const char PORT[]="46000"; const int THREADS=3; const int SOCKETS=10; const int LOOPS=10; const int SLEEPTIME=10; // seconds void handler(const boost::system::error_code&amp; error) { std::cout &lt;&lt; "socket connected '" &lt;&lt; error &lt;&lt; "'" &lt;&lt; std::endl; } int main() { using namespace boost::asio::ip; boost::asio::io_service service; boost::asio::io_service::work work(service); boost::thread_group pool; for(int i=0;i&lt;THREADS;++i) pool.create_thread(boost::bind(&amp;boost::asio::io_service::run, boost::ref(service))); tcp::resolver resolver(service); tcp::resolver::query query(HOSTNAME,PORT); tcp::resolver::iterator result=resolver.resolve(query); const tcp::resolver::iterator end; if(result==end) { return -1; } for(int i=0;0&lt;LOOPS;++i) { typedef std::vector&lt;boost::shared_ptr&lt;boost::asio::ip::tcp::socket&gt; &gt; Sockets; Sockets sockets; for(int j=0;j&lt;SOCKETS;++j) { sockets.push_back( boost::shared_ptr&lt;boost::asio::ip::tcp::socket&gt; (new boost::asio::ip::tcp::socket(service,boost::asio::ip::tcp::v4()))); } for(Sockets::iterator si=sockets.begin(); si!=sockets.end(); ++si) { (*si)-&gt;async_connect(*result,handler); } sleep(SLEEPTIME); } } </pre><p> The cause seems to be an endless loop in dev_poll_reactor::run </p> <p> The problematic stacktrace: </p> <pre class="wiki">#0 0xffffffff7e0d3a50 in _write () from /lib/64/libc.so.1 #1 0xffffffff7e0c4394 in write () from /lib/64/libc.so.1 #2 0x0000000100024258 in boost::asio::detail::dev_poll_reactor&lt;false&gt;::run (this=0x100131e80, block=true) at /opt/gnu4u_prince_ext/include/boost-1_41/boost/asio/detail/dev_poll_reactor.hpp:467 #3 0x0000000100024850 in boost::asio::detail::task_io_service&lt;boost::asio::detail::dev_poll_reactor&lt;false&gt; &gt;::do_one (this=0x10012f090, lock=@0xffffffff7d0fba28, this_idle_thread=0xffffffff7d0fba08, ec=@0xffffffff7d0fbb18) at /opt/gnu4u_prince_ext/include/boost-1_41/boost/asio/detail/task_io_service.hpp:260 #4 0x0000000100024adc in boost::asio::detail::task_io_service&lt;boost::asio::detail::dev_poll_reactor&lt;false&gt; &gt;::run (this=0x10012f090, ec=@0xffffffff7d0fbb18) at /opt/gnu4u_prince_ext/include/boost-1_41/boost/asio/detail/task_io_service.hpp:103 #5 0x0000000100024c34 in boost::asio::io_service::run (this=0xffffffff7ffff028) at /opt/gnu4u_prince_ext/include/boost-1_41/boost/asio/impl/io_service.ipp:58 #6 0x000000010000d33c in boost::_mfi::mf0&lt;unsigned long, boost::asio::io_service&gt;::operator() (this=0x100131b20, t=@0xffffffff7ffff028) at /opt/gnu4u_prince_ext/include/boost-1_41/boost/bind/mem_fn_template.hpp:68 #7 0x000000010000d400 in boost::_bi::list1&lt;boost::reference_wrapper&lt;boost::asio::io_service&gt; &gt;::operator()&lt;unsigned long, boost::_mfi::mf0&lt;unsigned long, boost::asio::io_service&gt;, boost::_bi::list0&gt; (this=0x100131b30, f=@0x100131b20, a=@0xffffffff7d0fbd9f) at /opt/gnu4u_prince_ext/include/boost-1_41/boost/bind/bind.hpp:236 #8 0x000000010000d460 in boost::_bi::bind_t&lt;unsigned long, boost::_mfi::mf0&lt;unsigned long, boost::asio::io_service&gt;, boost::_bi::list1&lt;boost::reference_wrapper&lt;boost::asio::io_service&gt; &gt; &gt;::operator() (this=0x100131b20) at /opt/gnu4u_prince_ext/include/boost-1_41/boost/bind/bind_template.hpp:20 #9 0x000000010000d48c in boost::detail::thread_data&lt;boost::_bi::bind_t&lt;unsigned long, boost::_mfi::mf0&lt;unsigned long, boost::asio::io_service&gt;, boost::_bi::list1&lt;boost::reference_wrapper&lt;boost::asio::io_service&gt; &gt; &gt; &gt;::run ( this=0x100131a50) at /opt/gnu4u_prince_ext/include/boost-1_41/boost/thread/detail/thread.hpp:56 #10 0xffffffff7f00ba30 in thread_proxy () from /opt/gnu4u_prince_ext/lib/libboost_thread-gcc43-mt-1_41.so.1.41.0 #11 0xffffffff7e0d24e0 in _lwp_start () from /lib/64/libc.so.1 #12 0xffffffff7e0d24e0 in _lwp_start () from /lib/64/libc.so.1 </pre><p> the local variables just after the write: </p> <pre class="wiki">(gdb) info locals ev = {fd = 13, events = 24, revents = 0} result = 8 more_reads = false more_writes = false more_except = false ec = {m_val = 0, m_cat = 0xffffffff7f4033a8} descriptor = 13 i = 0 lock = {&lt;boost::noncopyable_::noncopyable&gt; = {&lt;No data fields&gt;}, mutex_ = @0x100131ea8, locked_ = true} events_size = 0 timeout = -1 events = {{fd = 13, events = 28, revents = 4}, {fd = 14, events = 28, revents = 4}, {fd = 16, events = 28, revents = 4}, {fd = 0, events = 0, revents = 0} &lt;repeats 125 times&gt;} dp = {dp_fds = 0xffffffff7d0fb2d0, dp_nfds = 128, dp_timeout = -1} num_events = 3 sb = {&lt;boost::noncopyable_::noncopyable&gt; = {&lt;No data fields&gt;}, blocked_ = true, old_mask_ = {__sigbits = {0, 0, 0, 0}}} </pre><p> The program works fine on Linux and works fine on Solaris when compiled with the flag -DBOOST_ASIO_DISABLE_DEV_POLL. </p> <p> I am pretty sure that this behaviour has been present since at least boost 1.37. </p> en-us Boost C++ Libraries /htdocs/site/boost.png https://svn.boost.org/trac10/ticket/3670 Trac 1.4.3 chris_kohlhoff Sat, 02 Jan 2010 12:51:17 GMT <link>https://svn.boost.org/trac10/ticket/3670#comment:1 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/3670#comment:1</guid> <description> <p> I cannot reproduce the problem. However, please try the following fix and report the results: </p> <p> Index: dev_poll_reactor.hpp =================================================================== --- dev_poll_reactor.hpp (revision 58628) +++ dev_poll_reactor.hpp (working copy) @@ -435,7 +435,6 @@ </p> <blockquote> <p> more_writes = write_op_queue_.has_operation(descriptor); </p> </blockquote> <blockquote> <p> if ((events[i].events &amp; (POLLERR | POLLHUP)) != 0 </p> </blockquote> <ul><li> &amp;&amp; (events[i].events &amp; ~(POLLERR | POLLHUP)) == 0 </li></ul><blockquote> <p> &amp;&amp; !more_except &amp;&amp; !more_reads &amp;&amp; !more_writes) </p> </blockquote> <blockquote> <p> { </p> <blockquote> <p> <em> If we have an event and no operations associated with the </em></p> </blockquote> </blockquote> </description> <category>Ticket</category> </item> <item> <author>dirkmoermans@…</author> <pubDate>Mon, 04 Jan 2010 13:01:17 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/3670#comment:2 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/3670#comment:2</guid> <description> <p> Thanks for the patch. I will try it out as soon as I get access to a solaris box (somewhere next week). </p> </description> <category>Ticket</category> </item> <item> <author>dirkmoermans@…</author> <pubDate>Wed, 06 Jan 2010 09:11:51 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/3670#comment:3 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/3670#comment:3</guid> <description> <p> The patch solves the problem for me. </p> <p> I dumped some traces with the contents of events[i].events and got as output without the patch in an endless loop: </p> <pre class="wiki">events[i].events=1c events[i].events &amp; ~(POLLERR | POLLHUP) = 4 </pre><p> 4 corresponds to POLLOUT in sys/poll.h: </p> <pre class="wiki">#define POLLOUT 0x0004 /* fd is writeable (won't block) */ </pre><p> I don't know why it is not reproducable, but it would be great if you would include your patch in the next boost-release. </p> </description> <category>Ticket</category> </item> <item> <dc:creator>chris_kohlhoff</dc:creator> <pubDate>Wed, 06 Jan 2010 12:36:53 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/3670#comment:4 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/3670#comment:4</guid> <description> <p> (In <a class="changeset" href="https://svn.boost.org/trac10/changeset/58762" title="Apply fix for reported excessive CPU usage under Solaris. Refs #3670. ">[58762]</a>) Apply fix for reported excessive CPU usage under Solaris. Refs <a class="closed ticket" href="https://svn.boost.org/trac10/ticket/3670" title="#3670: Bugs: endless loop in dev_poll_reactor on Solaris (closed: fixed)">#3670</a>. </p> </description> <category>Ticket</category> </item> <item> <dc:creator>chris_kohlhoff</dc:creator> <pubDate>Mon, 11 Jan 2010 02:34:08 GMT</pubDate> <title>status changed; resolution set https://svn.boost.org/trac10/ticket/3670#comment:5 https://svn.boost.org/trac10/ticket/3670#comment:5 <ul> <li><strong>status</strong> <span class="trac-field-old">new</span> → <span class="trac-field-new">closed</span> </li> <li><strong>resolution</strong> → <span class="trac-field-new">fixed</span> </li> </ul> <p> (In <a class="changeset" href="https://svn.boost.org/trac10/changeset/58883" title="Merge from trunk. Fixes #3743, #3670, #3822. ........ r58670 | ...">[58883]</a>) Merge from trunk. Fixes <a class="closed ticket" href="https://svn.boost.org/trac10/ticket/3743" title="#3743: Patches: support for partially broken compilers/platforms (closed: fixed)">#3743</a>, <a class="closed ticket" href="https://svn.boost.org/trac10/ticket/3670" title="#3670: Bugs: endless loop in dev_poll_reactor on Solaris (closed: fixed)">#3670</a>, <a class="closed ticket" href="https://svn.boost.org/trac10/ticket/3822" title="#3822: Bugs: boost/asio/buffer.hpp MSVC workaround is inaccurately targeted (closed: fixed)">#3822</a>. </p> <p> ........ </p> <blockquote> <p> <a class="changeset" href="https://svn.boost.org/trac10/changeset/58670" title="Fix example to compile with MSVC 10 beta 2. ">r58670</a> | chris_kohlhoff | 2010-01-04 23:33:04 +1100 (Mon, 04 Jan 2010) | 2 lines </p> </blockquote> <p> </p> <blockquote> <p> Fix example to compile with MSVC 10 beta 2. </p> </blockquote> <p> ........ </p> <blockquote> <p> <a class="changeset" href="https://svn.boost.org/trac10/changeset/58671" title="Fix Win64 warnings. ">r58671</a> | chris_kohlhoff | 2010-01-04 23:33:42 +1100 (Mon, 04 Jan 2010) | 2 lines </p> </blockquote> <p> </p> <blockquote> <p> Fix Win64 warnings. </p> </blockquote> <p> ........ </p> <blockquote> <p> <a class="changeset" href="https://svn.boost.org/trac10/changeset/58703" title="Include boost/limits.hpp rather than &lt;limits&gt;, to support older ...">r58703</a> | chris_kohlhoff | 2010-01-05 22:51:41 +1100 (Tue, 05 Jan 2010) | 3 lines </p> </blockquote> <p> </p> <blockquote> <p> Include boost/limits.hpp rather than &lt;limits&gt;, to support older compilers. Refs <a class="closed ticket" href="https://svn.boost.org/trac10/ticket/3743" title="#3743: Patches: support for partially broken compilers/platforms (closed: fixed)">#3743</a>. </p> </blockquote> <p> ........ </p> <blockquote> <p> <a class="changeset" href="https://svn.boost.org/trac10/changeset/58704" title="Drop back to second_clock if microsec_clock is unavailable. Refs #3743. ">r58704</a> | chris_kohlhoff | 2010-01-05 23:20:10 +1100 (Tue, 05 Jan 2010) | 2 lines </p> </blockquote> <p> </p> <blockquote> <p> Drop back to second_clock if microsec_clock is unavailable. Refs <a class="closed ticket" href="https://svn.boost.org/trac10/ticket/3743" title="#3743: Patches: support for partially broken compilers/platforms (closed: fixed)">#3743</a>. </p> </blockquote> <p> ........ </p> <blockquote> <p> <a class="changeset" href="https://svn.boost.org/trac10/changeset/58705" title="Use sockatmark if SIOCATMARK is not defined. Refs #3743. ">r58705</a> | chris_kohlhoff | 2010-01-05 23:50:39 +1100 (Tue, 05 Jan 2010) | 2 lines </p> </blockquote> <p> </p> <blockquote> <p> Use sockatmark if SIOCATMARK is not defined. Refs <a class="closed ticket" href="https://svn.boost.org/trac10/ticket/3743" title="#3743: Patches: support for partially broken compilers/platforms (closed: fixed)">#3743</a>. </p> </blockquote> <p> ........ </p> <blockquote> <p> <a class="changeset" href="https://svn.boost.org/trac10/changeset/58740" title="Use buffer debugging workaround with MSVC 8 only. ">r58740</a> | chris_kohlhoff | 2010-01-06 13:38:39 +1100 (Wed, 06 Jan 2010) | 2 lines </p> </blockquote> <p> </p> <blockquote> <p> Use buffer debugging workaround with MSVC 8 only. </p> </blockquote> <p> ........ </p> <blockquote> <p> <a class="changeset" href="https://svn.boost.org/trac10/changeset/58761" title="Disable iostreams-related functionality if BOOST_NO_IOSTREAMS is ...">r58761</a> | chris_kohlhoff | 2010-01-06 23:27:05 +1100 (Wed, 06 Jan 2010) | 2 lines </p> </blockquote> <p> </p> <blockquote> <p> Disable iostreams-related functionality if BOOST_NO_IOSTREAMS is defined. Refs <a class="closed ticket" href="https://svn.boost.org/trac10/ticket/3743" title="#3743: Patches: support for partially broken compilers/platforms (closed: fixed)">#3743</a>. </p> </blockquote> <p> ........ </p> <blockquote> <p> <a class="changeset" href="https://svn.boost.org/trac10/changeset/58762" title="Apply fix for reported excessive CPU usage under Solaris. Refs #3670. ">r58762</a> | chris_kohlhoff | 2010-01-06 23:36:51 +1100 (Wed, 06 Jan 2010) | 2 lines </p> </blockquote> <p> </p> <blockquote> <p> Apply fix for reported excessive CPU usage under Solaris. Refs <a class="closed ticket" href="https://svn.boost.org/trac10/ticket/3670" title="#3670: Bugs: endless loop in dev_poll_reactor on Solaris (closed: fixed)">#3670</a>. </p> </blockquote> <p> ........ </p> <blockquote> <p> <a class="changeset" href="https://svn.boost.org/trac10/changeset/58782" title="Disable handler allocation and invocation hooks on g++ 2.x. Refs #3743. ">r58782</a> | chris_kohlhoff | 2010-01-07 09:44:48 +1100 (Thu, 07 Jan 2010) | 2 lines </p> </blockquote> <p> </p> <blockquote> <p> Disable handler allocation and invocation hooks on g++ 2.x. Refs <a class="closed ticket" href="https://svn.boost.org/trac10/ticket/3743" title="#3743: Patches: support for partially broken compilers/platforms (closed: fixed)">#3743</a>. </p> </blockquote> <p> ........ </p> <blockquote> <p> <a class="changeset" href="https://svn.boost.org/trac10/changeset/58793" title="Support platforms that don't define INET6_ADDRSTRLEN. Refs #3743. ">r58793</a> | chris_kohlhoff | 2010-01-08 09:18:16 +1100 (Fri, 08 Jan 2010) | 2 lines </p> </blockquote> <p> </p> <blockquote> <p> Support platforms that don't define INET6_ADDRSTRLEN. Refs <a class="closed ticket" href="https://svn.boost.org/trac10/ticket/3743" title="#3743: Patches: support for partially broken compilers/platforms (closed: fixed)">#3743</a>. </p> </blockquote> <p> ........ </p> Ticket