Boost C++ Libraries: Ticket #1755: Boost.tokenizer is broken for wide characters - patch attached https://svn.boost.org/trac10/ticket/1755 <p> bug in token_functions.hpp - char_separator::is_kept and is_dropped call std::isspace and std::isdropped. But these cause an assert failure in debug build when parsing a wstring (at least on MSVC 7.1) since the CRT function isspace etc expect input to lie in the range 0-0xFF. MSDN documentation says that the behaviour is undefined outside this range; iswspace() etc are provided for use with wide characters. Please see the attached path for a fix to the problem. Note that this problem persists since at least 1.33, maybe even before then. </p> en-us Boost C++ Libraries /htdocs/site/boost.png https://svn.boost.org/trac10/ticket/1755 Trac 1.4.3 amit@… Wed, 02 Apr 2008 14:33:55 GMT attachment set https://svn.boost.org/trac10/ticket/1755 https://svn.boost.org/trac10/ticket/1755 <ul> <li><strong>attachment</strong> → <span class="trac-field-new">token_functions.diff</span> </li> </ul> <p> Patch containing a fix for 1755 </p> Ticket Noel Belcourt Thu, 10 Jun 2010 02:04:42 GMT status changed; resolution set https://svn.boost.org/trac10/ticket/1755#comment:1 https://svn.boost.org/trac10/ticket/1755#comment:1 <ul> <li><strong>status</strong> <span class="trac-field-old">new</span> → <span class="trac-field-new">closed</span> </li> <li><strong>resolution</strong> → <span class="trac-field-new">fixed</span> </li> </ul> <p> (In <a class="changeset" href="https://svn.boost.org/trac10/changeset/62706" title="Fix ticket 1755 by applying the attached patch. ">[62706]</a>) Fix ticket 1755 by applying the attached patch. </p> Ticket