Boost C++ Libraries: Ticket #293: tokenizer and signed char https://svn.boost.org/trac10/ticket/293 <pre class="wiki">boost::tokenizer incorrect works on platforms on which plain char is signed. For example, at least in boost::char_delimiters_separator::is_nonret should be explicit conversion from char type to int type. Something like this std::isspace (std::char_traits&lt;Char&gt;::to_int_type(E)). Minimal reproducer for this issue which uses Russian letters results in assert in MSVC 7.1: // cl /Zi /MDd -D_DEBUG test.cpp #include &lt;boost\tokenizer.hpp&gt; #include &lt;string&gt; #include &lt;iostream&gt; using namespace std; using namespace boost; int main() { string s = "4@02AB2C9 &lt;8@!"; tokenizer&lt;&gt; tok(s); for (tokenizer&lt;&gt;::iterator beg=tok.begin(); beg! =tok.end();++beg) { cout &lt;&lt; *beg &lt;&lt; "\n"; } } </pre> en-us Boost C++ Libraries /htdocs/site/boost.png https://svn.boost.org/trac10/ticket/293 Trac 1.4.3 dupamid Thu, 05 Aug 2004 11:55:34 GMT <link>https://svn.boost.org/trac10/ticket/293#comment:1 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/293#comment:1</guid> <description> <pre class="wiki">Logged In: YES user_id=1098099 Unfortunately Russian string was converted to something strange and I can’t attach file after submitting. If you need it please contact with me dupamid(at)yandex.ru. </pre> </description> <category>Ticket</category> </item> <item> <dc:creator>dupamid</dc:creator> <pubDate>Thu, 05 Aug 2004 12:05:33 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/293#comment:2 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/293#comment:2</guid> <description> <pre class="wiki">Logged In: YES user_id=1098099 Minimal reproducer without national letters: #include &lt;boost\tokenizer.hpp&gt; #include &lt;string&gt; #include &lt;iostream&gt; using namespace std; using namespace boost; int main() { string s = "Hello \xE4 world!"; tokenizer&lt;&gt; tok(s); for (tokenizer&lt;&gt;::iterator beg=tok.begin(); beg!=tok.end ();++beg) { cout &lt;&lt; *beg &lt;&lt; "\n"; } } </pre> </description> <category>Ticket</category> </item> <item> <dc:creator>Markus Schöpflin</dc:creator> <pubDate>Mon, 21 Nov 2005 10:09:16 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/293#comment:3 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/293#comment:3</guid> <description> <pre class="wiki">Logged In: YES user_id=91733 I could not reproduce this with the current 1.33 branch using cygwin and either -fsigned-char or -funsigned-char. Your reproduces always worked as expected. Could you please check if the problem still persists? </pre> </description> <category>Ticket</category> </item> <item> <dc:creator>sf-robot</dc:creator> <pubDate>Mon, 05 Dec 2005 18:43:30 GMT</pubDate> <title>status changed https://svn.boost.org/trac10/ticket/293#comment:4 https://svn.boost.org/trac10/ticket/293#comment:4 <ul> <li><strong>status</strong> <span class="trac-field-old">assigned</span> → <span class="trac-field-new">closed</span> </li> </ul> <pre class="wiki">Logged In: YES user_id=1312539 This Tracker item was closed automatically by the system. It was previously set to a Pending status, and the original submitter did not respond within 14 days (the time period specified by the administrator of this Tracker). </pre> Ticket