Boost C++ Libraries: Ticket #943: tokenizer fails with wchar_t https://svn.boost.org/trac10/ticket/943 <pre class="wiki">Hi, The tokenizer fails when instantiated with wchar_t; there are a couple of problems with the char_delimiters_separator ctor: 1. The third parameter is typed "char" rather than "Char" 2. The initializers of returnable_ and nonreturnable_ are using "" which is of type "char const *" rather than "Char const *". A patch is appended below. Ram (amberarrow@yahoo.com) ==================================================== *** token_functions.old.hpp Fri Jul 6 14:01:45 2001 --- token_functions.hpp Mon Jul 30 15:34:34 2001 *************** *** 249,258 **** public: explicit char_delimiters_separator(bool return_delims = false, ! const Char * returnable = 0,const char * nonreturnable = 0) ! :nonreturnable_(nonreturnable?nonreturnable:""), ! returnable_(returnable?returnable:""),no_ispunct_(returnable), ! no_isspace_(nonreturnable),return_delims_(return_delims){} void reset(){} --- 249,263 ---- public: explicit char_delimiters_separator(bool return_delims = false, ! const Char * returnable = 0,const Char * nonreturnable = 0) ! :no_ispunct_(returnable), ! no_isspace_(nonreturnable),return_delims_(return_delims) ! { ! if (nonreturnable) ! nonreturnable_ = nonreturnable; ! if (returnable) ! returnable_ = returnable; ! } void reset(){} </pre> en-us Boost C++ Libraries /htdocs/site/boost.png https://svn.boost.org/trac10/ticket/943 Trac 1.4.3 jbandela Tue, 31 Jul 2001 18:29:53 GMT <link>https://svn.boost.org/trac10/ticket/943#comment:1 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/943#comment:1</guid> <description> <pre class="wiki">Logged In: YES user_id=244347 Version 1.7 of token_functions.hpp should be able to work. It's constructor is belos: explicit char_delimiters_separator(bool return_delims = false, const Char * returnable = 0,const Char * nonreturnable = 0) :nonreturnable_(nonreturnable? nonreturnable:string_type().c_str()), returnable_(returnable? returnable:string_type().c_str()), no_ispunct_(returnable),no_isspace_ (nonreturnable), return_delims_(return_delims){} Regards, John R. Bandela </pre> </description> <category>Ticket</category> </item> <item> <dc:creator>Markus Schöpflin</dc:creator> <pubDate>Mon, 21 Nov 2005 09:18:15 GMT</pubDate> <title>status changed https://svn.boost.org/trac10/ticket/943#comment:2 https://svn.boost.org/trac10/ticket/943#comment:2 <ul> <li><strong>status</strong> <span class="trac-field-old">assigned</span> → <span class="trac-field-new">closed</span> </li> </ul> <pre class="wiki">Logged In: YES user_id=91733 As far as I can tell this has been fixed ages ago. </pre> Ticket