id,summary,reporter,owner,description,type,status,milestone,component,version,severity,resolution,keywords,cc
293,tokenizer and signed char,nobody,Markus Schöpflin,"{{{
boost::tokenizer incorrect works on platforms on which 
plain char is signed. For example, at least in 
boost::char_delimiters_separator::is_nonret should be 
explicit conversion from char type to int type. 
Something like this std::isspace
(std::char_traits<Char>::to_int_type(E)). Minimal 
reproducer for this issue which uses Russian letters 
results in assert in MSVC 7.1:

// cl /Zi /MDd -D_DEBUG test.cpp
#include <boost\tokenizer.hpp>
#include <string>
#include <iostream>
using namespace std;
using namespace boost;  
  
int main()
{
  string s = ""4@02AB2C9 <8@!"";
    
  tokenizer<> tok(s);
  for (tokenizer<>::iterator beg=tok.begin(); beg!
=tok.end();++beg)
  {
    cout << *beg << ""\n"";
  }
}


}}}",Bugs,closed,,None,None,,None,,