tokenizer and signed char

[nobody]

boost::tokenizer incorrect works on platforms on which 
plain char is signed. For example, at least in 
boost::char_delimiters_separator::is_nonret should be 
explicit conversion from char type to int type. 
Something like this std::isspace
(std::char_traits<Char>::to_int_type(E)). Minimal 
reproducer for this issue which uses Russian letters 
results in assert in MSVC 7.1:

// cl /Zi /MDd -D_DEBUG test.cpp
#include <boost\tokenizer.hpp>
#include <string>
#include <iostream>
using namespace std;
using namespace boost;  
  
int main()
{
  string s = "4@02AB2C9 <8@!";
    
  tokenizer<> tok(s);
  for (tokenizer<>::iterator beg=tok.begin(); beg!
=tok.end();++beg)
  {
    cout << *beg << "\n";
  }
}

[dupamid]

Logged In: YES 
user_id=1098099

Unfortunately Russian string was converted to something 
strange and I can’t attach file after submitting. If you need it 
please contact with me dupamid(at)yandex.ru.

[dupamid]

Logged In: YES 
user_id=1098099

Minimal reproducer without national letters:

#include <boost\tokenizer.hpp>
#include <string>
#include <iostream>
using namespace std;
using namespace boost;  
  
int main()
{
  string s = "Hello \xE4 world!";

  tokenizer<> tok(s);
  for (tokenizer<>::iterator beg=tok.begin(); beg!=tok.end
();++beg)
  {
    cout << *beg << "\n";
  }
}

[Markus Schöpflin]

Logged In: YES 
user_id=91733

I could not reproduce this with the current 1.33 branch
using cygwin and either -fsigned-char or -funsigned-char.
Your reproduces always worked as expected.

Could you please check if the problem still persists?

[sf-robot]

Logged In: YES 
user_id=1312539

This Tracker item was closed automatically by the system. It was
previously set to a Pending status, and the original submitter
did not respond within 14 days (the time period specified by
the administrator of this Tracker).