id,summary,reporter,owner,description,type,status,milestone,component,version,severity,resolution,keywords,cc
1755,Boost.tokenizer is broken for wide characters - patch attached,amit@…,jsiek,"bug in token_functions.hpp - char_separator::is_kept and is_dropped call std::isspace and std::isdropped. But these cause an assert failure in debug build when parsing a wstring (at least on MSVC 7.1) since the CRT function isspace etc expect input to lie in the range 0-0xFF. MSDN documentation says that the behaviour is undefined outside this range; iswspace() etc are provided for use with wide characters.
Please see the attached path for a fix to the problem.
Note that this problem persists since at least 1.33, maybe even before then.",Bugs,closed,Boost 1.36.0,tokenizer,Boost 1.35.0,Problem,fixed,"wchar_t, isspace, ispunct",