id	summary	reporter	owner	description	type	status	milestone	component	version	severity	resolution	keywords	cc
1755	Boost.tokenizer is broken for wide characters - patch attached	amit@…	jsiek	"bug in token_functions.hpp - char_separator::is_kept and is_dropped call std::isspace and std::isdropped. But these cause an assert failure in debug build when parsing a wstring (at least on MSVC 7.1) since the CRT function isspace etc expect input to lie in the range 0-0xFF. MSDN documentation says that the behaviour is undefined outside this range; iswspace() etc are provided for use with wide characters.
Please see the attached path for a fix to the problem.
Note that this problem persists since at least 1.33, maybe even before then."	Bugs	closed	Boost 1.36.0	tokenizer	Boost 1.35.0	Problem	fixed	wchar_t, isspace, ispunct