tokenizer fails with wchar_t

[nobody]

Hi,

The tokenizer fails when instantiated with wchar_t; there are a couple of
problems with the char_delimiters_separator ctor:
1. The third parameter is typed "char" rather than "Char"
2. The initializers of returnable_ and nonreturnable_ are using ""
    which is of type "char const *" rather than "Char const *".
A patch is appended below.

Ram (amberarrow@yahoo.com)

====================================================

*** token_functions.old.hpp	Fri Jul  6 14:01:45 2001
--- token_functions.hpp	Mon Jul 30 15:34:34 2001
***************
*** 249,258 ****
    public:
  
       explicit char_delimiters_separator(bool return_delims = false, 
! 		 const Char * returnable = 0,const char * nonreturnable = 0)
! 		 :nonreturnable_(nonreturnable?nonreturnable:""),
! 		 returnable_(returnable?returnable:""),no_ispunct_(returnable),
! 		 no_isspace_(nonreturnable),return_delims_(return_delims){}
  
       void reset(){}
  
--- 249,263 ----
    public:
  
       explicit char_delimiters_separator(bool return_delims = false, 
! 		 const Char * returnable = 0,const Char * nonreturnable = 0)
! 		 :no_ispunct_(returnable),
! 		 no_isspace_(nonreturnable),return_delims_(return_delims)
! 		  {
! 			  if (nonreturnable)
! 				  nonreturnable_ = nonreturnable;
! 			  if (returnable)
! 				  returnable_ = returnable;
! 		  }
  
       void reset(){}
  

[jbandela]

Logged In: YES 
user_id=244347

Version 1.7 of token_functions.hpp should be able to work. 
It's constructor is belos:


     explicit char_delimiters_separator(bool return_delims 
= false, 
		 const Char * returnable = 0,const Char * 
nonreturnable = 0)
		 :nonreturnable_(nonreturnable?
nonreturnable:string_type().c_str()),
		 returnable_(returnable?
returnable:string_type().c_str()),
		 no_ispunct_(returnable),no_isspace_
(nonreturnable),
		 return_delims_(return_delims){}


Regards,

John R. Bandela

[Markus Schöpflin]

Logged In: YES 
user_id=91733

As far as I can tell this has been fixed ages ago.