Boost C++ Libraries: Ticket #12174: Tokenizer delivers additional null byte - string token

attachment set

Tue, 03 May 2016 10:29:38 GMT

attachment → sep.cpp

minimal test program producing the bug

damian.meden@… — Fri, 06 May 2016 22:25:10 GMT

I just tried with several boost versions and I have the same issue but in my case all the streams you have in your example fails after getting out the string directly into the Container.

a workaround would be:

std::string str = tmp.str(); boost::tokenizer<boost::char_separator<char> > t(str, sep);

then it will work.

I've tested this with VS14, gcc4.8 and gcc4.9.

I'll have a look at the implementation.

Thanks.

damian.meden@… — Thu, 12 May 2016 13:42:39 GMT

Hi, more data here:

tokeniser holds an iterator to the beginning and the end of the container, ostringstream::str() will return a temporary container(in this case std::string) which will be deleted at the end of the expression where you are using it, meaning that the tokeniser will holds a iterator to a temporary object which was already destroyed, this is Undefined Behaviour, different compiler are doing different things. If you hold the temporal object in scope(string str=tmp.str()) meanwhile you tokenize it will fix the issue, that's the proper Idiom I would say.

Also this is not related to the 'escaped' character \n, you can use any proper character there and will have the same results.

Dam.

Fri, 13 May 2016 06:55:58 GMT

Hi Dam, thanks a lot. It's clear now what is going on, and easy to avoid following your suggestion. Kind Regards, Peter