Opened 10 years ago
Closed 5 years ago
#6868 closed Bugs (fixed)
lex tokenize result error
| Reported by: | icosagon | Owned by: | Joel de Guzman | 
|---|---|---|---|
| Milestone: | To Be Determined | Component: | spirit | 
| Version: | Boost 1.49.0 | Severity: | Problem | 
| Keywords: | spirit lex | Cc: | 
Description
if uncomment the line #include <boost/spirit/include/phoenix_statement.hpp> the tokenize result is correct, why?
the result is lines: 7, words: 0, characters: 155
the correct result is lines: 7, words: 33, characters: 162
visual studio 2010
// #define BOOST_SPIRIT_LEXERTL_DEBUG
 
#include <boost/config/warning_disable.hpp>
#include <boost/spirit/include/lex_lexertl.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
//if uncomment next line, the tokenize result is correct, why?
//#include <boost/spirit/include/phoenix_statement.hpp>
#include <boost/spirit/include/phoenix_algorithm.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <iostream>
#include <fstream>
#include <string>
namespace lex = boost::spirit::lex;
struct distance_func
{
	template <typename Iterator1, typename Iterator2>
	struct result : boost::iterator_difference<Iterator1> {};
	template <typename Iterator1, typename Iterator2>
	typename result<Iterator1, Iterator2>::type 
		operator()(Iterator1& begin, Iterator2& end) const
	{
		return std::distance(begin, end);
	}
};
boost::phoenix::function<distance_func> const distance1 = distance_func();
template <typename Lexer>
struct word_count_tokens : lex::lexer<Lexer>
{
	word_count_tokens()
		: c(0), w(0), l(0)
		, word("[^ \t\n]+")     
		, eol("\n")
		, any(".")
	{
		using boost::spirit::lex::_start;
		using boost::spirit::lex::_end;
		using boost::phoenix::ref;
		this->self 
			=   word  [++ref(w), ref(c) += distance1(_start, _end)]
		|   eol   [++ref(c), ++ref(l)] 
		|   any   [++ref(c)]
		;
	}
	std::size_t c, w, l;
	lex::token_def<> word, eol, any;
};
int main(int argc, char* argv[])
{
	typedef 
		lex::lexertl::token<char const*, lex::omit, boost::mpl::false_> 
		token_type;
	typedef lex::lexertl::actor_lexer<token_type> lexer_type;
	word_count_tokens<lexer_type> word_count_lexer;
	std::string str ("Our hiking boots are ready.  So, let's pack!\n\
					 \n\
		Have you the plane tickets for there and back?\n\
		\n\
		I do, I do.  We're all ready to go.  Grab my hand and be my beau.\n\
		\n\
		\n");
	char const* first = str.c_str();
	char const* last = &first[str.size()];
	lexer_type::iterator_type iter = word_count_lexer.begin(first, last);
	lexer_type::iterator_type end = word_count_lexer.end();
	while (iter != end && token_is_valid(*iter))
		++iter;
	if (iter == end) {
		std::cout << "lines: " << word_count_lexer.l 
			<< ", words: " << word_count_lexer.w 
			<< ", characters: " << word_count_lexer.c 
			<< "\n";
	}
	else {
		std::string rest(first, last);
		std::cout << "Lexical analysis failed\n" << "stopped at: \"" 
			<< rest << "\"\n";
	}
	return 0;
}
      Change History (2)
comment:1 by , 5 years ago
comment:2 by , 5 years ago
| Resolution: | → fixed | 
|---|---|
| Status: | new → closed | 
  Note:
 See   TracTickets
 for help on using tickets.
    
I tried your example and it run fine (except #6869) and I have got
lines: 7, words: 33, characters: 178in both cases. I'm sorry I'll not going to find the commit where it was fixed (it might was phoenix v2 problem).