Boost C++ Libraries: Ticket #5566: Spirit.Lex converting a token to its value in a lexer semantic action fails

owner changed

Hartmut Kaiser — Fri, 27 May 2011 00:33:03 GMT

owner changed from Joel de Guzman to Hartmut Kaiser

Hartmut Kaiser — Fri, 27 May 2011 01:21:23 GMT

It does not exactly work as you expect as the type of teh token value is a variant (initially holding a pair of iterators pointing to the matched input sequence). You need to explicitly convert this iterator range into the attribute type. Normally this is done while parsing on first access to the attribute. Here is your modified code doing this:

#include <string>
#include <vector>
#include <boost/spirit/include/lex_lexertl.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_stl.hpp>
namespace lex = boost::spirit::lex;
namespace phoenix = boost::phoenix;
struct get_string_impl
{
    template <typename Value>
    struct result
    {
        typedef std::string type;
    };
    template <typename Value>
    std::string operator()(Value const& val) const
    {
        // transform the token value (here a variant) into a string
        // at this point the variant holds a pair of iterators
        typedef boost::iterator_range<char const*> iterpair_type;
        iterpair_type const& ip = boost::get<iterpair_type>(val);
        return std::string(ip.begin(), ip.end());
    }
};
boost::phoenix::function<get_string_impl> get_string;
template <typename Lexer>
struct text_decorator_lexer : lex::lexer<Lexer>
{
        text_decorator_lexer()
                : tag_open("\\{\\{.+?\\}\\}")
                , tag_close("\\{\\{\\/.+?\\}\\}")
                , var("\\$\\$.+?\\$\\$")
                , any(".")
        {
                using boost::spirit::lex::_val;
                using boost::phoenix::push_back;
                using boost::phoenix::ref;
                this->self
                        = tag_open  [push_back(ref(tokens), get_string(_val))]
                        | tag_close [push_back(ref(tokens), get_string(_val))]
                        | var       [push_back(ref(tokens), get_string(_val))]
                        | any       [push_back(ref(tokens), get_string(_val))]
                        ;
        }
        std::vector<std::string> tokens;
        lex::token_def<std::string> tag_open, tag_close, var, any;
};
int main()
{
        typedef lex::lexertl::token<
            const char*,
            boost::mpl::vector<std::string>,
            boost::mpl::false_
        > token_type;
        typedef lex::lexertl::actor_lexer<token_type> lexer_type;
        text_decorator_lexer<lexer_type> text_decorator_tokenizer;
        std::string input = "Hello {{b}}there{{/b}}, the time is $$time$$.";
        char const* first = input.c_str();
        char const* last = &first[input.size()];
        lex::tokenize(first, last, text_decorator_tokenizer);
}

But to be honest, I needed to fix a (unrelated) problem to make this code work, therefore you'll need to update from SVN or wait for Boost V1.47.

status changed; resolution set

Hartmut Kaiser — Fri, 27 May 2011 02:02:24 GMT

status new → closed
resolution → fixed

(In [72201]) Spirit: fixed #5566: Spirit.Lex converting a token to its value in a lexer semantic action fails

Thu, 11 Apr 2013 17:23:04 GMT

What I see from the token's variant is that the value replaces the pair of iterators pointing to the matched input sequence. Is there a way to store the converted value and, in the same time, keeping the pair of iterators pointing to the matched input sequence ?

Hartmut Kaiser — Thu, 11 Apr 2013 17:37:51 GMT

Replying to Gilles <gillesb68@…>:

What I see from the token's variant is that the value replaces the pair of iterators pointing to the matched input sequence. Is there a way to store the converted value and, in the same time, keeping the pair of iterators pointing to the matched input sequence ?

Yes, simply use position_token instead of the plain token class.

Thu, 11 Apr 2013 18:37:01 GMT

Replying to hkaiser:

Replying to Gilles <gillesb68@…>:

What I see from the token's variant is that the value replaces the pair of iterators pointing to the matched input sequence. Is there a way to store the converted value and, in the same time, keeping the pair of iterators pointing to the matched input sequence ?

Yes, simply use position_token instead of the plain token class.

I understand this for a token enclosing the pair pointing the matched input sequence alone, but how do I declare a token enclosing together :

1/ the pair of iterators pointing to the matched input sequence; and

2/ the converted value of the same input sequence ?

Hartmut Kaiser — Thu, 11 Apr 2013 23:09:40 GMT

Replying to Gilles <gillesb68@…>:

Replying to hkaiser:

Replying to Gilles <gillesb68@…>:

What I see from the token's variant is that the value replaces the pair of iterators pointing to the matched input sequence. Is there a way to store the converted value and, in the same time, keeping the pair of iterators pointing to the matched input sequence ?

Yes, simply use position_token instead of the plain token class.

I understand this for a token enclosing the pair pointing the matched input sequence alone, but how do I declare a token enclosing together :

1/ the pair of iterators pointing to the matched input sequence; and

2/ the converted value of the same input sequence ?

Yes, that's what you can do with the position_token type. It is semantically equivalent to the token type, except for the additionally stored iterator pair of the matched input sequence. Please see libs/example/qi/compiler_tutorial/conjure3 for an example.

anonymous — Fri, 12 Apr 2013 14:06:59 GMT

Ah, the position_token class, of course, I should have thought. Thanks a lot, and I apologize for making you wasting time about something so obvious.