[spirit.lex] self("state") += calls a wrong operator +=

[Vyacheslav Andrejev]

This construct

      this->self("SOME_STATE") += someTokenDef;

doesn't call

        template <typename LexerDef, typename Expr>
        inline lexer_def_<LexerDef>&
        boost::spirit::lex::detail::operator+= (lexer_def_<LexerDef>& lexdef, Expr&& xpr)

as it should in order for the definition get added to the lexer. Instead this operator

        template <typename Left, typename Right>
        typename boost::proto::detail::enable_binary<
            deduce_domain
          , deduce_domain::proto_grammar
          , boost::mpl::or_<is_extension<Left>, is_extension<Right> >
          , boost::proto::tag::plus_assign
          , Left const &
          , Right const &
          >::type const
        operator +=(Left &&left, Right &&right)
        {
            return boost::proto::detail::make_expr_<
                boost::proto::tag::plus_assign
              , deduce_domain
              , Left const &
              , Right const &
              >()(left, right);
        }

is called from boost::proto::exprns_ namespace.

[Vyacheslav Andrejev]

An example that demonstrates the problem:

#include <iostream>

#include <boost/spirit/include/lex_lexertl.hpp>

using namespace boost::spirit;

template <typename Lexer>
struct tokens : public lex::lexer<Lexer>
{
  tokens()
  {
    someTokenDef = '"';

    auto expr = (this->self("SOME_STATE") += someTokenDef);
    std::cout << "Arity = " << expr.proto_arity_c << ", should be 0" << std::endl;
    std::cout << "token_id_ = " << someTokenDef.id() << ", should be 65536" << std::endl;
    std::cout << "unique_id_ = " << someTokenDef.unique_id() << ", should be 0" << std::endl;
    std::cout << "token_state_ = " << someTokenDef.state() << ", should be 1" << std::endl;
  }

  lex::token_def<> someTokenDef;
};

int main(int argc, char* argv[]) {
  typedef std::string::iterator base_iterator_type;
  typedef lex::lexertl::token<
      base_iterator_type
    , boost::mpl::vector<>
    > token_type;

  typedef lex::lexertl::lexer<token_type> lexer_type;
  typedef tokens<lexer_type> a_lexer;

  a_lexer lexer;

  return 0;
}

The program above outputs:

Arity = 2, should be 0
token_id_ = 0, should be 65536
unique_id_ = 18446744073709551615, should be 0
token_state_ = 18446744073709551615, should be 1

[Vyacheslav Andrejev]

I found a workaround:

    auto lex_def = this->self("SOME_STATE");
    auto expr = (lex_def += someTokenDef);

With this change the output is

Arity = 0, should be 0
token_id_ = 65536, should be 65536
unique_id_ = 0, should be 0
token_state_ = 1, should be 1

I checked the WAR with GCC 4.8 and MSVC 12.0.

Any clue what is going on?

[Vyacheslav Andrejev]

I found out what is going on. An rvalue cannot be bound to a non-const reference. boost::spirit::lex::detail::operator+= has a non-const reference as the first parameter and this->self("SOME_STATE") returns an rvalue. Therefore the compiler doesn't call boost::spirit::lex::detail::operator+=, but uses template <typename Left, typename Right> operator +=(Left &&left, Right &&right) from boost::proto::expns_ instead. A solution could be adding another definition of boost::spirit::lex::detail::operator+=:

        // allow to assign a token definition expression
        template <typename LexerDef, typename Expr>
        inline lexer_def_<LexerDef>&
        operator+= (lexer_def_<LexerDef>&& lexdef, Expr&& xpr)
        {
            // Report invalid expression error as early as possible.
            // If you got an error_invalid_expression error message here,
            // then the expression (expr) is not a valid spirit lex 
            // expression.
            BOOST_SPIRIT_ASSERT_MATCH(lex::domain, Expr);

            lexdef.define(xpr);
            return lexdef;
        }

Do we have some kind of virtual regression system I could test my changes in before forming a pull request?