Boost C++ Libraries: Ticket #9580: [spirit.lex] self("state") += calls a wrong operator +=

owner changed

Joel de Guzman — Thu, 16 Jan 2014 00:39:13 GMT

owner changed from Joel de Guzman to Hartmut Kaiser

Vyacheslav Andrejev — Thu, 16 Jan 2014 07:58:09 GMT

An example that demonstrates the problem:

#include <iostream>
#include <boost/spirit/include/lex_lexertl.hpp>
using namespace boost::spirit;
template <typename Lexer>
struct tokens : public lex::lexer<Lexer>
{
  tokens()
  {
    someTokenDef = '"';
    auto expr = (this->self("SOME_STATE") += someTokenDef);
    std::cout << "Arity = " << expr.proto_arity_c << ", should be 0" << std::endl;
    std::cout << "token_id_ = " << someTokenDef.id() << ", should be 65536" << std::endl;
    std::cout << "unique_id_ = " << someTokenDef.unique_id() << ", should be 0" << std::endl;
    std::cout << "token_state_ = " << someTokenDef.state() << ", should be 1" << std::endl;
  }
  lex::token_def<> someTokenDef;
};
int main(int argc, char* argv[]) {
  typedef std::string::iterator base_iterator_type;
  typedef lex::lexertl::token<
      base_iterator_type
    , boost::mpl::vector<>
    > token_type;
  typedef lex::lexertl::lexer<token_type> lexer_type;
  typedef tokens<lexer_type> a_lexer;
  a_lexer lexer;
  return 0;
}

The program above outputs:

Arity = 2, should be 0
token_id_ = 0, should be 65536
unique_id_ = 18446744073709551615, should be 0
token_state_ = 18446744073709551615, should be 1

Vyacheslav Andrejev — Wed, 29 Jan 2014 08:20:13 GMT

I found a workaround:

    auto lex_def = this->self("SOME_STATE");
    auto expr = (lex_def += someTokenDef);

With this change the output is

Arity = 0, should be 0
token_id_ = 65536, should be 65536
unique_id_ = 0, should be 0
token_state_ = 1, should be 1

I checked the WAR with GCC 4.8 and MSVC 12.0.

Any clue what is going on?

Vyacheslav Andrejev — Thu, 30 Jan 2014 15:52:29 GMT

I found out what is going on. An rvalue cannot be bound to a non-const reference. boost::spirit::lex::detail::operator+= has a non-const reference as the first parameter and this->self("SOME_STATE") returns an rvalue. Therefore the compiler doesn't call boost::spirit::lex::detail::operator+=, but uses template <typename Left, typename Right> operator +=(Left &&left, Right &&right) from boost::proto::expns_ instead. A solution could be adding another definition of boost::spirit::lex::detail::operator+=:

        // allow to assign a token definition expression
        template <typename LexerDef, typename Expr>
        inline lexer_def_<LexerDef>&
        operator+= (lexer_def_<LexerDef>&& lexdef, Expr&& xpr)
        {
            // Report invalid expression error as early as possible.
            // If you got an error_invalid_expression error message here,
            // then the expression (expr) is not a valid spirit lex 
            // expression.
            BOOST_SPIRIT_ASSERT_MATCH(lex::domain, Expr);
            lexdef.define(xpr);
            return lexdef;
        }

Do we have some kind of virtual regression system I could test my changes in before forming a pull request?