boost::wave produces non-existent tokens

[Vyacheslav Andrejev]

Consider the following program:

#include <iostream>
#include <string>

#include <boost/wave.hpp>
#include <boost/wave/cpplexer/cpp_lex_iterator.hpp>

using namespace boost::wave::cpplexer;

int main(int argc, char* argv[])
{
    std::string str = "int a\n\n;\n";

    typedef lex_iterator<lex_token<> > LexIterator;
    typedef boost::wave::context<std::string::const_iterator, LexIterator> WaveContext;

    WaveContext ctx(str.cbegin(), str.cend());
    for (auto tok : ctx)
    {
        std::cout << boost::wave::get_token_name(boost::wave::token_id(tok)) << '('
                  << tok.get_value() << ')' << std::endl;
    }

    return 0;
}

It produces the following output:

INT(int)
SPACE( )
IDENTIFIER(a)
NEWLINE(
)
PP_LINE(#line)
SPACE( )
INTLIT(3)
SPACE( )
STRINGLIT("<Unknown>")
NEWLINE(
)
SEMICOLON(;)
NEWLINE(
)
EOF()

There are no PP_LINE(#line), INTLIT(3) and STRINGLIT("<Unknown>") in the input.

[Hartmut Kaiser]

This is on purpose as it reflects the token stream which would be generated by a preprocessor. If you don't want to see the #line directives generated, add:

      WaveContext.set_language(
          boost::wave::enable_emit_line_directives(
              ctx.get_language(), 
              false));

before iterating over the input (see here for more information: http://www.boost.org/doc/libs/1_58_0/libs/wave/doc/class_reference_context.html).

[Vyacheslav Andrejev]

Thanks!

[Vyacheslav Andrejev]

Actually enable_emit_line_directives has undesired effect: the lexer not only stops producing not existing #line, but also removes existing. For example

#include <iostream>
#include <string>

#include <boost/wave.hpp>
#include <boost/wave/cpplexer/cpp_lex_iterator.hpp>

using namespace boost::wave;
using namespace boost::wave::cpplexer;

int main(int argc, char* argv[])
{
    std::string str = "int a;\n#line 10 \"file.cpp\"\nint b;\n";

    typedef lex_iterator<lex_token<> > LexIterator;
    typedef boost::wave::context<std::string::const_iterator, LexIterator> WaveContext;

    WaveContext ctx(str.cbegin(), str.cend());
    ctx.set_language(
        boost::wave::enable_emit_line_directives(
        ctx.get_language(),
        false));
    for (auto tok : ctx)
    {
        std::cout << get_token_name(token_id(tok)) << '('
                  << tok.get_value() << ')' << std::endl;
    }

    return 0;
}

The output:

INT(int)
SPACE( )
IDENTIFIER(a)
SEMICOLON(;)
NEWLINE(
)
INT(int)
SPACE( )
IDENTIFIER(b)
SEMICOLON(;)
NEWLINE(
)
EOF()

Is there a way to leave just existing #line?

[Hartmut Kaiser]

Wave always consumes any #line directives in the input to correctly assign line numbers to the generated tokens (like it consumes all other preprocessor directives). It outputs those #line directives based on its own heuristics. There is currently no way to retain those from the input.

[Vyacheslav Andrejev]

It produces incorrect line numbers if a file already has #line directives.