Invalid regex recursion behavior

[Lucas Trzesniewski <lucas.trzesniewski@…>]

I have found an unexpected behavior regarding recursion in regexes (in Perl mode). I've reduced the issue to the following test pattern:

(?(DEFINE)
    (?<prefix>)
    (?<dummy>x)
)
(?&prefix) unused | (?&prefix) match

If you try to match this against foo match bar, the match word should be found. This works in both Perl and PCRE, see the ​regex101 demo here.

• Removing or commenting out the dummy group causes the pattern to match
• Making the dummy group empty causes the pattern to match
• Replacing (?&prefix) with (?&prefix)? causes Encountered an infinite recursion.
• Replacing (?&prefix) with (?&prefix)? but making the definition of pattern non-empty causes the pattern to match

Here's the full test program:

#include <string>
#include <iostream>
#include <boost/regex.hpp>

static void test()
{
    boost::regex re(R"regex(
        (?(DEFINE)
          (?<prefix>)
          (?<dummy>x)
        )
        (?&prefix) unused | (?&prefix) match
    )regex", boost::regex::perl | boost::regex::no_mod_s | boost::regex::mod_x | boost::regex::optimize);

    std::string subject("foo match bar");

    std::cout << boost::regex_replace(subject, re, "[$&]", boost::format_all) << std::endl;
}

int main(int argc, char **argv)
{
    try
    {
        test();
    }
    catch(std::exception ex)
    {
        std::cerr << ex.what() << std::endl;
    }

    return 0;
}

• Actual output is foo match bar
• Expected output is foo [match] bar

Tested with Boost 1.63.0 on MSVC 2015. The full pattern where the issue appeared can be found ​here.