Boost C++ Libraries: Ticket #7090: Provide an option to parse only preprocessor directives https://svn.boost.org/trac10/ticket/7090 <p> I use boost::wave to preprocess shader source files. I have a macro system that allows me to execute certain things if the code path the macro is in "survives" the preprocessing step. So far, so good. </p> <p> My concern is the speed. One source file with about 100 lines takes about 5 MS to parse. From what I read in the docs, boost::wave uses a cpp-lexer to parse _all_ code-tokens. But for my application this is not necessary. What I need is _not_ a list of all tokens (that I can assemble into a result string), I want only the result string. Is there a way to turn this off? I have looked through all samples and tests but did not find a way to parse only preprocessor directives and ignore the rest of the file. </p> <p> I have also read on this bugtracker that #define BOOST_WAVE_SUPPORT_THREADING 0 is supposed to make it much faster. However this is not an option for me because I can't ship the boost libraries, I expect the users to have them installed already. </p> en-us Boost C++ Libraries /htdocs/site/boost.png https://svn.boost.org/trac10/ticket/7090 Trac 1.4.3 anonymous Thu, 05 Jul 2012 18:38:28 GMT <link>https://svn.boost.org/trac10/ticket/7090#comment:1 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/7090#comment:1</guid> <description> <p> 5 MS does not sound like much, but if I do that for each shader permutation (there are usually hundreds of permutations for one uber-shader), it ends up with a loading time of a few seconds. This is ridiculous, considering that the _actual_ compiling on the GPU takes only about 0.05 MS per shader (as opposed to 5 MS for preprocessing). </p> </description> <category>Ticket</category> </item> <item> <author>scrawl123@…</author> <pubDate>Thu, 05 Jul 2012 18:44:13 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/7090#comment:2 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/7090#comment:2</guid> <description> <p> If this is not possible, I would appreciate a suggestion for an alternate library that is faster, or some tips how to make my own (is it possible to reuse code from boost::wave for that?) - what I need is only the preprocessor, I want the actual code tokens unchanged, so it should be possible to make a much faster library. </p> </description> <category>Ticket</category> </item> <item> <dc:creator>Hartmut Kaiser</dc:creator> <pubDate>Thu, 05 Jul 2012 18:49:45 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/7090#comment:3 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/7090#comment:3</guid> <description> <p> Replying to <a class="closed ticket" href="https://svn.boost.org/trac10/ticket/7090" title="#7090: Feature Requests: Provide an option to parse only preprocessor directives (closed: wontfix)">scrawl123@…</a>: </p> <blockquote class="citation"> <p> My concern is the speed. One source file with about 100 lines takes about 5 MS to parse. From what I read in the docs, boost::wave uses a cpp-lexer to parse _all_ code-tokens. But for my application this is not necessary. What I need is _not_ a list of all tokens (that I can assemble into a result string), I want only the result string. Is there a way to turn this off? I have looked through all samples and tests but did not find a way to parse only preprocessor directives and ignore the rest of the file. </p> </blockquote> <p> That's by design, so there is no way to turn it off. </p> <blockquote class="citation"> <p> I have also read on this bugtracker that #define BOOST_WAVE_SUPPORT_THREADING 0 is supposed to make it much faster. However this is not an option for me because I can't ship the boost libraries, I expect the users to have them installed already. </p> </blockquote> <p> The Wave libraries shipped with Boost are generated from small cpp files which essentially just explicitly instantiate a couple of templates (see $BOOST_ROOT/libs/wave/src). You could easily add those 5 or 6 cpp files to your projects and be independent from the Boost.Wave binaries a user might have installed. </p> </description> <category>Ticket</category> </item> <item> <dc:creator>Hartmut Kaiser</dc:creator> <pubDate>Thu, 05 Jul 2012 18:52:51 GMT</pubDate> <title>status changed; resolution set https://svn.boost.org/trac10/ticket/7090#comment:4 https://svn.boost.org/trac10/ticket/7090#comment:4 <ul> <li><strong>status</strong> <span class="trac-field-old">new</span> → <span class="trac-field-new">closed</span> </li> <li><strong>resolution</strong> → <span class="trac-field-new">wontfix</span> </li> </ul> <p> Replying to <a class="ticket" href="https://svn.boost.org/trac10/ticket/7090#comment:2" title="Comment 2">scrawl123@…</a>: </p> <blockquote class="citation"> <p> If this is not possible, I would appreciate a suggestion for an alternate library that is faster, or some tips how to make my own (is it possible to reuse code from boost::wave for that?) - what I need is only the preprocessor, I want the actual code tokens unchanged, so it should be possible to make a much faster library. </p> </blockquote> <p> I'm not aware of any library solution you could reuse, but there are definitely some open source preprocessors around you could have a look at. As a first step I'd disable threading in Wave as you outlined above and see how fast it will get after that change. </p> Ticket scrawl <scrawl123@…> Fri, 06 Jul 2012 12:50:16 GMT <link>https://svn.boost.org/trac10/ticket/7090#comment:5 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/7090#comment:5</guid> <description> <p> Thanks for your answer, I will try disable threading tomorrow. </p> <blockquote class="citation"> <p> I'm not aware of any library solution you could reuse, but there are definitely some open source preprocessors around you could have a look at. </p> </blockquote> <p> Can you give me some hints? The only other one that I've found is libcpp (used by gcc), but it's written in C and I can't find any examples how to use it, reading the header doesn't explain anything either. </p> </description> <category>Ticket</category> </item> <item> <dc:creator>Hartmut Kaiser</dc:creator> <pubDate>Fri, 06 Jul 2012 15:48:15 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/7090#comment:6 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/7090#comment:6</guid> <description> <p> Replying to <a class="ticket" href="https://svn.boost.org/trac10/ticket/7090#comment:5" title="Comment 5">scrawl &lt;scrawl123@…&gt;</a>: </p> <blockquote class="citation"> <blockquote class="citation"> <p> I'm not aware of any library solution you could reuse, but there are definitely some open source preprocessors around you could have a look at. </p> </blockquote> <p> Can you give me some hints? The only other one that I've found is libcpp (used by gcc), but it's written in C and I can't find any examples how to use it, reading the header doesn't explain anything either. </p> </blockquote> <p> Well, there is clang, which has a separate preprocessor compiler pass and there is mcpp (<a class="ext-link" href="http://mcpp.sourceforge.net/"><span class="icon">​</span>http://mcpp.sourceforge.net/</a>). Other than that, there is not much I know about. However, you might look at the various open sourced C/C++ compilers available. </p> </description> <category>Ticket</category> </item> <item> <author>scrawl <scrawl123@…></author> <pubDate>Fri, 06 Jul 2012 21:42:07 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/7090#comment:7 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/7090#comment:7</guid> <description> <p> mcpp is 10-15x faster than boost::wave, even though I had to copy my source string into a file first because it apparently only takes file input and no string from memory. </p> <p> Here is quick guide, maybe it will help someone </p> <pre class="wiki"> - build mcpp-2.7.2/src as static lib with -DMCPP_LIB - #include "../mcpp/mcpp_lib.h" - add this code, in this example my method gets passed a std::vector&lt;std::string&gt; of definitions and an std::string include path mcpp_use_mem_buffers(1); int num_args = 4 + definitions.size()*2; char* args[num_args]; args[0] = "mcpp"; args[1] = "/tmp/test.shader"; // file you want to process args[2] = "-I"; std::vector&lt;char&gt; writable(includePath.size()+1); std::copy(includePath.begin(), includePath.end(), writable.begin()); char* include = &amp;writable[0]; args[3] = include; std::vector&lt;char&gt; vectors[definitions.size()]; int i=4; int cur=0; for (std::vector&lt;std::string&gt;::iterator it = definitions.begin(); it != definitions.end(); ++it) { args[i] = "-D"; ++i; std::string val = *it; std::vector&lt;char&gt; writable2(val.size()+1); std::copy(val.begin(), val.end(), writable2.begin()); vectors[cur] = writable2; args[i] = &amp;vectors[cur][0]; ++cur; ++i; } mcpp_lib_main(num_args, args); char* result = mcpp_get_mem_buffer (OUT); </pre> </description> <category>Ticket</category> </item> <item> <dc:creator>Hartmut Kaiser</dc:creator> <pubDate>Fri, 06 Jul 2012 22:19:11 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/7090#comment:8 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/7090#comment:8</guid> <description> <p> Replying to <a class="ticket" href="https://svn.boost.org/trac10/ticket/7090#comment:7" title="Comment 7">scrawl &lt;scrawl123@…&gt;</a>: </p> <blockquote class="citation"> <p> mcpp is 10-15x faster than boost::wave, even though I had to copy my source string into a file first because it apparently only takes file input and no string from memory. </p> </blockquote> <p> FWIW, using Wave with threading disabled should make the times for both libraries comparable (while Wave being probably by a factor of 2 slower, but that's because of Wave doing much more stuff in the background). The reason for this dramatic difference is that Wave uses Spirit V1, which is utterly inefficient when threading is enabled. </p> </description> <category>Ticket</category> </item> <item> <author>scrawl <scrawl123@…></author> <pubDate>Sat, 21 Jul 2012 13:56:57 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/7090#comment:9 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/7090#comment:9</guid> <description> <p> Discovered an issue with mcpp, the #line directives are not GLSL compliant; then i looked at glcpp from mesa, but there it doesn't emit #line directives at all. I don't feel like hacking with a horrible bulk of C-code, so I guess I'm back to using boost::wave. </p> <p> From what I can tell, in order to disable threading, I should add the files libs/wave/*.cpp to my project, and modify the includes to use my own wave_config.hpp instead of the one from the boost sources? Is that correct? </p> </description> <category>Ticket</category> </item> <item> <dc:creator>Hartmut Kaiser</dc:creator> <pubDate>Sat, 21 Jul 2012 15:00:48 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/7090#comment:10 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/7090#comment:10</guid> <description> <p> Replying to <a class="ticket" href="https://svn.boost.org/trac10/ticket/7090#comment:9" title="Comment 9">scrawl &lt;scrawl123@…&gt;</a>: </p> <blockquote class="citation"> <p> Discovered an issue with mcpp, the #line directives are not GLSL compliant; then i looked at glcpp from mesa, but there it doesn't emit #line directives at all. I don't feel like hacking with a horrible bulk of C-code, so I guess I'm back to using boost::wave. </p> </blockquote> <p> Cool. </p> <blockquote class="citation"> <p> From what I can tell, in order to disable threading, I should add the files libs/wave/*.cpp to my project, </p> </blockquote> <p> Yes. </p> <blockquote class="citation"> <p> and modify the includes to use my own wave_config.hpp instead of the one from the boost sources? Is that correct? </p> </blockquote> <p> There is no need to modify the sources. If you look at <code>wave_config.hpp</code> you'll see that all configuration constants (such as threading) are handled as </p> <pre class="wiki">#if !defined(BOOST_WAVE_SUPPORT_THREADING) #if defined(BOOST_HAS_THREADS) #define BOOST_WAVE_SUPPORT_THREADING 1 #else #define BOOST_WAVE_SUPPORT_THREADING 0 #endif #endif </pre><p> Thus all you have to do is to pass <code>-DBOOST_WAVE_SUPPORT_THREADING=0</code> on the commandline while compiling. </p> </description> <category>Ticket</category> </item> <item> <author>scrawl <scrawl123@…></author> <pubDate>Sat, 21 Jul 2012 22:43:17 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/7090#comment:11 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/7090#comment:11</guid> <description> <p> Thank you, it worked, and integrating the custom_line_directives sample with my code was surprisingly easy :) </p> </description> <category>Ticket</category> </item> </channel> </rss>