Boost C++ Libraries: Ticket #10718: program_options: Multitoken option is limited to 32000 token https://svn.boost.org/trac10/ticket/10718 <p> Dear all, </p> <p> I've created an application using boost::program_options to handle command line parameters. Beyond several other parameters, it uses a multitoken parameter to handle input file names passed. The application is basically executed like: </p> <p> ./prog -i *.nii </p> <p> where *.nii resolves to a substantial number of files. </p> <p> The relevant part of my application's code is </p> <pre class="wiki">typedef std::vector&lt;std::string&gt; FileContainer; FileContainer files; namespace po = boost::program_options; //... po::options_description opts("Test"); opts.add_options() ("input,i", po::value&lt;FileContainer&gt;(&amp;files)-&gt;multitoken(), "The input file(s)"); po::variables_map vm; po::store(po::parse_command_line(ac, av, opts), vm); po::notify(vm); </pre><p> So far it works very well. A problem occurs, as soon as the number of input files and therefore the number of tokens exceeds 32000 (which is a regular scenario, as the application is running on a HPC cluster and is working with massive amounts of data). In such cases, the first 32000 file names are parsed, the remaining ones are skipped silently. </p> <p> I searched a little in the program_options code and found the reason for this behavior in </p> <p> boost/program_options/value_semantic.hpp </p> <p> from line 315 </p> <pre class="wiki"> unsigned max_tokens() const { if (m_multitoken) { return 32000; } else if (m_zero_tokens) { return 0; } else { return 1; } } </pre><p> This is the case for all recent boost versions, including the latest from git. </p> <p> I replaced the 32000 with UINT_MAX from &lt;climits&gt; and it seems to work. Now my questions: </p> <ol><li>Is 32000 intentional? UINT_MAX would also create an upper limit, but a much larger one. </li><li>Is there a reason to "fail" silently, when the limit is reached? If not, I'd suggest to add an error/warning or something. </li></ol><p> I assume this is a rare use case and therefor won't cause any problems for the majority of applications and furthermore there are plenty of workaround (including simply not using boost::program_options), but nevertheless... </p> <p> If you need any further details, just let me know. </p> <p> Thank you. </p> <p> best regards, </p> <p> Hans </p> en-us Boost C++ Libraries /htdocs/site/boost.png https://svn.boost.org/trac10/ticket/10718 Trac 1.4.3 Vladimir Prus Thu, 30 Oct 2014 11:07:26 GMT <link>https://svn.boost.org/trac10/ticket/10718#comment:1 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/10718#comment:1</guid> <description> <p> Hans, </p> <p> it looks a dubious code from distant past, and using UINT_MAX shall be the right solution. </p> <p> Would you like to create a pull request (slightly preferred), or you'd rather me fix it myself? </p> </description> <category>Ticket</category> </item> <item> <author>hans.hohenfeld@…</author> <pubDate>Fri, 31 Oct 2014 10:18:37 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/10718#comment:2 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/10718#comment:2</guid> <description> <p> Dear Vladimir, </p> <p> I'll do so during the weekend. Thanks. </p> <p> best regards, </p> <p> Hans </p> </description> <category>Ticket</category> </item> <item> <dc:creator>anonymous</dc:creator> <pubDate>Mon, 24 Nov 2014 14:03:59 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/10718#comment:3 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/10718#comment:3</guid> <description> <p> Dear Vladimir, </p> <p> pull request is on GH for some time now. </p> <p> best regards, </p> <p> Hans </p> </description> <category>Ticket</category> </item> <item> <dc:creator>Vladimir Prus</dc:creator> <pubDate>Mon, 24 Nov 2014 14:14:36 GMT</pubDate> <title>status changed; resolution set https://svn.boost.org/trac10/ticket/10718#comment:4 https://svn.boost.org/trac10/ticket/10718#comment:4 <ul> <li><strong>status</strong> <span class="trac-field-old">new</span> → <span class="trac-field-new">closed</span> </li> <li><strong>resolution</strong> → <span class="trac-field-new">fixed</span> </li> </ul> <p> Hans, </p> <p> my apologies - it seems I did not actually watch the repo :-( Merged now. </p> Ticket