program_options: Multitoken option is limited to 32000 token

[hans.hohenfeld@…]

Dear all,

I've created an application using boost::program_options to handle command line parameters. Beyond several other parameters, it uses a multitoken parameter to handle input file names passed. The application is basically executed like:

./prog -i *.nii

where *.nii resolves to a substantial number of files.

The relevant part of my application's code is

typedef std::vector<std::string> FileContainer;
FileContainer files;
namespace po = boost::program_options;

//...

po::options_description opts("Test");
opts.add_options() ("input,i", po::value<FileContainer>(&files)->multitoken(), 
                    "The input file(s)");

po::variables_map vm;
po::store(po::parse_command_line(ac, av, opts), vm);
po::notify(vm);

So far it works very well. A problem occurs, as soon as the number of input files and therefore the number of tokens exceeds 32000 (which is a regular scenario, as the application is running on a HPC cluster and is working with massive amounts of data). In such cases, the first 32000 file names are parsed, the remaining ones are skipped silently.

I searched a little in the program_options code and found the reason for this behavior in

boost/program_options/value_semantic.hpp

from line 315

       unsigned max_tokens() const {
            if (m_multitoken) {
                return 32000;
            } else if (m_zero_tokens) {
                return 0;
            } else {
                return 1;
            }
        }

This is the case for all recent boost versions, including the latest from git.

I replaced the 32000 with UINT_MAX from <climits> and it seems to work. Now my questions:

1. Is 32000 intentional? UINT_MAX would also create an upper limit, but a much larger one.
2. Is there a reason to "fail" silently, when the limit is reached? If not, I'd suggest to add an error/warning or something.

I assume this is a rare use case and therefor won't cause any problems for the majority of applications and furthermore there are plenty of workaround (including simply not using boost::program_options), but nevertheless...

If you need any further details, just let me know.

Thank you.

best regards,

Hans

[Vladimir Prus]

Hans,

it looks a dubious code from distant past, and using UINT_MAX shall be the right solution.

Would you like to create a pull request (slightly preferred), or you'd rather me fix it myself?

[hans.hohenfeld@…]

Dear Vladimir,

I'll do so during the weekend. Thanks.

best regards,

Hans

[anonymous]

Dear Vladimir,

pull request is on GH for some time now.

best regards,

Hans

[Vladimir Prus]

Hans,

my apologies - it seems I did not actually watch the repo :-( Merged now.