id,summary,reporter,owner,description,type,status,milestone,component,version,severity,resolution,keywords,cc
10718,program_options: Multitoken option is limited to 32000 token,hans.hohenfeld@…,Vladimir Prus,"Dear all,

I've created an application using boost::program_options to handle command line parameters. Beyond several other parameters, it uses a multitoken parameter to handle input file names passed. The application is basically executed like:

./prog -i *.nii

where *.nii resolves to a substantial number of files.

The relevant part of my application's code is

{{{
typedef std::vector<std::string> FileContainer;
FileContainer files;
namespace po = boost::program_options;

//...

po::options_description opts(""Test"");
opts.add_options() (""input,i"", po::value<FileContainer>(&files)->multitoken(), 
                    ""The input file(s)"");

po::variables_map vm;
po::store(po::parse_command_line(ac, av, opts), vm);
po::notify(vm);
}}}

So far it works very well. A problem occurs, as soon as the number of input files and therefore the number of tokens exceeds 32000 (which is a regular scenario, as the application is running on a HPC cluster and is working with massive amounts of data). In such cases, the first 32000 file names are parsed, the remaining ones are skipped silently.

I searched a little in the program_options code and found the reason for this behavior in

boost/program_options/value_semantic.hpp

from line 315

{{{       
       unsigned max_tokens() const {
            if (m_multitoken) {
                return 32000;
            } else if (m_zero_tokens) {
                return 0;
            } else {
                return 1;
            }
        }
}}}

This is the case for all recent boost versions, including the latest from git.

I replaced the 32000 with UINT_MAX from <climits> and it seems to work. Now my questions:

1. Is 32000 intentional? UINT_MAX would also create an upper limit, but a much larger one.
2. Is there a reason to ""fail"" silently, when the limit is reached? If not, I'd suggest to add an error/warning or something.

I assume this is a rare use case and therefor won't cause any problems for the majority of applications and furthermore there are plenty of workaround (including simply not using boost::program_options), but nevertheless...

If you need any further details, just let me know.

Thank you.

best regards,

Hans",Bugs,closed,To Be Determined,program_options,Boost 1.55.0,Problem,fixed,,