boost::tokenize fails on Ubuntu8.10 64bit

[grumbel@…]

The code in the attached file fails to report the proper tokens on Ubuntu8.10 64bit, while it works fine in the 32bit edition:

The output on 64bit is:

$ ./token hallo:du:da Token: 'hallo' Token: 'o' Token: 'da'

On 32bit it is as expected:

$ ./token hallo:du:da Token: 'hallo' Token: 'du' Token: 'da'

[grumbel@…]

Tokenizer Example

[Steven Watanabe]

tokenizer does not store a copy of the string, it stores a pair of iterators. The temporary string that you are passing to the constructor is immediately destroyed, invalidating the iterators.

This works:

      std::string arg(argv[1]);
      tokenizer tokens(arg, sep);

[william.l.blevins@…]

How is this ticket invalid?

BoostTokenizer provides a constructor that allows for:

1. memory corruption
2. tokens are arbitrarily parsed incorrectly

// This should be valid
tokenizer tokens(std::string(argv[1]), sep);

There should at least be documentation for a precondition {@precondition} where this case is listed explicitly.