Implement operator[] for Tokenizer

[nobody]

I constantly find myself creating this structure:

tokenizer tok(str, sep);
vector<string> s;
for(tokenizer::iterator iter=tok.begin();
iter!=tok.end(); ++iter) {
  s.push_back(*iter);
}

Why not just:
tok[0];

to retrieve the first token?

(Darel Finkbeiner <darel.finkbeiner@sbcglobal.net>)

[Steven Watanabe]

The correct solution to this is

tokenizer tok(str, sep);
vector<string> s(tok.begin(), tok.end());

tokenizer cannot support random access efficiently because it is lazy and only has forward iterators.