Help with Tokenizer Please

[electrixnow]

I use a code snippet called tokenize that works quite well. But I need
to modify it so it returns an empty string if two commas are found
together: ie. string test="1,2,,4,5"

I would like to have

tokens[0] = "1"
tokens[1] = "2"
tokens[3] = ""
tokens[4] = "4"
tokens[5] = "5"

This would allow me to find missing tokens. The following is the code I
use.
Can anyone show me the modification I would need?
-----------------

void Tokenize(const string& str,
vector<string>& tokens,
const string& delimiters = " ,\t\n")
{
string::size_type lastPos = str.find_first_not_of(delimiters, 0);
// Skip delimiters at beginning.
string::size_type pos = str.find_first_of(delimiters, lastPos);
// Find first "non-delimiter".
while (string::npos != pos || string::npos != lastPos)
{
tokens.push_back(str.substr(lastPos, pos - lastPos));
// Found a token, add it to the vector.
lastPos = str.find_first_not_of(delimiters, pos);
// Skip delimiters. Note the "not_of"
pos = str.find_first_of(delimiters, lastPos);
// Find next "non-delimiter"
}
}

 

[Victor Bazarov]

I use a code snippet called tokenize that works quite well. But I need
to modify it so it returns an empty string if two commas are found
together: ie. string test="1,2,,4,5"

I would like to have

tokens[0] = "1"
tokens[1] = "2"
tokens[3] = ""
tokens[4] = "4"
tokens[5] = "5"

This would allow me to find missing tokens. The following is the code
I use.
Can anyone show me the modification I would need?
-----------------

void Tokenize(const string& str,
vector<string>& tokens,
const string& delimiters = " ,\t\n")
{
string::size_type lastPos = str.find_first_not_of(delimiters, 0);
// Skip delimiters at beginning.
string::size_type pos = str.find_first_of(delimiters, lastPos);
// Find first "non-delimiter".
while (string::npos != pos || string::npos != lastPos)
{
tokens.push_back(str.substr(lastPos, pos - lastPos));
// Found a token, add it to the vector.
lastPos = str.find_first_not_of(delimiters, pos);
// Skip delimiters. Note the "not_of"

I think you shouldn't skip delimiters here. You just need to make
'lastPos' to be the same as 'pos'. Replace the statement above with

lastPos = pos;

pos = str.find_first_of(delimiters, lastPos);
// Find next "non-delimiter"
}
}

--------------

I didn't check it. It's just a hunch.

V