A question for Leigh

[Paul]

Your recently posted tokeniser functionS seem to be a bit buggy.
Please see the following, which seems to exhibit a bug in your code:

/**************code*******************/
#include <iostream>
#include <vector>
#include <string>
#include <algorithm>


template <typename C, typename FwdIter1, typename FwdIter2>
inline FwdIter1 do_tokens(FwdIter1 aFirst, FwdIter1 aLast, FwdIter2
aDelimeterFirst, FwdIter2 aDelimiterLast, C& aTokens, std::size_t
aMaxTokens = 0, bool aSkipEmptyTokens = true, bool
aDelimeterIsSubsequence = false)
{
if (aFirst >= aLast)
return aFirst;

typedef typename C::value_type value_type;

FwdIter1 b = aFirst;
FwdIter1 e = aDelimeterIsSubsequence ? std::search(b, aLast,
aDelimeterFirst, aDelimiterLast) : std::find_first_of(b, aLast,
aDelimeterFirst, aDelimiterLast);
std::size_t tokens = 0;
while(e != aLast && (aMaxTokens == 0 || tokens < aMaxTokens))
{
if (b == e && !aSkipEmptyTokens)
{
aTokens.push_back(value_type(b, b));
++tokens;
}
else if (b != e)
{
aTokens.push_back(value_type(b, e));
++tokens;
}
b = e;
for (std::size_t i = aDelimeterIsSubsequence ?
std::distance(aDelimeterFirst, aDelimiterLast) : 1; i > 0; --i)
++b;
e = aDelimeterIsSubsequence ? std::search(b, aLast, aDelimeterFirst,
aDelimiterLast) : std::find_first_of(b, aLast, aDelimeterFirst,
aDelimiterLast);
}
if (b != e && (aMaxTokens == 0 || tokens < aMaxTokens))
{
aTokens.push_back(value_type(b, e));
b = e;
}
return b;
}
template <typename C, typename FwdIter1, typename FwdIter2>
inline FwdIter1 tokens(FwdIter1 aFirst, FwdIter1 aLast, FwdIter2
aDelimeterFirst, FwdIter2 aDelimiterLast, C& aTokens, std::size_t
aMaxTokens = 0, bool aSkipEmptyTokens = true, bool
aDelimeterIsSubsequence = false)
{
return do_tokens(aFirst, aLast, aDelimeterFirst, aDelimiterLast,
aTokens, aMaxTokens, aSkipEmptyTokens, aDelimeterIsSubsequence);
}
template <typename CharT, typename Traits, typename Alloc, typename C>
inline void tokens(const std::basic_string<CharT, Traits, Alloc>&
aLine, const std::basic_string<CharT, Traits, Alloc>& aDelimeter, C&
aTokens, std::size_t aMaxTokens = 0, bool aSkipEmptyTokens = true, bool
aDelimeterIsSubsequence = false)
{
do_tokens(aLine.begin(), aLine.end(), aDelimeter.begin(),
aDelimeter.end(), aTokens, aMaxTokens, aSkipEmptyTokens,
aDelimeterIsSubsequence);
}


int main(){
typedef std::vector<std::string> words_t;
words_t words;
char p[] ="abcd\0xyz\0mnop\0vvv";
std::string source(p, 18);
std::string delim = "\0";
tokens(source, delim, words);
for (words_t::const_iterator i = words.begin(); i != words.end(); ++i)
std::cout << "[" << *i << "] ";
}
/**********end code*******************/


Maybe my compiler is buggy, because i have not traced your code to bother
finding the exact problem, I simply compiled on command line but I find
unexpected output which suggests your tokeniser functions may be
problematic. Perhaps you could add another function or two to sort out this
problem.

I suggest you start again from scratch because there seems to be something
noobish about your use of templates.
One example is your use template typenames such as "FwdIter1" when a char*
is excepted.
Perhaps you wrote this code along time ago , and I have taught you alot
since then , therefore you will be in a better position now to rewrite this
code so that it does not contain any bugs, inappropriate template names,
bloatation or lack of CPU efficiency..

HTH & GL.

 

[Paul]

Your recently posted tokeniser functionS seem to be a bit buggy.
Please see the following, which seems to exhibit a bug in your code:

C & P error . Please replace previous main with the following:

int main(){
typedef std::vector<std::string> words_t;
words_t words;
wchar_t p[4] = {'a', '\0', '\0', 'b'};
std::wstring source(p, 4);
std::string delim = "\0";
tokens(source.begin(),source.end(), delim.begin(), delim.end(), words);
for (words_t::const_iterator i = words.begin(); i != words.end(); ++i)
std::cout << "[" << *i << "] ";
}

 

[Paul]

Ok this one breaks it:

int main(){
typedef std::vector<std::string> words_t;
words_t words;
char c[4] = {'\0', '\0', '\0', 'b'};
std::string source(c, 4);
wchar_t wc = '\0';
std::wstring delim(wc,1);
tokens(source.begin(),source.end(), delim.begin(), delim.end(), words);
for (words_t::const_iterator i = words.begin(); i != words.end(); ++i)
std::cout << "[" << *i << "] ";
}


Your templates aint typesafe. I knew it was breakable as soon as I seen it.