Trying to tokenize a string

Discussion in 'C++' started by Lans, Jul 8, 2003.

  1. Lans

    Lans Guest

    I have a string that I need to tokenize but I need to use a string
    token
    see example i am trying the following but strtok only uses characters
    as delimiters and I need to seperate bu a certain word

    char *mystring "Jane and Peter and Tom and Cindy"
    char *delim = " and ";
    char *token;

    token = strtok(mystring, delim);

    while (token !=NULL )
    {
    //do some work
    cout << token << endl;
    token = strtok(NULL,delim)
    }

    my output would return JePererTomCiy
    I need my output to be Jane Peter Tome Cindy

    How can I accomplish this output

    thanks!
    Lans, Jul 8, 2003
    #1
    1. Advertising

  2. "Lans" <> wrote in message
    news:...
    > I have a string that I need to tokenize but I need to use a string
    > token
    > see example i am trying the following but strtok only uses characters
    > as delimiters and I need to seperate bu a certain word
    >
    > char *mystring "Jane and Peter and Tom and Cindy"
    > char *delim = " and ";
    > char *token;
    >
    > token = strtok(mystring, delim);
    >
    > while (token !=NULL )
    > {
    > //do some work
    > cout << token << endl;
    > token = strtok(NULL,delim)
    > }
    >
    > my output would return JePererTomCiy
    > I need my output to be Jane Peter Tome Cindy
    >



    JePererTomCiy

    That output doesn't make any sense, unless you mis-typed it... <?>


    Jeremy
    Jeremy Cowles, Jul 8, 2003
    #2
    1. Advertising

  3. Lans

    Default User Guest

    Lans wrote:
    >
    > I have a string that I need to tokenize but I need to use a string
    > token
    > see example i am trying the following but strtok only uses characters
    > as delimiters and I need to seperate bu a certain word



    If you are dead set on using C-style strings instead of C++ std::string
    class, then the best way is probably to use strstr(). Here's an example,
    written almost completely in C (except for the bool). Note that no
    precaution for overrun of buf[] is taken, it's done by inspection for
    this problem.


    #include <stdio.h>
    #include <string.h>


    int main()
    {
    char *mystring = "Jane and Peter and Tom and Cindy";
    char *delim = " and ";
    char *head;
    char *tail;
    char buf[80];
    bool flag = true;

    size_t len = strlen (delim);

    head = mystring;

    while (flag)
    {
    buf[0] = 0;

    if ((tail = strstr (head, delim)) == 0)
    {
    strcpy (buf, head);
    flag = false;
    }
    else
    {
    strncat (buf, head, tail-head);
    head = tail + len;
    }

    puts (buf);
    }

    return 0;
    }



    Result:

    Jane
    Peter
    Tom
    Cindy



    I don't recommend doing it this way, of course.




    Brian Rodenborn
    Default User, Jul 8, 2003
    #3
  4. Lans

    sw Guest

    It looks as if you misunderstood what the <<char*delim>> does
    You tokenized the string with delimiting chars ' ','a','n','d',' '
    to something like "J\n" "e\n" "Pe\n" later you put them into std::cout
    without spaces.
    Delimiters are single characters, not strings.

    try something like this :

    char mystring[] = "Jane and Peter and Tom and Cindy";
    char *delim = " "; // only blank
    char *token;

    token = strtok(mystring, delim);

    while (token !=NULL)
    {
    //compare ... (strcmp (token,"and") )
    //continue loop if equal
    cout << token << endl;
    token = strtok(NULL,delim)
    }




    "Lans" <> wrote in message
    news:...
    > I have a string that I need to tokenize but I need to use a string
    > token
    > see example i am trying the following but strtok only uses characters
    > as delimiters and I need to seperate bu a certain word
    >
    > char *mystring "Jane and Peter and Tom and Cindy"
    > char *delim = " and ";
    > char *token;
    >
    > token = strtok(mystring, delim);
    >
    > while (token !=NULL )
    > {
    > //do some work
    > cout << token << endl;
    > token = strtok(NULL,delim)
    > }
    >
    > my output would return JePererTomCiy
    > I need my output to be Jane Peter Tome Cindy
    >
    > How can I accomplish this output
    >
    > thanks!
    sw, Jul 8, 2003
    #4
  5. Lans

    Mike Wahler Guest

    Lans <> wrote in message
    news:...
    > I have a string that I need to tokenize but I need to use a string
    > token
    > see example i am trying the following but strtok only uses characters
    > as delimiters and I need to seperate bu a certain word
    >
    > char *mystring "Jane and Peter and Tom and Cindy"
    > char *delim = " and ";
    > char *token;
    >
    > token = strtok(mystring, delim);
    >
    > while (token !=NULL )
    > {
    > file://do some work
    > cout << token << endl;
    > token = strtok(NULL,delim)
    > }
    >
    > my output would return JePererTomCiy
    > I need my output to be Jane Peter Tome Cindy
    >
    > How can I accomplish this output


    First, I'd use a 'std::string' object instead of
    an array of characters.

    The code below accomodates either a character array
    or a std::string.


    #include <algorithm>
    #include <iostream>
    #include <iterator>
    #include <sstream>
    #include <string>

    typedef std::istream_iterator<std::string> istrit;
    typedef std::eek:stream_iterator<std::string> ostrit;

    std::string xfrm(const std::string& s)
    {
    return s == "and" ? "" : s + ' ';
    }

    /* void replace_delims(std::string& s) */
    /* -- modifies argument 's' as follows: */
    /* */
    /* - Removes all occurrences of the string "and" which are */
    /* delimited by whitespace and/or end-of-string */
    /* */
    /* - Replaces consecutive whitespace characters with a */
    /* single space character */
    /* */
    /* - Removes leading and trailing whitespace */
    void replace_delims(std::string& s)
    {
    std::eek:stringstream oss;

    std::transform(istrit(std::istringstream(s)), istrit(),
    ostrit(oss), xfrm);

    const std::string& ref = oss.str();
    s = ref.substr(0, ref.size() - !ref.empty());
    }

    /* void replace_delims(char* s) */
    /* -- Same as 'void replace_delims(std::string&)', */
    /* but operates on a 'C-style string' */
    void replace_delims(char *s)
    {
    std::string result(s);
    std::string::size_type sz(result.size());
    replace_delims(result);
    std::copy(result.begin(), result.begin() + sz, s);
    s[sz] = 0;
    }

    int main()
    {
    char mystring[] = "Jane and Peter and Tom and Cindy";
    std::cout << "Before:\n" << '#' << mystring << '#' << "\n\n";
    replace_delims(mystring);
    std::cout << "After: \n" << '#' << mystring << '#' << "\n\n";
    return 0;
    }


    Output:

    Before:
    #Jane and Peter and Tom and Cindy#

    After:
    #Jane Peter Tom Cindy#

    -Mike
    Mike Wahler, Jul 9, 2003
    #5
  6. Lans

    Default User Guest

    lredmond wrote:
    >
    > Can you give me a C++ example.


    Don't top-post.

    The solution I gave you WAS C++, it's just adapted from a C program I
    wrote. If you are going to use char * types, that's all you need. If you
    are going to use std::string, then there are other solutions. Read up on
    them, take a stab, post your code.




    Brian Rodenborn
    Default User, Jul 9, 2003
    #6
  7. Lans

    lredmond Guest

    Here is what I ended up doing using std::string

    string line = "Tom and Peter and Jane and Joe and Bill";
    string newline;

    cout << "Before: " << line << endl;
    std::string::size_type pos = 0;
    std::string delim = " and ";
    std::string newdelim = "\n";

    while (( pos = line.find(delim,pos ) ) != std::string::npos )
    {
    line.replace(pos, delim.length(),newdelim);
    pos +=newdelim.length();
    }
    cout << "After: " << line << endl;
    --------------------------------------------------------------



    "Default User" <> wrote in message
    news:...
    >
    >
    > lredmond wrote:
    > >
    > > Can you give me a C++ example.

    >
    > Don't top-post.
    >
    > The solution I gave you WAS C++, it's just adapted from a C program I
    > wrote. If you are going to use char * types, that's all you need. If you
    > are going to use std::string, then there are other solutions. Read up on
    > them, take a stab, post your code.
    >
    >
    >
    >
    > Brian Rodenborn
    lredmond, Jul 10, 2003
    #7
  8. "lredmond" <> wrote in message
    news:ph6Pa.21803$...
    > Here is what I ended up doing using std::string
    >
    > string line = "Tom and Peter and Jane and Joe and Bill";
    > string newline;
    >
    > cout << "Before: " << line << endl;
    > std::string::size_type pos = 0;
    > std::string delim = " and ";
    > std::string newdelim = "\n";
    >
    > while (( pos = line.find(delim,pos ) ) != std::string::npos )
    > {
    > line.replace(pos, delim.length(),newdelim);
    > pos +=newdelim.length();
    > }
    > cout << "After: " << line << endl;
    > --------------------------------------------------------------
    >


    Its not the most efficient since you repeatedly hack the same string. It
    probably better to build up your output string as a completely seperate
    string, copying over everything from the original string except the
    delimiters.

    john
    John Harrison, Jul 10, 2003
    #8
  9. Lans

    Simon Turner Guest

    "Jeremy Cowles" <> wrote in message news:<WiFOa.47079$>...
    > "Lans" <> wrote in message
    > news:...
    > > I have a string that I need to tokenize but I need to use a string
    > > token
    > > see example i am trying the following but strtok only uses characters
    > > as delimiters and I need to seperate bu a certain word
    > >
    > > char *mystring "Jane and Peter and Tom and Cindy"
    > > char *delim = " and ";
    > > char *token;
    > >
    > > token = strtok(mystring, delim);
    > >
    > > while (token !=NULL )
    > > {
    > > //do some work
    > > cout << token << endl;
    > > token = strtok(NULL,delim)
    > > }
    > >
    > > my output would return JePererTomCiy
    > > I need my output to be Jane Peter Tome Cindy
    > >

    >
    >
    > JePererTomCiy
    >
    > That output doesn't make any sense, unless you mis-typed it... <?>
    >


    strtok writes a null character '\0' at the end of each token, and
    takes tokens to be seperated by sequences of ' ', 'a', 'n' and 'd'
    characters.
    So, the first token is 'J', since the "an" after it is a delimiter
    sequence.
    The 'J' is null-terminated by strtok writing '\0' over the 'a', and
    duly printed.
    The next token returned is "Peter" (I assume the "Perer" was a typo).
    The null terminator is written over the following space character, and
    the token is printed.

    .... and so on.
    This clearly isn't what the author wanted, but the output looks
    plausible for the code.

    >
    > Jeremy
    Simon Turner, Jul 10, 2003
    #9
  10. "Mike Wahler" <> wrote in message
    news:befjti$qf1$...
    | Lans <> wrote in message
    | news:...

    [snip]

    | /* void replace_delims(std::string& s) */
    | /* -- modifies argument 's' as follows: */
    | /* */
    | /* - Removes all occurrences of the string "and" which are */
    | /* delimited by whitespace and/or end-of-string */
    | /* */
    | /* - Replaces consecutive whitespace characters with a */
    | /* single space character */
    | /* */
    | /* - Removes leading and trailing whitespace */

    Hey Mike.

    Anyone would think you were a 'C' programmer :).

    Cheers.
    Chris Val
    Chris \( Val \), Jul 10, 2003
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kelvin@!!!

    tokenize a string

    Kelvin@!!!, Feb 24, 2005, in forum: C++
    Replies:
    4
    Views:
    7,884
  2. Replies:
    20
    Views:
    3,086
    Ben Bacarisse
    Feb 18, 2006
  3. Sree

    string tokenize...

    Sree, Mar 8, 2007, in forum: Java
    Replies:
    1
    Views:
    461
    Robert Klemme
    Mar 8, 2007
  4. WP
    Replies:
    3
    Views:
    348
    David Harmon
    Nov 23, 2007
  5. vijayanand2k
    Replies:
    0
    Views:
    533
    vijayanand2k
    Apr 2, 2009
Loading...

Share This Page