Parse tokens from a string

Discussion in 'C Programming' started by Steven Woody, Aug 23, 2005.

  1. Steven Woody

    Steven Woody Guest

    in C, is there any function can be used to decompose tokens from a string? if
    not, can i find it in CPP? thanks!

    --
    steven woody (id: narke)

    Celine: Well, who says relationships have to last forever?

    - Before Sunrise (1995)
    Steven Woody, Aug 23, 2005
    #1
    1. Advertising

  2. Steven Woody

    Simon Biber Guest

    Steven Woody wrote:
    > in C, is there any function can be used to decompose tokens from a string? if
    > not, can i find it in CPP? thanks!


    The original C library comes with a function to do that, it's called
    strtok, which stands for "string tokenise" IIRC. The interface leaves
    something to be desired though. It's not that hard to do it manually by
    manipulating the string as an array of char or through pointers.

    char foo[] = "joy to the world";
    char *p = strtok(foo, " ");
    char *q = strtok(0, " ");
    char *r = strtok(0, " ");
    char *s = strtok(0, " ");

    now
    p == foo
    q == foo + 4
    r == foo + 7
    s == foo + 11

    and
    foo[3] == 0 (previously was ' ')
    foo[6] == 0 (previously was ' ')
    foo[10] == 0 (previously was ' ')
    foo[16] == 0 (unchanged)

    Make sure you understand that it modifies the original array, and
    returns pointers into that array! The strings you get back should be
    copied into some more permanent storage if you plan to re-use the
    original array to read in another line of input.

    Also understand that it treats several consecutive separators as one, so
    the string "a,b,,d" will be parsed as {"a", "b", "d"} instead of {"a",
    "b", "", "d"}.

    --
    Simon.
    Simon Biber, Aug 23, 2005
    #2
    1. Advertising

  3. Steven Woody

    Eric Sosman Guest

    Steven Woody wrote:
    > in C, is there any function can be used to decompose tokens from a string? if
    > not, can i find it in CPP? thanks!


    There's strtok(). It has some drawbacks and may not do
    exactly the kind of "decomposition" you want -- but since
    you didn't describe what you want, all I can do is suggest
    that you take a look.

    Another possibility is sscanf(). It, too, has drawbacks
    and may not do exactly what you had in mind, but take a
    look anyhow.

    Finally, there's roll-your-own string-bashing with
    strchr(), strrchr(), strstr(), strspn(), strcspn(), ...

    --
    Eric Sosman, Aug 23, 2005
    #3
  4. Steven Woody

    tedu Guest

    Eric Sosman wrote:
    > Steven Woody wrote:
    > > in C, is there any function can be used to decompose tokens from a string? if
    > > not, can i find it in CPP? thanks!

    >
    > There's strtok(). It has some drawbacks and may not do
    > exactly the kind of "decomposition" you want -- but since
    > you didn't describe what you want, all I can do is suggest
    > that you take a look.


    thought not standard, strsep() may also be available in some
    environments. if it's not, the source is easy to come by and
    incorporate.
    tedu, Aug 23, 2005
    #4
  5. Steven Woody

    Ben Pfaff Guest

    Eric Sosman <> writes:

    > Steven Woody wrote:
    >> in C, is there any function can be used to decompose tokens from a string? if
    >> not, can i find it in CPP? thanks!

    >
    > There's strtok(). It has some drawbacks and may not do
    > exactly the kind of "decomposition" you want -- but since
    > you didn't describe what you want, all I can do is suggest
    > that you take a look.


    For what it's worth, here is my standard list of drawbacks to
    strtok():

    * It merges adjacent delimiters. If you use a comma as
    your delimiter, then "a,,b,c" is three tokens, not
    four. This is often the wrong thing to do. In fact,
    it is only the right thing to do, in my experience,
    when the delimiter set is limited to white space.

    * The identity of the delimiter is lost, because it is
    changed to a null terminator.

    * It modifies the string that it tokenizes. This is bad
    because it forces you to make a copy of the string if
    you want to use it later. It also means that you can't
    tokenize a string literal with it; this is not
    necessarily something you'd want to do all the time but
    it is surprising.

    * It can only be used once at a time. If a sequence of
    strtok() calls is ongoing and another one is started,
    the state of the first one is lost. This isn't a
    problem for small programs but it is easy to lose track
    of such things in hierarchies of nested functions in
    large programs. In other words, strtok() breaks
    encapsulation.

    --
    "I hope, some day, to learn to read.
    It seems to be even harder than writing."
    --Richard Heathfield
    Ben Pfaff, Aug 23, 2005
    #5
  6. Steven Woody

    pete Guest

    tedu wrote:
    >
    > Eric Sosman wrote:
    > > Steven Woody wrote:
    > > > in C, is there any function
    > > > can be used to decompose tokens from a string? if
    > > > not, can i find it in CPP? thanks!

    > >
    > > There's strtok(). It has some drawbacks and may not do
    > > exactly the kind of "decomposition" you want -- but since
    > > you didn't describe what you want, all I can do is suggest
    > > that you take a look.

    >
    > thought not standard, strsep() may also be available in some
    > environments. if it's not, the source is easy to come by and
    > incorporate.


    Here's what I have for strsep:

    #include <string.h>

    char *str_sep(char **s1, const char *s2)
    {
    char *const p1 = *s1;

    if (p1 != NULL) {
    *s1 = strpbrk(p1, s2);
    if (*s1 != NULL) {
    *(*s1)++ = '\0';
    }
    }
    return p1;
    }


    /*
    ** K&R2 Exercise 2-4
    ** alternate squeeze function using str_sep.
    */

    char *str_squeeze_s(char *s1, const char *s2)
    {
    char *const p1 = s1;
    const char *const p2 = s2;
    char * p3 = p1;

    do {
    s2 = str_sep(&p3, p2);
    while (*s2 != '\0') {
    *s1++ = *s2++;
    }
    } while (p3 != NULL);
    *s1 = '\0';
    return p1;
    }

    --
    pete
    pete, Aug 23, 2005
    #6
  7. Steven Woody

    Steven Woody Guest

    Eric Sosman <> writes:

    > Steven Woody wrote:
    >> in C, is there any function can be used to decompose tokens from a string? if
    >> not, can i find it in CPP? thanks!

    >
    > There's strtok(). It has some drawbacks and may not do
    > exactly the kind of "decomposition" you want -- but since
    > you didn't describe what you want, all I can do is suggest
    > that you take a look.


    strtok's man page say, it can not handle empty field. what exactly does this
    mean ?

    actually, what i want to do is:

    1, read a line in a text configuration file.
    2, the line are just string tokens seperated white spaces.
    3, goto 1 until eof.

    >
    > Another possibility is sscanf(). It, too, has drawbacks
    > and may not do exactly what you had in mind, but take a
    > look anyhow.
    >
    > Finally, there's roll-your-own string-bashing with
    > strchr(), strrchr(), strstr(), strspn(), strcspn(), ...
    >
    > --
    >
    >


    --
    steven woody (id: narke)

    Celine: Well, who says relationships have to last forever?

    - Before Sunrise (1995)
    Steven Woody, Aug 24, 2005
    #7
  8. Steven Woody

    Steven Woody Guest

    pete <> writes:

    > tedu wrote:
    >>
    >> Eric Sosman wrote:
    >> > Steven Woody wrote:
    >> > > in C, is there any function
    >> > > can be used to decompose tokens from a string? if
    >> > > not, can i find it in CPP? thanks!
    >> >
    >> > There's strtok(). It has some drawbacks and may not do
    >> > exactly the kind of "decomposition" you want -- but since
    >> > you didn't describe what you want, all I can do is suggest
    >> > that you take a look.

    >>
    >> thought not standard, strsep() may also be available in some
    >> environments. if it's not, the source is easy to come by and
    >> incorporate.

    >
    > Here's what I have for strsep:
    >
    > #include <string.h>
    >
    > char *str_sep(char **s1, const char *s2)
    > {
    > char *const p1 = *s1;
    >
    > if (p1 != NULL) {
    > *s1 = strpbrk(p1, s2);
    > if (*s1 != NULL) {
    > *(*s1)++ = '\0';
    > }
    > }
    > return p1;
    > }
    >
    >
    > /*
    > ** K&R2 Exercise 2-4
    > ** alternate squeeze function using str_sep.
    > */
    >
    > char *str_squeeze_s(char *s1, const char *s2)
    > {
    > char *const p1 = s1;
    > const char *const p2 = s2;
    > char * p3 = p1;
    >
    > do {
    > s2 = str_sep(&p3, p2);
    > while (*s2 != '\0') {
    > *s1++ = *s2++;
    > }
    > } while (p3 != NULL);
    > *s1 = '\0';
    > return p1;
    > }
    >
    > --
    > pete


    thanks for sharing the code, it's worthy to read.

    --
    steven woody (id: narke)

    Virginia Woolf: Someone has to die Leonard, in order that the rest of
    us should value our life more.

    - The Hours (2002)
    Steven Woody, Aug 24, 2005
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?RWx0b24gVw==?=

    RE: string into tokens

    =?Utf-8?B?RWx0b24gVw==?=, Oct 13, 2005, in forum: ASP .Net
    Replies:
    0
    Views:
    412
    =?Utf-8?B?RWx0b24gVw==?=
    Oct 13, 2005
  2. =?Utf-8?B?TFc=?=

    string into tokens

    =?Utf-8?B?TFc=?=, Oct 13, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    379
    =?Utf-8?B?TFc=?=
    Oct 13, 2005
  3. Christopher Benson-Manica

    String tokens/parsing

    Christopher Benson-Manica, Jan 29, 2004, in forum: C++
    Replies:
    10
    Views:
    831
    David Rubin
    Feb 2, 2004
  4. Replies:
    19
    Views:
    1,092
    Daniel Vallstrom
    Mar 15, 2005
  5. Cirene
    Replies:
    4
    Views:
    411
    Cirene
    Aug 15, 2008
Loading...

Share This Page