strtok() and EOL

Discussion in 'C Programming' started by Fernando Barsoba, Nov 29, 2005.

  1. Hi,

    I'm using strtok() in the following way:

    void obtain_param(char *pmsg, CONF_PARAMS *cnf ) {
    char *s1, *s2;
    size_t msg_len;

    s1 = strtok (pmsg,":");
    if (s1) {
    s2 = strtok (0, "");
    msg_len = strcspn(s2, "\n");
    if (memcmp(s1, "message", sizeof("message")) == 0) {
    cnf->msg_length = msg_len;
    cnf->message = malloc(msg_len);
    memcpy(cnf->message, s2, msg_len);
    cnf->message[msg_len]='\0';

    ....

    I extract the information from a configuration file. For some reason,
    sometimes I get for msg_len 15 for this message:

    >>message:this is a test


    (when 'this is a test\n' is only 14 characters long, w/o the '\n')

    In other installations (Linux) works just fine, but in my
    Eclipse/CDT/cygwin sometimes is not.

    Am I using strtok incorrectly? is the 's2' pointer messing things up?

    Thanks,

    F~
     
    Fernando Barsoba, Nov 29, 2005
    #1
    1. Advertising

  2. Fernando Barsoba

    Jack Klein Guest

    On Tue, 29 Nov 2005 15:16:17 GMT, Fernando Barsoba
    <> wrote in comp.lang.c:

    > Hi,
    >
    > I'm using strtok() in the following way:
    >
    > void obtain_param(char *pmsg, CONF_PARAMS *cnf ) {
    > char *s1, *s2;
    > size_t msg_len;
    >
    > s1 = strtok (pmsg,":");
    > if (s1) {
    > s2 = strtok (0, "");
    > msg_len = strcspn(s2, "\n");
    > if (memcmp(s1, "message", sizeof("message")) == 0) {
    > cnf->msg_length = msg_len;
    > cnf->message = malloc(msg_len);
    > memcpy(cnf->message, s2, msg_len);
    > cnf->message[msg_len]='\0';
    >
    > ...
    >
    > I extract the information from a configuration file. For some reason,
    > sometimes I get for msg_len 15 for this message:
    >
    > >>message:this is a test

    >
    > (when 'this is a test\n' is only 14 characters long, w/o the '\n')
    >
    > In other installations (Linux) works just fine, but in my
    > Eclipse/CDT/cygwin sometimes is not.
    >
    > Am I using strtok incorrectly? is the 's2' pointer messing things up?
    >
    > Thanks,
    >
    > F~


    Where did the text string passed in 'pmsg' come from? Was it read
    from a file? If it was read from a file, was the file opened in text
    or binary mode? If you are reading a text file opened in binary mode
    under Windows, you are getting the string as it appears in the file,
    which would be:

    "message:this is a test\r\n"

    --
    Jack Klein
    Home: http://JK-Technology.Com
    FAQs for
    comp.lang.c http://www.eskimo.com/~scs/C-faq/top.html
    comp.lang.c++ http://www.parashift.com/c -faq-lite/
    alt.comp.lang.learn.c-c++
    http://www.contrib.andrew.cmu.edu/~ajo/docs/FAQ-acllc.html
     
    Jack Klein, Nov 29, 2005
    #2
    1. Advertising

  3. Fernando Barsoba

    Mike Wahler Guest

    "Fernando Barsoba" <> wrote in message
    news:5h_if.6989$6e5.1530@trnddc09...
    > Hi,
    >
    > I'm using strtok() in the following way:
    >
    > void obtain_param(char *pmsg, CONF_PARAMS *cnf ) {
    > char *s1, *s2;
    > size_t msg_len;
    >
    > s1 = strtok (pmsg,":");
    > if (s1) {
    > s2 = strtok (0, "");
    > msg_len = strcspn(s2, "\n");
    > if (memcmp(s1, "message", sizeof("message")) == 0) {
    > cnf->msg_length = msg_len;
    > cnf->message = malloc(msg_len);
    > memcpy(cnf->message, s2, msg_len);
    > cnf->message[msg_len]='\0';
    >
    > ...
    >
    > I extract the information from a configuration file. For some reason,
    > sometimes I get for msg_len 15 for this message:
    >
    > >>message:this is a test

    >
    > (when 'this is a test\n' is only 14 characters long, w/o the '\n')
    >
    > In other installations (Linux) works just fine, but in my
    > Eclipse/CDT/cygwin sometimes is not.
    >
    > Am I using strtok incorrectly? is the 's2' pointer messing things up?


    Are you perhaps opening your file in binary mode? Note that
    some OS's store newline as more than one character in text files.
    E.g. Windows uses CR/LF pair and typical Windows C implementations
    represent '\n' internally as LF (this would give the behavior
    you describe if file were opened in binary mode on such systems
    -- the 'extra' CR character would count as part of length returned
    by 'strcspn()').

    If you're not using it already, try text mode.

    -Mike
     
    Mike Wahler, Nov 29, 2005
    #3
  4. Fernando Barsoba wrote:
    > Hi,
    >
    > I'm using strtok() in the following way:


    Probably fine, but you do know the issues with strtok, right?
    In clc it is not re-entrant. In (OT) land, it is often not threadsafe
    either. It is probably fine as you use it below.

    >
    > void obtain_param(char *pmsg, CONF_PARAMS *cnf ) {
    > char *s1, *s2;
    > size_t msg_len;
    >
    > s1 = strtok (pmsg,":");
    > if (s1) {
    > s2 = strtok (0, "");


    I'm not totally sure the empty string is legit here, perhaps others
    can say. Works fine if you say ":" too.

    > msg_len = strcspn(s2, "\n");
    > if (memcmp(s1, "message", sizeof("message")) == 0) {


    Do you intend to compare the '\0' as you are? sizeof("message") is 8.
    Fine either way, strtok will have put one in.

    > cnf->msg_length = msg_len;
    > cnf->message = malloc(msg_len);


    You don't allocate enough space here for the string including the '\0'

    > memcpy(cnf->message, s2, msg_len);
    > cnf->message[msg_len]='\0';


    One byte overwrite here.

    >
    > ...
    >
    > I extract the information from a configuration file. For some reason,
    > sometimes I get for msg_len 15 for this message:
    >
    > >>message:this is a test

    >
    > (when 'this is a test\n' is only 14 characters long, w/o the '\n')
    >
    > In other installations (Linux) works just fine, but in my
    > Eclipse/CDT/cygwin sometimes is not.
    >
    > Am I using strtok incorrectly? is the 's2' pointer messing things up?


    I think you are using strtok mostly OK. I'd guess you are getting
    messed up by DOS end of line vs unix here. Perhaps depends
    on how the configuration file is read (binary vs text?).

    -David
     
    David Resnick, Nov 29, 2005
    #4
  5. Fernando Barsoba

    Ben Pfaff Guest

    Fernando Barsoba <> writes:

    > I'm using strtok() in the following way:


    I don't generally recommend using strtok(). It has at least
    these problems:

    * It merges adjacent delimiters. If you use a comma as
    your delimiter, then "a,,b,c" is three tokens, not
    four. This is often the wrong thing to do. In fact,
    it is only the right thing to do, in my experience,
    when the delimiter set is limited to white space.

    * The identity of the delimiter is lost, because it is
    changed to a null terminator.

    * It modifies the string that it tokenizes. This is bad
    because it forces you to make a copy of the string if
    you want to use it later. It also means that you can't
    tokenize a string literal with it; this is not
    necessarily something you'd want to do all the time but
    it is surprising.

    * It can only be used once at a time. If a sequence of
    strtok() calls is ongoing and another one is started,
    the state of the first one is lost. This isn't a
    problem for small programs but it is easy to lose track
    of such things in hierarchies of nested functions in
    large programs. In other words, strtok() breaks
    encapsulation.

    --
    "You call this a *C* question? What the hell are you smoking?" --Kaz
     
    Ben Pfaff, Nov 29, 2005
    #5
  6. Jack Klein wrote:
    > On Tue, 29 Nov 2005 15:16:17 GMT, Fernando Barsoba
    > <> wrote in comp.lang.c:
    >
    >> Hi,
    >>
    >> I'm using strtok() in the following way:
    >>
    >> void obtain_param(char *pmsg, CONF_PARAMS *cnf ) {
    >> char *s1, *s2;
    >> size_t msg_len;
    >>
    >> s1 = strtok (pmsg,":");

    (...)
    > Where did the text string passed in 'pmsg' come from? Was it read
    > from a file? If it was read from a file, was the file opened in text
    > or binary mode? If you are reading a text file opened in binary mode
    > under Windows, you are getting the string as it appears in the file,
    > which would be:
    >
    > "message:this is a test\r\n"
    >

    Thanks to all for your comments. Indeed, the string I'm getting is
    "message:this is a test\r\n". And 'pmsg' comes also from a file. Oddly
    enough, I thought I opened the file in text mode. Here's the sequence of
    calls:

    fp = open_file(argv[1], "r");
    cnf = get_param(fp);

    -------------
    CONF_PARAMS *get_param(FILE *f_param) {
    ....
    p_msg = fgets(param, PARAM_LENGTH, f_param);
    ....
    }
    --------
    FILE * open_file(char *file_name, char *foptions) {
    FILE *fp;
    if ((fp = fopen(file_name, foptions)) == NULL) {
    printf("Can't open %s\n", file_name);
    exit(1);
    }
    return fp;
    }

    As the comments pointed out, this seems to be a Windows issue. That's
    why in Linux works fine. I'm not opening the file in binary mode though..

    F~
     
    Fernando Barsoba, Nov 29, 2005
    #6
  7. Fernando Barsoba

    Flash Gordon Guest

    Jack Klein wrote:
    > On Tue, 29 Nov 2005 15:16:17 GMT, Fernando Barsoba
    > <> wrote in comp.lang.c:


    <snip>

    >> >>message:this is a test

    >>
    >> (when 'this is a test\n' is only 14 characters long, w/o the '\n')
    >>
    >> In other installations (Linux) works just fine, but in my
    >> Eclipse/CDT/cygwin sometimes is not.
    >>
    >> Am I using strtok incorrectly? is the 's2' pointer messing things up?
    >>
    >> Thanks,
    >>
    >> F~

    >
    > Where did the text string passed in 'pmsg' come from? Was it read
    > from a file? If it was read from a file, was the file opened in text
    > or binary mode? If you are reading a text file opened in binary mode
    > under Windows, you are getting the string as it appears in the file,
    > which would be:
    >
    > "message:this is a test\r\n"


    Note that since Cygwin is emulating some of Unix it may think of \n as
    being the line terminator rather than \r\n even when you open a stream
    in text mode, so you could still get a spurious \r. For the intricacies
    of Cygwin line termination you will have to ask on a Cygwin mailing
    list, but if this is your problem one option is to check if the last
    character of the line you read is \r and if so strip it.
    --
    Flash Gordon
    Living in interesting times.
    Although my email address says spam, it is real and I read it.
     
    Flash Gordon, Nov 29, 2005
    #7
  8. Fernando Barsoba

    Flash Gordon Guest

    Fernando Barsoba wrote:
    > Jack Klein wrote:
    >> On Tue, 29 Nov 2005 15:16:17 GMT, Fernando Barsoba
    >> <> wrote in comp.lang.c:


    <snip>

    >> "message:this is a test\r\n"
    >>

    > Thanks to all for your comments. Indeed, the string I'm getting is
    > "message:this is a test\r\n". And 'pmsg' comes also from a file. Oddly
    > enough, I thought I opened the file in text mode. Here's the sequence of
    > calls:
    >
    > fp = open_file(argv[1], "r");


    Yes, that is opening in text mode.

    <snip>

    > As the comments pointed out, this seems to be a Windows issue. That's
    > why in Linux works fine. I'm not opening the file in binary mode though..


    See my previous post, it's a Cygwin issue. Specifically, Cygwin is
    emulation Unix to a degree (that is its purpose) so by default it used
    the Unix text file conventions. Ask on a Cygwin mailing list for the
    details.
    --
    Flash Gordon
    Living in interesting times.
    Although my email address says spam, it is real and I read it.
     
    Flash Gordon, Nov 29, 2005
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Daves

    replacing EOL with <br>

    Daves, Apr 13, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    521
    Karl Seguin
    Apr 14, 2005
  2. Tim Tyler
    Replies:
    7
    Views:
    23,010
    danielson317
    Sep 15, 2011
  3. jkv
    Replies:
    0
    Views:
    405
  4. jkv
    Replies:
    0
    Views:
    423
  5. Jean-Paul Calderone

    Re: EOL for sys.stdin.readline() and raw_input()

    Jean-Paul Calderone, Feb 26, 2009, in forum: Python
    Replies:
    0
    Views:
    595
    Jean-Paul Calderone
    Feb 26, 2009
Loading...

Share This Page