Data via serial port breaks down in multiple packets

Discussion in 'C Programming' started by Muffinman, Aug 9, 2013.

  1. Muffinman

    Muffinman Guest

    Hello all,

    I'm not sure whether this is the appropriate place for this question,
    but I couldn't find a better place. So here it goes:

    I have an application that needs to receive some data via a serial port.
    The data is received, however, it breaks down in multiple packets. So
    sending 18 bytes often results in two packets of 8 bytes and one of 2
    (but sometimes also 8,9&1). Setting VMIN allows me to push it up to how
    many bytes I want. However, sometimes I need to receive 4 bytes, other
    times 25. It is not consistent, so that's no option.

    I need to process commands.
    - Processing each packet at a time would not work because it can break
    down commands.
    - I could fill my own buffer, but timing is quite crucial...

    1) So, can I solve this? Is there anything I can do with my
    configuration (see below)?

    2) Why does this happen? The only thing I can think of is that the
    sending device skips a opportunity to send data every so bytes...

    I hope someone can help.

    Thanks in advance, Maarten


    ################# serial port polling #################
    if(poll_fds[0].revents & POLLIN)
    {
    bytes_read = read(serial_fd, device_status_message,
    sizeof(device_status_message));
    if(bytes_read <= 0)
    continue;
    [...]

    ############# serial port configuraton #################
    /* c_cflag
    * setting all the variables according to the configuration file
    * and some default settings */
    serial_new_attr.c_cflag |= (CLOCAL | CREAD);
    if(common_data.serial_parity == 0)
    serial_new_attr.c_cflag &= ~PARENB;
    else
    {
    serial_new_attr.c_cflag |= PARENB;
    /* enable parity check and strip it from output */
    serial_new_attr.c_iflag |= (INPCK | ISTRIP);
    if(common_data.serial_even == 0)
    serial_new_attr.c_cflag |= PARODD;
    else
    serial_new_attr.c_cflag &= ~PARODD;
    }

    if(common_data.serial_2stop == 0)
    serial_new_attr.c_cflag &= ~CSTOPB;
    else
    serial_new_attr.c_cflag |= CSTOPB;
    serial_new_attr.c_cflag &= ~CSIZE;
    serial_new_attr.c_cflag |= common_data.serial_bits;

    /* c_lflag
    * enable raw input */
    serial_new_attr.c_lflag &= ~(ICANON | ECHO | ECHOE | ISIG);

    /* Disabling hardware flow control */
    serial_new_attr.c_cflag &= ~CRTSCTS;
    /* Disabling software flow control */
    serial_new_attr.c_iflag &= ~(IXON | IXOFF | IXANY);

    /* Disabling any postprocessing */
    serial_new_attr.c_oflag &= ~OPOST;

    /* Minimum characters to read (non-canonical) */
    serial_new_attr.c_cc[VMIN] = 1;
    /* Wait indefinitely for input (non-canonical) */
    serial_new_attr.c_cc[VTIME] = 3;

    cfsetospeed(&serial_new_attr, common_data.serial_speed);
    cfsetispeed(&serial_new_attr, common_data.serial_speed);
    Muffinman, Aug 9, 2013
    #1
    1. Advertising

  2. Muffinman

    James Kuyper Guest

    On 08/09/2013 11:55 AM, Muffinman wrote:
    > Hello all,
    >
    > I'm not sure whether this is the appropriate place for this question,
    > but I couldn't find a better place.


    It isn't. The C standard says nothing about serial ports, read(),
    cfsetispeed() or cfsetospeed(), nor about any of the macros or
    structures you were using. Functions with those names are installed on
    my desktop system, and the man pages for those functions describe them
    as POSIX functions. If the functions you're using are the same ones,
    then the best place to get answers to questions like this is
    comp.unix.programmer.
    James Kuyper, Aug 9, 2013
    #2
    1. Advertising

  3. James Kuyper <> wrote:

    > On 08/09/2013 11:55 AM, Muffinman wrote:


    >> I'm not sure whether this is the appropriate place for this question,
    >> but I couldn't find a better place.


    > It isn't. The C standard says nothing about serial ports, read(),
    > cfsetispeed() or cfsetospeed(), nor about any of the macros or
    > structures you were using.


    But C does describe fread(), which you could use with serial ports,
    or other data streams.

    The Unix/C model for data is as a stream of bytes. If you write
    data with a series of fwrite() calls, there is no reason to
    expect the number of bytes read be each of a series of fread()
    calls to return the same succession of record lengths.

    There are systems that do preserve record lengths. Blocks on unix
    tapes being one example. (I believe that comes through to fread(),
    but right now I am not sure about that.)

    If you need to preserve record marks, you might consider:

    https://www.ietf.org/rfc/rfc1831.txt

    -- glen
    glen herrmannsfeldt, Aug 9, 2013
    #3
  4. Muffinman

    James Kuyper Guest

    On 08/09/2013 02:51 PM, glen herrmannsfeldt wrote:
    > James Kuyper <> wrote:
    >
    >> On 08/09/2013 11:55 AM, Muffinman wrote:

    >
    >>> I'm not sure whether this is the appropriate place for this question,
    >>> but I couldn't find a better place.

    >
    >> It isn't. The C standard says nothing about serial ports, read(),
    >> cfsetispeed() or cfsetospeed(), nor about any of the macros or
    >> structures you were using.

    >
    > But C does describe fread(), which you could use with serial ports,
    > or other data streams.


    And it says nothing useful about what happens when you do so, or at
    least, about how it's different from what happens when you open an
    ordinary file. There's really no useful answer to his question that can
    be derived from the C standard, not even if you re-write it to use
    fread() rather than read().

    And since Unix open() and ioctl() can enable unix features that can
    cause read() to behave quite differently from the way it normally works
    when invoked by fread() (consider, for example, blocking vs.
    non-blocking I/O), it's generally not a good idea to simply assume that
    unix code calling read() can simply be replaced by C code calling
    fread(). I'm not sure whether that is or could be relevant in this case
    - but that's precisely the point. There will be people in
    comp.unix.programmer who would be sure.
    James Kuyper, Aug 9, 2013
    #4
  5. Muffinman

    Joe Pfeiffer Guest

    Muffinman <news**REMOVETHIS**@koster.tk> writes:

    > Hello all,
    >
    > I'm not sure whether this is the appropriate place for this question,
    > but I couldn't find a better place. So here it goes:
    >
    > I have an application that needs to receive some data via a serial port.
    > The data is received, however, it breaks down in multiple packets. So
    > sending 18 bytes often results in two packets of 8 bytes and one of 2
    > (but sometimes also 8,9&1). Setting VMIN allows me to push it up to how
    > many bytes I want. However, sometimes I need to receive 4 bytes, other
    > times 25. It is not consistent, so that's no option.
    >
    > I need to process commands.
    > - Processing each packet at a time would not work because it can break
    > down commands.
    > - I could fill my own buffer, but timing is quite crucial...
    >
    > 1) So, can I solve this? Is there anything I can do with my
    > configuration (see below)?
    >
    > 2) Why does this happen? The only thing I can think of is that the
    > sending device skips a opportunity to send data every so bytes...


    You're pretty much out of luck. The read() man page says,

    On success, the number of bytes read is returned (zero indicates end of
    file), and the file position is advanced by this number. It is not an
    error if this number is smaller than the number of bytes requested;
    this may happen for example because fewer bytes are actually available
    right now (maybe because we were close to end-of-file, or because we
    are reading from a pipe, or from a terminal),

    So... you tell the system call how many bytes you want, it tells you
    how many bytes you get. Lots of my code has loops like this:

    numread = 0;
    while (numread < numwanted) {
    thisread = read(fd, &buf[numread], numwanted-numread);
    if (thisread < 0) {
    perror("read at line xxx");
    break;
    }
    numread += thisread;
    }

    Asking in a unix-specific newsgroup certainly couldn't hurt.
    Joe Pfeiffer, Aug 10, 2013
    #5
  6. On Friday, August 9, 2013 8:55:03 AM UTC-7, Muffinman wrote:
    > Hello all,
    > I'm not sure whether this is the appropriate place for this question,
    > but I couldn't find a better place. So here it goes:
    >
    > I have an application that needs to receive some data via a serial port.
    > The data is received, however, it breaks down in multiple packets. So
    > sending 18 bytes often results in two packets of 8 bytes and one of 2
    > (but sometimes also 8,9&1). Setting VMIN allows me to push it up to how
    > many bytes I want. However, sometimes I need to receive 4 bytes, other
    > times 25. It is not consistent, so that's no option.
    > I need to process commands.
    >
    > - Processing each packet at a time would not work because it can break
    > down commands.
    >
    > - I could fill my own buffer, but timing is quite crucial...
    >
    >
    >
    > 1) So, can I solve this? Is there anything I can do with my
    >
    > configuration (see below)?
    >
    >
    >
    > 2) Why does this happen? The only thing I can think of is that the
    >
    > sending device skips a opportunity to send data every so bytes...
    >
    >
    >
    > I hope someone can help.
    >
    >
    >
    > Thanks in advance, Maarten
    >
    >
    >
    >
    >
    > ################# serial port polling #################
    >
    > if(poll_fds[0].revents & POLLIN)
    >
    > {
    >
    > bytes_read = read(serial_fd, device_status_message,
    >
    > sizeof(device_status_message));
    >
    > if(bytes_read <= 0)
    >
    > continue;
    >
    > [...]
    >
    >
    >
    > ############# serial port configuraton #################
    >
    > /* c_cflag
    >
    > * setting all the variables according to the configuration file
    >
    > * and some default settings */
    >
    > serial_new_attr.c_cflag |= (CLOCAL | CREAD);
    >
    > if(common_data.serial_parity == 0)
    >
    > serial_new_attr.c_cflag &= ~PARENB;
    >
    > else
    >
    > {
    >
    > serial_new_attr.c_cflag |= PARENB;
    >
    > /* enable parity check and strip it from output */
    >
    > serial_new_attr.c_iflag |= (INPCK | ISTRIP);
    >
    > if(common_data.serial_even == 0)
    >
    > serial_new_attr.c_cflag |= PARODD;
    >
    > else
    >
    > serial_new_attr.c_cflag &= ~PARODD;
    >
    > }
    >
    >
    >
    > if(common_data.serial_2stop == 0)
    >
    > serial_new_attr.c_cflag &= ~CSTOPB;
    >
    > else
    >
    > serial_new_attr.c_cflag |= CSTOPB;
    >
    > serial_new_attr.c_cflag &= ~CSIZE;
    >
    > serial_new_attr.c_cflag |= common_data.serial_bits;
    >
    >
    >
    > /* c_lflag
    >
    > * enable raw input */
    >
    > serial_new_attr.c_lflag &= ~(ICANON | ECHO | ECHOE | ISIG);
    >
    >
    >
    > /* Disabling hardware flow control */
    >
    > serial_new_attr.c_cflag &= ~CRTSCTS;
    >
    > /* Disabling software flow control */
    >
    > serial_new_attr.c_iflag &= ~(IXON | IXOFF | IXANY);
    >
    >
    >
    > /* Disabling any postprocessing */
    >
    > serial_new_attr.c_oflag &= ~OPOST;
    >
    >
    >
    > /* Minimum characters to read (non-canonical) */
    >
    > serial_new_attr.c_cc[VMIN] = 1;
    >
    > /* Wait indefinitely for input (non-canonical) */
    >
    > serial_new_attr.c_cc[VTIME] = 3;
    >
    >
    >
    > cfsetospeed(&serial_new_attr, common_data.serial_speed);
    >
    > cfsetispeed(&serial_new_attr, common_data.serial_speed);



    This comes up wether you are reading a serial port, a socket, a terminal, or some other device (even text with funny delimiters from a disk file).

    If you need to process data, you must fill a buffer until you have what the command requires and be prepared to save the leftovers for the next time.

    If you can't tell where a command ends, you are badly stuck. But most things have either a length designator or a delimiter of some kind.

    There are two main techniques:
    One is to read data (perhaps a byte at a time) until you have a command and then read the amount of data (perhaps a byte at a time) required for that command (dealing with leftovers, if you somehow read too much).

    The other is to read as much data as you can get comfortably and to parse through it (and either keeping the buffer filled or dealing with getting more whenever you aren't sure that you have enough).

    It doesn't matter whether you use read (), read_ (), _read (), __read (), fread (), pread (), ioread (), Read (), READ (), recv (), recvfrom (), portread (), readport (), or GimmeSomeLovin ().
    Michael Angelo Ravera, Aug 10, 2013
    #6
  7. Muffinman

    Muffinman Guest

    On 09-08-13 17:55, Muffinman wrote:
    > 1) So, can I solve this? Is there anything I can do with my
    > configuration (see below)?
    >
    > 2) Why does this happen? The only thing I can think of is that the
    > sending device skips a opportunity to send data every so bytes...


    I get a few explanations why this might happen from here and
    comp.unix.programer. However, the end result seems to be: deal with it.
    So I guess, that's the advice I'm gonna go with. Not sure how, but I'll
    manage.

    Thanks all, Maarten
    Muffinman, Aug 11, 2013
    #7
  8. Muffinman <news**REMOVETHIS**@koster.tk> wrote:
    > On 09-08-13 17:55, Muffinman wrote:
    >> 1) So, can I solve this? Is there anything I can do with my
    >> configuration (see below)?


    >> 2) Why does this happen? The only thing I can think of is that the
    >> sending device skips a opportunity to send data every so bytes...


    > I get a few explanations why this might happen from here and
    > comp.unix.programer. However, the end result seems to be: deal with it.
    > So I guess, that's the advice I'm gonna go with. Not sure how, but I'll
    > manage.


    In the serial case, each byte is pretty much independent, but it also
    happens in the case of TCP. Datagram boundaries are preserved for UDP,
    but TCP is a stream protocol.

    I previously posted the way it is done when protocols switch from UDP
    to TCP, such as NFS.

    -- glen
    glen herrmannsfeldt, Aug 11, 2013
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Manoj K.S
    Replies:
    1
    Views:
    686
    127.0.0.1
    Dec 2, 2003
  2. Steve C. Orr, MCSD

    Re: Serial Printing via COM port

    Steve C. Orr, MCSD, Aug 21, 2003, in forum: ASP .Net
    Replies:
    0
    Views:
    615
    Steve C. Orr, MCSD
    Aug 21, 2003
  3. Pom
    Replies:
    2
    Views:
    1,617
    Bas-i
    Jan 31, 2007
  4. msalerno
    Replies:
    3
    Views:
    401
    Ilmari Karonen
    Jul 14, 2005
  5. Replies:
    2
    Views:
    80
    Grant Edwards
    Dec 16, 2013
Loading...

Share This Page