clearing read stream buffers

Discussion in 'C Programming' started by JG, Feb 11, 2005.

  1. JG

    JG Guest

    Does anyone know a standard (or supported on Linux, Mac, Win32) way to
    clear a read stream buffer (standard ANSI C file stream)?

    I would even settle for a platform specific way of doing it.

    And no, I know I can use direct low level I/O or non-buffered to do
    reads, but for my app, I need the buffering. I can implement myself,
    but this is not optimal.

    Example, I open a read only file using fopen(). I periodically know
    from external methods, that the underlying file has been updated.
    Before I do my next read on the stream, I want to flush the read
    buffers and force it to read from disk (or kernel cache).

    fflush seems to work on Linux, but not sure it is standard.

    I don't see any function to do this, but it seems very needed.

    I can also trigger a flush of the buffers by fseeking to a much
    different area of the file. Does any fseek, even offseting by 1 index
    always force a flusshing of the buffers?
    JG, Feb 11, 2005
    1. Advertisements

  2. JG

    Luke Wu Guest

    once you've determined that there is still a bunch of characters
    sitting in the input stream (should include '\n' at least), then you
    can use:

    while( (c = fgetc(FOOFILE) != '\n' )
    Luke Wu, Feb 11, 2005
    1. Advertisements

  3. JG

    jschultz Guest

    He is not asking the typical "How do I forget whatever is in stdin

    He is asking with an actual file on disk (not a pipe/terminal/etc) that
    he performs a read upon, how can he portably and efficiently FORCE an
    open read stream to re-read data he wants from disk. The problem being
    that a write from another thread/stream/fd/process has changed the
    underlying file and he wants to ensure that the input stream doesn't
    just re-give him what is in its buffers from an earlier read.
    jschultz, Feb 11, 2005
  4. The only sure way I can think of is to close and reopen the stream, using,
    say, freopen().
    It isn't. In standard C applying fflush() to an input stream invokes
    undefined behaviour.
    I know what you mean, although surprisingly I don't remember needing it.
    I hope not, that would be inefficient if the data at the new position was
    already in the buffer. This approach seems to be your best bet however
    short of closing and reopening the file. Maybe a double seek, one to some
    far off position and then to the position you want. Even that isn't 100%
    guaranteed though.

    Lawrence Kirby, Feb 11, 2005
  5. JG

    jschultz Guest

    One way I was thinking of doing this is:

    if (setvbuf(read_stream, NULL, _IONBF, 0) != 0 ||
    setvbuf(read_stream, NULL, _IOFBF, 0) != 0) {
    return -1;

    /* do read */

    Any comments from the peanut gallery about how advisable / guaranteed
    this is to work?
    jschultz, Feb 11, 2005
  6. JG

    Ben Pfaff Guest

    From C99:

    The setvbuf function may be used only after the stream
    pointed to by stream has been associated with an open file
    and before any other operation (other than an unsuccessful
    call to setvbuf) is performed on the stream.

    Thus, calling setvbuf() twice is always undefined, regardless of
    when you do it. But you're only allowed to do it at all just
    after the file is opened.
    Ben Pfaff, Feb 11, 2005
  7. JG

    infobahn Guest

    You missed a ), and forgot to check for EOF.

    while( (c = fgetc(FOOFILE)) != '\n' && c != EOF)
    infobahn, Feb 11, 2005
  8. JG

    Alan Balmer Guest

    fclose(), fopen()?
    Alan Balmer, Feb 11, 2005
  9. JG

    jschultz Guest

    Ok, so it seems the only portable and guaranteed way to force a C input
    stream to re-read from disk is to close/re-open/re-seek the file?
    Doesn't that seem like a lot of overkill for a simple/common idea?
    jschultz, Feb 11, 2005
  10. JG

    jschultz Guest

    I mean performance overkill specifically. Because our program needs to
    do these kinds of reads very quickly and opening/closing/seeking and
    THEN reading seems like a ton more work than it should be.
    jschultz, Feb 11, 2005
  11. JG

    Eric Sosman Guest

    The C Standard tries to stear clear of describing how
    multiple programs can run concurrently in the same environment,
    of how they can synchronize their operations, and of what
    effects one program can have on another. This is a Good Thing,
    because a programming language standard that tried to address
    such matters would limit its applicability rather severely.
    If the Standard adopted in 1989 had embodied the Windows 3.1
    process model, for example, how useful would that Standard be
    for Unix, VMS, OS/400, MVS -- or even current Windows versions?

    In short, the idea is less "simple" than it may appear to
    someone who's struggling with one particular issue. (And in
    my three decades of writing C I can't recall needing to do
    what you describe, so "common" may be hard to defend, too.)

    .... and in a follow-up:
    When you measured the speed of fclose()/fopen()/fseek(),
    how slow did you find it to be? ("It is a capital offense
    to theorize before one has data." -- S. Holmes)
    Eric Sosman, Feb 11, 2005
  12. It isn't, fflush is only specified on output.
    fseek to the end? Or, if you want to read the newly added data, fseek
    to the last read position (possibly with an fseek to the start first).
    It's implementation dependent, the C standard doesn't say (or know)
    anything about underlying buffers. At least one implementation I've
    used would treat a backwards seek within a buffer as just adjusting a
    pointer. The only thing which is guaranteed to work is closing the file
    and opening it again (and seeking to the last read position in your

    However, I'd guess that doing fseek to the end of the file should always
    cause any /new/ data to be read in (not necessarily anything changed in
    data already read), so if the external changes are only appending to the
    file I'd guess that it should work. But really it's an implementation
    question, I don't think thre's any portable solution apart from closing
    and opening the file each time.

    Chris C
    Chris Croughton, Feb 11, 2005
  13. Which is why specific implementations may nor may not offer a facilityto do
    this more efficiently, but in a nonstandard way. You really need to ask
    experts in your platform, not generalists in C.
    Mark McIntyre, Feb 11, 2005
  14. JG

    jschultz Guest

    I was just asking if there was a standard way to force a C input stream
    to forget what's in its buffers without destroying the stream itself.
    Apparently there isn't. That seems like a rather silly hole to me as
    it would be VERY simple for a stream to forget what it has already read
    in and then re-read on the next read call.
    jschultz, Feb 12, 2005
  15. JG

    Chris Torek Guest

    Pretty much, yes.
    It is actually quite uncommon, and sometimes not all that simple.
    In particular, you first have to discover that the file has changed,
    and that will take code that lies outside the purview of the C
    standards (at which point you can perhaps abandon stdio entirely
    and simply use open/read/lseek/close, if those are the underlying

    I included an fpurge() function in the 4.xBSD stdio, but it does
    not seem to have caught on. The specification for fpurge() is,
    roughly speaking, "forget all buffered data recorded in this stdio
    `FILE *' thing". An fpurge() followed by an fseek() to the desired
    location would do the trick (although I am not 100% sure that my
    seek-optimization code would actually work right with the purge in
    this case :) ).

    Note that the 4.xBSD fpurge() also works on output files, discarding
    as-yet-unwritten-to-the-underlying-system data.
    Chris Torek, Feb 12, 2005
  16. JG

    Randy Howard Guest

    Unless it fails the first time and you retry? I don't see this
    as very likely, but ...
    Randy Howard, Feb 12, 2005
  17. Not if the cards have already fallen into the hopper the first time they
    were read! Not everything is a disk file...

    Chris C
    Chris Croughton, Feb 12, 2005
  18. JG

    Eric Sosman Guest

    "Very simple?" Let's see you do it when reading
    from the keyboard. Or from a socket, or from a sensor
    connected to a serial port, or from /dev/random, or ...

    You've been given a solution that works for streams
    attached to reopenable data sources (seekability is helpful,
    but not essential). Use it, and stop whining!
    Eric Sosman, Feb 12, 2005
  19. JG

    jschultz Guest

    I imagine on a non-random seek stream (socket, serial port, terminal,
    /dev/random, etc.) that such a function would discard whatever it has
    already read in -- say, for example, if you are using line buffering
    instead of block buffering this could be useful.

    To make file streams consistent with "real" streams, the file position
    should probably be advanced by how ever many bytes were discarded and
    on such streams could be ascertained with a call to ftell or fgetpos.

    Anyway, immature and snide remarks aside, it would very easy to do this
    and would prevent suprising results from pedagoguish code like this:

    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <time.h>

    #define RECORD_SIZE 100

    int main(int argc, char **argv)
    FILE * writer;
    FILE * reader;
    char write_buf[RECORD_SIZE];
    char read_buf[RECORD_SIZE];
    int i;


    if (argc != 2) {
    exit(fprintf(stderr, "Usage: %s <file_out>\r\n", argv[0]));

    if ((writer = fopen(argv[1], "r+")) == NULL) {
    exit(fprintf(stderr, "fopen '%s' w/ r+ failed!\r\n", argv[1]));

    if ((reader = fopen(argv[1], "r")) == NULL) {
    exit(fprintf(stderr, "fopen '%s' w/ r failed!\r\n", argv[1]));

    while (1) {

    for (i = 0; i < RECORD_SIZE; ++i) {
    write_buf = (char) rand();

    if (fwrite(write_buf, RECORD_SIZE, 1, writer) != 1 ||
    fflush(writer) != 0) {
    exit(fprintf(stderr, "fwrite/fflush failed!\r\n"));

    /* for random seek media -- insert fpurge/fseek call here */

    if (fread(read_buf, RECORD_SIZE, 1, reader) != 1) {
    exit(fprintf(stderr, "fread failed!\r\n"));

    if (memcmp(read_buf, write_buf, RECORD_SIZE) != 0) {

    return 0;

    The output file must exist before running. If the output file <=
    RECORD_SIZE bytes long then the program will run. Otherwise it will
    immediately fail due to cache incoherence caused by the C library's
    read ahead caching of "junk" data from the file's previous contents.

    That would be fine, IF the C standard offerred a way around this
    problem without the expensive operation of tearing down and rebuilding
    the stream. The fact that it doesn't is suprising to me.
    jschultz, Feb 14, 2005
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.