tailing a log file using c ...

Discussion in 'C Programming' started by Tony Stock, Nov 20, 2003.

  1. Tony Stock

    Tony Stock Guest

    Hi all,

    problem - need to read a log (text) file looking for certain entries,
    this file will still be written to by another 3rd party process, and
    hence constantly require monitoring (dare I say tailing ?)

    Is it possible / practical / efficient to 'tail' a log file using c.


    Questions are :

    1. If I was reading file and I reached EOF, would I see new file
    writes - or would I have to re-open file to see new additions to it ?

    2. Would I use unix 'tail' command fed into a c program ?

    3. Any advice appreciated ?

    Cheers

    Tony
     
    Tony Stock, Nov 20, 2003
    #1
    1. Advertising

  2. "Tony Stock" <> schrieb im Newsbeitrag
    news:...
    > 1. If I was reading file and I reached EOF, would I see new file
    > writes - or would I have to re-open file to see new additions to it ?


    Depends on the operating system. On many UNIXes, and on Windows, yes, you
    will see the new writes.

    You have to open the file in a shared mode using unbuffered I/O (open(),
    read(), close()).

    Copy the data from read() to an internal buffer and print it when it's ready
    (complete line).

    Keep the file open and attempt to read() beyond the EOF. If an error is
    returned, no new data is available, otherwise read as much as you can
    process.


    > 2. Would I use unix 'tail' command fed into a c program ?


    You could also do that, calling system() to issue a tail command and
    retrieving its output. But the other solution is simpler. :)
     
    Ekkehard Morgenstern, Nov 20, 2003
    #2
    1. Advertising

  3. Tony Stock

    Simon Biber Guest

    "Ekkehard Morgenstern" <> wrote:
    > "Tony Stock" <> schrieb:
    > > 1. If I was reading file and I reached EOF, would I see new
    > > file writes - or would I have to re-open file to see new
    > > additions to it ?


    It's not guaranteed in standard C. When you open the file it may be
    completely locked down so no other program could append log entries
    to it.

    > Depends on the operating system. On many UNIXes, and on Windows,
    > yes, you will see the new writes.
    >
    > You have to open the file in a shared mode using unbuffered I/O
    > (open(), read(), close()).


    Unbuffered I/O is not part of Standard C, and is off-topic on comp.lang.c.
    The open, read, close functions can be discussed on comp.unix.programmer,
    even if you are writing code for Windows, since they are POSIX functions.

    --
    Simon.
     
    Simon Biber, Nov 21, 2003
    #3
  4. In article <3fbd5911$0$13681$>,
    Simon Biber <> wrote:
    >
    >Unbuffered I/O is not part of Standard C, and is off-topic on comp.lang.c.
    >The open, read, close functions can be discussed on comp.unix.programmer,
    >even if you are writing code for Windows, since they are POSIX functions.


    I don't think that true. The standard defines setvbuf() to set/change
    the buffering mode of a stream.

    #include <stdio.h>
    int setvbuf(FILE *stream, char *buf, int mode , size_t size);

    --
    Rouben Rostamian
     
    Rouben Rostamian, Nov 21, 2003
    #4
  5. (Tony Stock) wrote:
    >Hi all,
    >
    >problem - need to read a log (text) file looking for certain entries,
    >this file will still be written to by another 3rd party process, and
    >hence constantly require monitoring (dare I say tailing ?)
    >
    >Is it possible / practical / efficient to 'tail' a log file using c.


    Yes.

    >Questions are :
    >
    >1. If I was reading file and I reached EOF, would I see new file
    >writes - or would I have to re-open file to see new additions to it ?


    If you clear the eof condition and re-read the file, you'll see
    new data.

    >2. Would I use unix 'tail' command fed into a c program ?


    If you had asked this on comp.unix.programmer the answer would
    be to read the man page for popen(), but...

    >3. Any advice appreciated ?


    #include <stdio.h>
    #include <stdlib.h>
    #include <ctype.h>

    int main(int argc, char *argv[])
    {
    char buf[BUFSIZ];
    fpos_t pos;
    FILE *fp;

    if (argc != 2) {
    exit(EXIT_FAILURE);
    }

    if (NULL == (fp = fopen(argv[1], "r"))) {
    fprintf(stderr,"Cannot find file: %s\n", argv[1]);
    exit(EXIT_FAILURE);
    }

    if ( 0 != fseek(fp, 0, SEEK_END)) {
    fprintf(stderr,"Cannot read file: %s\n", argv[1]);
    exit(EXIT_FAILURE);
    }

    if (-1 == fgetpos(fp, &pos)) {
    fprintf(stderr,"Cannot read file: %s\n", argv[1]);
    exit(EXIT_FAILURE);
    }

    for (;;) {
    fgets(buf, BUFSIZ, fp);
    if (ferror(fp)) {
    fprintf(stderr,"Error reading file: %s\n", argv[1]);
    exit(EXIT_FAILURE);
    }
    if (feof(fp)) {
    fsetpos(fp, &pos);
    clearerr(fp);
    #if 0
    /* non-ansi code! */
    sleep(1); /* give up the cpu to other processes */
    #endif
    continue;
    }
    fgetpos(fp, &pos);
    printf("%s", buf);
    fflush(stdout);
    }

    return EXIT_SUCCESS;
    }


    --
    Floyd L. Davidson <http://web.newsguy.com/floyd_davidson>
    Ukpeagvik (Barrow, Alaska)
     
    Floyd Davidson, Nov 21, 2003
    #5
  6. Tony Stock

    T.M. Sommers Guest

    Tony Stock wrote:
    > Hi all,
    >
    > problem - need to read a log (text) file looking for certain entries,
    > this file will still be written to by another 3rd party process, and
    > hence constantly require monitoring (dare I say tailing ?)
    >
    > Is it possible / practical / efficient to 'tail' a log file using c.
    >
    > Questions are :
    >
    > 1. If I was reading file and I reached EOF, would I see new file
    > writes - or would I have to re-open file to see new additions to it ?
    >
    > 2. Would I use unix 'tail' command fed into a c program ?


    Off-topic here, but 'tail -f logfile | your_program' would be the
    simplest, and most flexible, solution. Or use the syslog facility.
     
    T.M. Sommers, Nov 21, 2003
    #6
  7. Tony Stock

    Tony Stock Guest

    Thanks for the responses - food for thought.

    It will be on a unix platform (unknown at present though)

    Just a little confused, one poster said that it needs to be UNBUFFERED
    io (i.e. open / read / close), but the sample code posted uses a
    normal fileopen (fopen) ????


    Thanks

    Tony
     
    Tony Stock, Nov 21, 2003
    #7
  8. (Tony Stock) wrote:
    >Thanks for the responses - food for thought.
    >
    >It will be on a unix platform (unknown at present though)
    >
    >Just a little confused, one poster said that it needs to be UNBUFFERED
    >io (i.e. open / read / close), but the sample code posted uses a
    >normal fileopen (fopen) ????


    Data will be delivered in chunks, and the specifics depend on
    whether you use setvbuf (or not) to change the buffering.

    A file (as opposed to a terminal) accessed by fopen() will default
    to block buffering; hence, you won't see any changes until the
    total equals the block size (probably BUFSIZ, defined in stdio.h,
    and likely to be 4k or 8k.).

    If your logs are line oriented (almost certainly true) you'll
    probably want to use setvbuf to change access to line buffered, in
    which case each newline will cause data to be available. Otherwise
    a single line log entry might not be available until other log
    entries occur in sufficient numbers to fill the buffer when block
    buffering is used.

    The other alternative is to turn off buffering (or use non-portable
    low level unbuffered i/o functions), in which case the size of the
    chunk will be 1 character, and you're program can happily thrash
    away at each and every byte...

    --
    Floyd L. Davidson <http://web.newsguy.com/floyd_davidson>
    Ukpeagvik (Barrow, Alaska)
     
    Floyd Davidson, Nov 21, 2003
    #8
  9. Tony Stock

    Dan Pop Guest

    In <bpjre4$vu0$> (Rouben Rostamian) writes:

    >In article <3fbd5911$0$13681$>,
    >Simon Biber <> wrote:
    >>
    >>Unbuffered I/O is not part of Standard C, and is off-topic on comp.lang.c.
    >>The open, read, close functions can be discussed on comp.unix.programmer,
    >>even if you are writing code for Windows, since they are POSIX functions.

    >
    >I don't think that true. The standard defines setvbuf() to set/change
    >the buffering mode of a stream.
    >
    > #include <stdio.h>
    > int setvbuf(FILE *stream, char *buf, int mode , size_t size);


    This only suppresses the buffering performed by the standard C library.
    Sometimes, you may need to suppress the buffering performed by the OS,
    too, and setvbuf is very unlikely to help (although, in theory, it could).

    But when people talk about unbuffered I/O (or raw I/O or low level I/O)
    they usually mean I/O performed using the OS primitives, rather than the
    (relatively high level) <stdio.h> stuff.

    Dan
    --
    Dan Pop
    DESY Zeuthen, RZ group
    Email:
     
    Dan Pop, Nov 21, 2003
    #9
  10. Floyd Davidson <> wrote:
    > (Tony Stock) wrote:
    >>Thanks for the responses - food for thought.
    >>
    >>It will be on a unix platform (unknown at present though)
    >>
    >>Just a little confused, one poster said that it needs to be UNBUFFERED
    >>io (i.e. open / read / close), but the sample code posted uses a
    >>normal fileopen (fopen) ????

    >
    >Data will be delivered in chunks, and the specifics depend on
    >whether you use setvbuf (or not) to change the buffering.
    >
    >A file (as opposed to a terminal) accessed by fopen() will default
    >to block buffering; hence, you won't see any changes until the
    >total equals the block size (probably BUFSIZ, defined in stdio.h,
    >and likely to be 4k or 8k.).
    >
    >If your logs are line oriented (almost certainly true) you'll
    >probably want to use setvbuf to change access to line buffered, in
    >which case each newline will cause data to be available. Otherwise
    >a single line log entry might not be available until other log
    >entries occur in sufficient numbers to fill the buffer when block
    >buffering is used.
    >
    >The other alternative is to turn off buffering (or use non-portable
    >low level unbuffered i/o functions), in which case the size of the
    >chunk will be 1 character, and you're program can happily thrash
    >away at each and every byte...


    Well, all of that sounds good... but it doesn't do that.

    The code that I posted uses fgets() for input, and acts as described
    for line buffering because of that. By changing the input to fread()
    behavior as described for unbuffered input can be obtained.

    Each read either reaches the end of a line or the end of the
    buffer (which ever is important for the mode used) _or_ it hits
    an end of file condition. Hence the new data is always available.

    --
    Floyd L. Davidson <http://web.newsguy.com/floyd_davidson>
    Ukpeagvik (Barrow, Alaska)
     
    Floyd Davidson, Nov 21, 2003
    #10
  11. Tony Stock

    CBFalconer Guest

    Tony Stock wrote:
    >
    > Thanks for the responses - food for thought.
    >
    > It will be on a unix platform (unknown at present though)
    >
    > Just a little confused, one poster said that it needs to be UNBUFFERED
    > io (i.e. open / read / close), but the sample code posted uses a
    > normal fileopen (fopen) ????


    This is OT on c.l.c. but FYI here is an extract from the info file
    for GNU tail:

    > `-f'
    > `--follow[=HOW]'
    > Loop forever trying to read more characters at the end of the file,
    > presumably because the file is growing. This option is ignored
    > when reading from a pipe. If more than one file is given, `tail'
    > prints a header whenever it gets output from a different file, to
    > indicate which file that output is from.
    >
    > There are two ways to specify how you'd like to track files with
    > this option, but that difference is noticeable only when a
    > followed file is removed or renamed. If you'd like to continue to
    > track the end of a growing file even after it has been unlinked,
    > use `--follow=descriptor'. This is the default behavior, but it
    > is not useful if you're tracking a log file that may be rotated
    > (removed or renamed, then reopened). In that case, use
    > `--follow=name' to track the named file by reopening it
    > periodically to see if it has been removed and recreated by some
    > other program.


    --
    Chuck F () ()
    Available for consulting/temporary embedded and systems.
    <http://cbfalconer.home.att.net> USE worldnet address!
     
    CBFalconer, Nov 21, 2003
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. lebo
    Replies:
    0
    Views:
    353
  2. jalkadir
    Replies:
    6
    Views:
    445
    jalkadir
    Oct 15, 2005
  3. Evan Klitzke

    Tailing a log file?

    Evan Klitzke, Jun 22, 2007, in forum: Python
    Replies:
    0
    Views:
    294
    Evan Klitzke
    Jun 22, 2007
  4. Kenji Noguchi

    Re: Tailing a log file?

    Kenji Noguchi, Jun 22, 2007, in forum: Python
    Replies:
    2
    Views:
    379
  5. SamXiao

    Tailing a series of log files

    SamXiao, Jun 12, 2013, in forum: Java
    Replies:
    8
    Views:
    270
    Arved Sandstrom
    Jun 16, 2013
Loading...

Share This Page