Tailing a series of log files

Discussion in 'Java' started by SamXiao, Jun 12, 2013.

  1. SamXiao

    SamXiao Guest

    Hi all,

    I am working on a solution to keep track of a series of log files and read the tail lines. At the very beginning, there is only one log file named “TEST-YYYYMMDD-00.log” (where YYYYMMDD is the current date). When the sizeof this log reaches to 10 MB, another log file “TEST-YYYYMMDD-01.log” will be generated and so forth. After midnight (12:00), a new log file withthe next date starting with 00 again will be generated.

    I know Apache Commons IO containing tail.java. However, seems it is for tailing a single log file only. How could I keep tailing a series of log files?

    Any help would be highly appreciated. Thanks.
    SamXiao, Jun 12, 2013
    #1
    1. Advertising

  2. SamXiao

    Eric Sosman Guest

    On 6/12/2013 8:04 AM, SamXiao wrote:
    > Hi all,
    >
    > I am working on a solution to keep track of a series of log files and read the tail lines. At the very beginning, there is only one log file named “TEST-YYYYMMDD-00.log” (where YYYYMMDD is the current date). When the size of this log reaches to 10 MB, another log file “TEST-YYYYMMDD-01.log” will be generated and so forth. After midnight (12:00), a new log file with the next date starting with 00 again will be generated.
    >
    > I know Apache Commons IO containing tail.java. However, seems it is for tailing a single log file only. How could I keep tailing a series of log files?
    >
    > Any help would be highly appreciated. Thanks.


    It appears you need to monitor the log files' directory for
    the creation of new files. When a new file appears you know
    that the log you've been tailing will cease to grow, so you can
    stop tailing it and switch your attention to the new one. Maybe
    you should continue tailing the old for a few seconds before
    making the switch (just in case a final buffer needs to get
    flushed or something), but I think that should do it.

    I don't know whether anybody's already packaged such a thing
    in a form you can just pick up and use, but it seems straightforward
    enough. Take a look at

    http://docs.oracle.com/javase/tutorial/essential/io/notification.html

    --
    Eric Sosman
    d
    Eric Sosman, Jun 12, 2013
    #2
    1. Advertising

  3. On 06/12/2013 09:04 AM, SamXiao wrote:
    > Hi all,
    >
    > I am working on a solution to keep track of a series of log files and read the tail lines. At the very beginning, there is only one log file named “TEST-YYYYMMDD-00.log” (where YYYYMMDD is the current date). When the size of this log reaches to 10 MB, another log file “TEST-YYYYMMDD-01.log” will be generated and so forth. After midnight (12:00), a new log file with the next date starting with 00 again will be generated.
    >
    > I know Apache Commons IO containing tail.java. However, seems it is for tailing a single log file only. How could I keep tailing a series of log files?
    >
    > Any help would be highly appreciated. Thanks.
    >

    For what it's worth, a sane logging system with a rollover policy has a
    base log file which always has the same name, like TEST.log. Decent text
    editors can follow a filename, UNIX command line "tail" doesn't follow a
    filename but rather the precise file in question. But point being, if
    you're looking to follow an "active" log file it may help if it's always
    got the same name. It's the rolled-over logs that have the suffixes.

    AHS

    --
    When a true genius appears, you can know him by this sign:
    that all the dunces are in a confederacy against him.
    -- Jonathan Swift
    Arved Sandstrom, Jun 15, 2013
    #3
  4. SamXiao

    Eric Sosman Guest

    On 6/14/2013 7:00 PM, Arved Sandstrom wrote:
    > On 06/12/2013 09:04 AM, SamXiao wrote:
    >> Hi all,
    >>
    >> I am working on a solution to keep track of a series of log files and
    >> read the tail lines. At the very beginning, there is only one log file
    >> named “TEST-YYYYMMDD-00.log” (where YYYYMMDD is the current date).
    >> When the size of this log reaches to 10 MB, another log file
    >> “TEST-YYYYMMDD-01.log” will be generated and so forth. After midnight
    >> (12:00), a new log file with the next date starting with 00 again will
    >> be generated.
    >>
    >> I know Apache Commons IO containing tail.java. However, seems it is
    >> for tailing a single log file only. How could I keep tailing a series
    >> of log files?
    >>
    >> Any help would be highly appreciated. Thanks.
    >>

    > For what it's worth, a sane logging system with a rollover policy has a
    > base log file which always has the same name, like TEST.log. Decent text
    > editors can follow a filename, UNIX command line "tail" doesn't follow a
    > filename but rather the precise file in question. But point being, if
    > you're looking to follow an "active" log file it may help if it's always
    > got the same name. It's the rolled-over logs that have the suffixes.


    This is true, but I don't think it's going to help the
    O.P. much. Argument: The tailer (or any program) will use
    the name to open the file, but subsequent accesses will use a
    name-independent means -- "vnode," perhaps, or "handle." If
    the name-to-vnode/handle mapping changes after the program has
    created its connection to the file, that's unlikely to affect
    the existing connection: The program will, most likely, just
    keep on tailing the original, oblivious to the name change.

    A tailer might, I suppose, open TEST.log anew for each
    access: open, tail, close, pause, open, tail, close, pause,
    and so on. That approach would catch the renames, but would
    also risk missing the last few lines of the pre-rename file:

    Tailer: open, tail, close, pause, open, tail, close, ...
    Writer: ... write ... write ... rename ... write ...
    Upshot: ^^^^^
    this batch isn't tailed

    If the log writer switches from file to file occasionally,
    I think the only reliable approach is to have the log reader be
    explicitly aware of the fact.

    --
    Eric Sosman
    d
    Eric Sosman, Jun 15, 2013
    #4
  5. SamXiao

    Daniel Pitts Guest

    On 6/14/13 6:06 PM, Eric Sosman wrote:
    > On 6/14/2013 7:00 PM, Arved Sandstrom wrote:
    >> On 06/12/2013 09:04 AM, SamXiao wrote:
    >>> Hi all,
    >>>
    >>> I am working on a solution to keep track of a series of log files and
    >>> read the tail lines. At the very beginning, there is only one log file
    >>> named “TEST-YYYYMMDD-00.log” (where YYYYMMDD is the current date).
    >>> When the size of this log reaches to 10 MB, another log file
    >>> “TEST-YYYYMMDD-01.log” will be generated and so forth. After midnight
    >>> (12:00), a new log file with the next date starting with 00 again will
    >>> be generated.
    >>>
    >>> I know Apache Commons IO containing tail.java. However, seems it is
    >>> for tailing a single log file only. How could I keep tailing a series
    >>> of log files?
    >>>
    >>> Any help would be highly appreciated. Thanks.
    >>>

    >> For what it's worth, a sane logging system with a rollover policy has a
    >> base log file which always has the same name, like TEST.log. Decent text
    >> editors can follow a filename, UNIX command line "tail" doesn't follow a
    >> filename but rather the precise file in question. But point being, if
    >> you're looking to follow an "active" log file it may help if it's always
    >> got the same name. It's the rolled-over logs that have the suffixes.

    >
    > This is true, but I don't think it's going to help the
    > O.P. much. Argument: The tailer (or any program) will use
    > the name to open the file, but subsequent accesses will use a
    > name-independent means -- "vnode," perhaps, or "handle." If
    > the name-to-vnode/handle mapping changes after the program has
    > created its connection to the file, that's unlikely to affect
    > the existing connection: The program will, most likely, just
    > keep on tailing the original, oblivious to the name change.
    >
    > A tailer might, I suppose, open TEST.log anew for each
    > access: open, tail, close, pause, open, tail, close, pause,
    > and so on. That approach would catch the renames, but would
    > also risk missing the last few lines of the pre-rename file:
    >
    > Tailer: open, tail, close, pause, open, tail, close, ...
    > Writer: ... write ... write ... rename ... write ...
    > Upshot: ^^^^^
    > this batch isn't tailed
    >
    > If the log writer switches from file to file occasionally,
    > I think the only reliable approach is to have the log reader be
    > explicitly aware of the fact.
    >


    from "man tail" on OS X:

    > -F The -F option implies the -f option, but tail will also check to see if the file being followed has been renamed or rotated. The file is closed and
    > reopened when tail detects that the filename being read from has a new inode number. The -F option is ignored if reading from standard input rather than a
    > file.



    So, it will reopen the file if it sees that it has a new inode. If you
    are renaming the files frequently enough, that might miss a rename, but
    that seems to be an unlikely situation.
    Daniel Pitts, Jun 15, 2013
    #5
  6. SamXiao

    Eric Sosman Guest

    On 6/15/2013 12:11 AM, Daniel Pitts wrote:
    > On 6/14/13 6:06 PM, Eric Sosman wrote:
    >> On 6/14/2013 7:00 PM, Arved Sandstrom wrote:
    >>> On 06/12/2013 09:04 AM, SamXiao wrote:
    >>>> Hi all,
    >>>>
    >>>> I am working on a solution to keep track of a series of log files and
    >>>> read the tail lines. At the very beginning, there is only one log file
    >>>> named “TEST-YYYYMMDD-00.log” (where YYYYMMDD is the current date).
    >>>> When the size of this log reaches to 10 MB, another log file
    >>>> “TEST-YYYYMMDD-01.log” will be generated and so forth. After midnight
    >>>> (12:00), a new log file with the next date starting with 00 again will
    >>>> be generated.
    >>>>
    >>>> I know Apache Commons IO containing tail.java. However, seems it is
    >>>> for tailing a single log file only. How could I keep tailing a series
    >>>> of log files?
    >>>>
    >>>> Any help would be highly appreciated. Thanks.
    >>>>
    >>> For what it's worth, a sane logging system with a rollover policy has a
    >>> base log file which always has the same name, like TEST.log. Decent text
    >>> editors can follow a filename, UNIX command line "tail" doesn't follow a
    >>> filename but rather the precise file in question. But point being, if
    >>> you're looking to follow an "active" log file it may help if it's always
    >>> got the same name. It's the rolled-over logs that have the suffixes.

    >>
    >> This is true, but I don't think it's going to help the
    >> O.P. much. Argument: The tailer (or any program) will use
    >> the name to open the file, but subsequent accesses will use a
    >> name-independent means -- "vnode," perhaps, or "handle." If
    >> the name-to-vnode/handle mapping changes after the program has
    >> created its connection to the file, that's unlikely to affect
    >> the existing connection: The program will, most likely, just
    >> keep on tailing the original, oblivious to the name change.
    >>
    >> A tailer might, I suppose, open TEST.log anew for each
    >> access: open, tail, close, pause, open, tail, close, pause,
    >> and so on. That approach would catch the renames, but would
    >> also risk missing the last few lines of the pre-rename file:
    >>
    >> Tailer: open, tail, close, pause, open, tail, close, ...
    >> Writer: ... write ... write ... rename ... write ...
    >> Upshot: ^^^^^
    >> this batch isn't tailed
    >>
    >> If the log writer switches from file to file occasionally,
    >> I think the only reliable approach is to have the log reader be
    >> explicitly aware of the fact.
    >>

    >
    > from "man tail" on OS X:
    >
    >> -F The -F option implies the -f option, but tail will also
    >> check to see if the file being followed has been renamed or rotated.
    >> The file is closed and
    >> reopened when tail detects that the filename being read
    >> from has a new inode number. The -F option is ignored if reading from
    >> standard input rather than a
    >> file.

    >
    >
    > So, it will reopen the file if it sees that it has a new inode. If you
    > are renaming the files frequently enough, that might miss a rename, but
    > that seems to be an unlikely situation.


    We're straying into system specifics here, but I think the
    quoted text makes my point. On Unixoid systems (where inodes
    live), renaming a file does not affect its inode number -- in
    fact, a file can have N different names in M directories and
    keep the same inode, unaltered, as those N names come and go
    and change.

    Since checking the inode number of an already-open file
    will not detect a rename (or even a deletion!), the program
    must be doing something else instead. I imagine it's looking
    up the file name (probably with a variant of stat) and comparing
    the inode number against that of the already-open file; if they
    disagree, somebody has renamed the current file and it's time
    to switch to the new one. That's what I meant by "have the
    log reader be explicitly aware of the fact:" It's got to poll
    or arrange for change notification in some other way.

    --
    Eric Sosman
    d
    Eric Sosman, Jun 15, 2013
    #6
  7. SamXiao

    JLP Guest

    Le 15/06/2013 15:20, Eric Sosman a écrit :
    > On 6/15/2013 12:11 AM, Daniel Pitts wrote:
    >> On 6/14/13 6:06 PM, Eric Sosman wrote:
    >>> On 6/14/2013 7:00 PM, Arved Sandstrom wrote:
    >>>> On 06/12/2013 09:04 AM, SamXiao wrote:
    >>>>> Hi all,
    >>>>>
    >>>>> I am working on a solution to keep track of a series of log files and
    >>>>> read the tail lines. At the very beginning, there is only one log file
    >>>>> named “TEST-YYYYMMDD-00.log” (where YYYYMMDD is the current date).
    >>>>> When the size of this log reaches to 10 MB, another log file
    >>>>> “TEST-YYYYMMDD-01.log” will be generated and so forth. After midnight
    >>>>> (12:00), a new log file with the next date starting with 00 again will
    >>>>> be generated.
    >>>>>
    >>>>> I know Apache Commons IO containing tail.java. However, seems it is
    >>>>> for tailing a single log file only. How could I keep tailing a series
    >>>>> of log files?
    >>>>>
    >>>

    [SNIP because of NNTP server]
    >

    if I understand the OP requirements, a solution may be the pattern
    Observer/Observable :
    - a thread Observable that follows the creation of file
    TEST-[\d]{6}(-[^\.])?.log . When a new file is created motify the thread
    Observer with the new file log
    - a thread Observer subscribed to the Observable that tails the last
    file log and rotate when notified
    http://en.wikipedia.org/wiki/Observer_pattern
    JLP, Jun 15, 2013
    #7
  8. SamXiao

    JLP Guest

    Le 15/06/2013 18:59, JLP a écrit :
    > Le 15/06/2013 15:20, Eric Sosman a écrit :
    >> On 6/15/2013 12:11 AM, Daniel Pitts wrote:
    >>> On 6/14/13 6:06 PM, Eric Sosman wrote:
    >>>> On 6/14/2013 7:00 PM, Arved Sandstrom wrote:
    >>>>> On 06/12/2013 09:04 AM, SamXiao wrote:
    >>>>>> Hi all,
    >>>>>>
    >>>>>> I am working on a solution to keep track of a series of log files and
    >>>>>> read the tail lines. At the very beginning, there is only one log
    >>>>>> file
    >>>>>> named “TEST-YYYYMMDD-00.log” (where YYYYMMDD is the current date).
    >>>>>> When the size of this log reaches to 10 MB, another log file
    >>>>>> “TEST-YYYYMMDD-01.log” will be generated and so forth. After midnight
    >>>>>> (12:00), a new log file with the next date starting with 00 again
    >>>>>> will
    >>>>>> be generated.
    >>>>>>
    >>>>>> I know Apache Commons IO containing tail.java. However, seems it is
    >>>>>> for tailing a single log file only. How could I keep tailing a series
    >>>>>> of log files?
    >>>>>>
    >>>>

    > [SNIP because of NNTP server]
    >>

    > if I understand the OP requirements, a solution may be the pattern
    > Observer/Observable :
    > - a thread Observable that follows the creation of file
    > TEST-[\d]{6}(-[^\.])?.log . When a new file is created motify the thread
    > Observer with the new file log
    > - a thread Observer subscribed to the Observable that tails the last
    > file log and rotate when notified
    > http://en.wikipedia.org/wiki/Observer_pattern
    >

    oops correct pattern for the name of the file is :
    TEST-[\d]{8}-\d\d+\.log => the Class File gives the date of the last
    modification => File.lastmodified(); putting files in a TreeSet with a
    Comparator craeted with the lasModified date.
    JLP, Jun 15, 2013
    #8
  9. On 06/14/2013 10:06 PM, Eric Sosman wrote:
    > On 6/14/2013 7:00 PM, Arved Sandstrom wrote:
    >> On 06/12/2013 09:04 AM, SamXiao wrote:
    >>> Hi all,
    >>>
    >>> I am working on a solution to keep track of a series of log files and
    >>> read the tail lines. At the very beginning, there is only one log file
    >>> named “TEST-YYYYMMDD-00.log” (where YYYYMMDD is the current date).
    >>> When the size of this log reaches to 10 MB, another log file
    >>> “TEST-YYYYMMDD-01.log” will be generated and so forth. After midnight
    >>> (12:00), a new log file with the next date starting with 00 again will
    >>> be generated.
    >>>
    >>> I know Apache Commons IO containing tail.java. However, seems it is
    >>> for tailing a single log file only. How could I keep tailing a series
    >>> of log files?
    >>>
    >>> Any help would be highly appreciated. Thanks.
    >>>

    >> For what it's worth, a sane logging system with a rollover policy has a
    >> base log file which always has the same name, like TEST.log. Decent text
    >> editors can follow a filename, UNIX command line "tail" doesn't follow a
    >> filename but rather the precise file in question. But point being, if
    >> you're looking to follow an "active" log file it may help if it's always
    >> got the same name. It's the rolled-over logs that have the suffixes.

    >
    > This is true, but I don't think it's going to help the
    > O.P. much. Argument: The tailer (or any program) will use
    > the name to open the file, but subsequent accesses will use a
    > name-independent means -- "vnode," perhaps, or "handle." If
    > the name-to-vnode/handle mapping changes after the program has
    > created its connection to the file, that's unlikely to affect
    > the existing connection: The program will, most likely, just
    > keep on tailing the original, oblivious to the name change.
    >
    > A tailer might, I suppose, open TEST.log anew for each
    > access: open, tail, close, pause, open, tail, close, pause,
    > and so on. That approach would catch the renames, but would
    > also risk missing the last few lines of the pre-rename file:
    >
    > Tailer: open, tail, close, pause, open, tail, close, ...
    > Writer: ... write ... write ... rename ... write ...
    > Upshot: ^^^^^
    > this batch isn't tailed
    >
    > If the log writer switches from file to file occasionally,
    > I think the only reliable approach is to have the log reader be
    > explicitly aware of the fact.
    >

    I agree. There are a number of reasons why having a "current" logfile
    always with the same name helps, but not in this situation.

    A nice approach would be if loggers that can do rollover allow consumers
    to register as listeners. In a sense this is what JLPs suggestion
    achieves without logger knowledge.

    AHS

    --
    When a true genius appears, you can know him by this sign:
    that all the dunces are in a confederacy against him.
    -- Jonathan Swift
    Arved Sandstrom, Jun 16, 2013
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Tony Stock

    tailing a log file using c ...

    Tony Stock, Nov 20, 2003, in forum: C Programming
    Replies:
    10
    Views:
    667
    CBFalconer
    Nov 21, 2003
  2. lebo
    Replies:
    0
    Views:
    333
  3. jalkadir
    Replies:
    6
    Views:
    406
    jalkadir
    Oct 15, 2005
  4. Evan Klitzke

    Tailing a log file?

    Evan Klitzke, Jun 22, 2007, in forum: Python
    Replies:
    0
    Views:
    268
    Evan Klitzke
    Jun 22, 2007
  5. Kenji Noguchi

    Re: Tailing a log file?

    Kenji Noguchi, Jun 22, 2007, in forum: Python
    Replies:
    2
    Views:
    344
Loading...

Share This Page