Two questions about logging

Discussion in 'Python' started by Matthew Pounsett, Jan 12, 2012.

  1. I'm trying to figure out a couple of things with the logging module,
    and I'm hoping someone can provide some pointers. I've read through
    the module docs on python.org, the basic and advanced tutorials, and
    the cookbook post, but a couple of things still elude me.

    First, I'd like to be able to permit users to do more typical log
    rotation, based on their OS's log rotation handler, rather than
    rotating logs from inside an application. This is usually handled by
    signalling an application with a HUP, whereupon it closes and then re-
    opens all of its logs, getting new file handles (and new inodes). I
    don't see anything in the Handler methods (or anywhere else) that
    would let me tell a logger object to refresh the file handles on a log
    file. Is there some standard way to deal with this?

    Second, I'm trying to get a handle on how libraries are meant to
    integrate with the applications that use them. The naming advice in
    the advanced tutorial is to use __name__ to name loggers, and to allow
    log messages to pass back up to the using application's logger for
    processing, but these two pieces of advice seem contradictory.. since
    log messages only pass back up to the root if the loggers are named
    hierarchically.

    > cat foo.py

    #!/usr/bin/env python

    import logging
    import bar

    logger = logging.getLogger(__name__)
    fh = logging.FileHandler('foo.log')
    ch = logging.StreamHandler()
    logger.addHandler(fh)
    logger.addHandler(ch)
    logger.warning('from the first')
    bar.funct()

    > cat bar.py

    #!/usr/bin/env python

    import logging

    def funct():
    logger = logging.getLogger(__name__)
    logger.warning('from the second')

    > ./foo.py

    from the first
    No handlers could be found for logger "bar"

    So, given this state of affairs, how is a library author to use
    loggers, given that he or she can't know how authors who use the
    library will name their logger objects? In the above example, what
    would the author of bar.py do to hook up bar's logger with foo's,
    without knowing in advance what foo's logger will be named?


    Thanks very much for any suggestions, or pointers to documentation
    that I've missed.
    Matthew Pounsett, Jan 12, 2012
    #1
    1. Advertising

  2. Matthew Pounsett

    Roy Smith Guest

    In article
    <>,
    Matthew Pounsett <> wrote:

    > First, I'd like to be able to permit users to do more typical log
    > rotation, based on their OS's log rotation handler, rather than
    > rotating logs from inside an application. This is usually handled by
    > signalling an application with a HUP, whereupon it closes and then re-
    > opens all of its logs, getting new file handles (and new inodes). I
    > don't see anything in the Handler methods (or anywhere else) that
    > would let me tell a logger object to refresh the file handles on a log
    > file. Is there some standard way to deal with this?


    What I would do is log to syslog (logging.handlers.SysLogHandler) and
    let syslog worry about rotating log files. Why reinvent the wheel?
    Roy Smith, Jan 12, 2012
    #2
    1. Advertising

  3. On Jan 11, 9:34 pm, Roy Smith <> wrote:
    > What I would do is log to syslog (logging.handlers.SysLogHandler) and
    > let syslog worry about rotating log files.  Why reinvent the wheel?


    Syslog is fine for an application run by an administrator, but isn't
    an option for a user.
    Matthew Pounsett, Jan 12, 2012
    #3
  4. Matthew Pounsett wrote:
    > [snip]
    >
    > Second, I'm trying to get a handle on how libraries are meant to
    > integrate with the applications that use them. The naming advice in
    > the advanced tutorial is to use __name__ to name loggers, and to allow
    > log messages to pass back up to the using application's logger for
    > processing, but these two pieces of advice seem contradictory.. since
    > log messages only pass back up to the root if the loggers are named
    > hierarchically.
    >
    > [snip]
    >
    > So, given this state of affairs, how is a library author to use
    > loggers, given that he or she can't know how authors who use the
    > library will name their logger objects? In the above example, what
    > would the author of bar.py do to hook up bar's logger with foo's,
    > without knowing in advance what foo's logger will be named?
    >
    >
    > Thanks very much for any suggestions, or pointers to documentation
    > that I've missed.
    >


    All loggers share at least one parent : the root logger. In the standard
    way of using the loggers, the root logger is *the* logger responsible of
    processing logs.


    Considering that extlib is an external library from anApp, and intlib an
    internal library to the anApp application:

    People may expect logs that way:

    anApp.extlib : INFO : a log message
    anApp.intlib : INFO : another message

    However, what will happen is :

    extlib : INFO : message from the external lib
    anApp.intlib : INFO : message from the internal lib

    As you mentioned in your post, there is no way to 'attach' an external
    lib to your application since the external lib knows nothing about the
    application, but there is no need to attach it actually.


    JM
    Jean-Michel Pichavant, Jan 12, 2012
    #4
  5. On 1/11/12 18:19 , Matthew Pounsett wrote:
    > Second, I'm trying to get a handle on how libraries are meant to
    > integrate with the applications that use them. The naming advice in
    > the advanced tutorial is to use __name__ to name loggers, and to allow
    > log messages to pass back up to the using application's logger for
    > processing, but these two pieces of advice seem contradictory.. since
    > log messages only pass back up to the root if the loggers are named
    > hierarchically.


    Here's the confusion. Each log named __name__ is under the root logger.
    If you want them all, then catch them all with the root logger.

    In foo.py, change getLogger(__name__) to getLogger(''). Only the
    included modules need __name__. Or use two - one logger for setting
    handlers based on the root logger, and another based on __name__ for
    logging from the top level.

    --rich
    K Richard Pixley, Jan 13, 2012
    #5
  6. On Jan 12, 8:03 pm, K Richard Pixley <> wrote:
    > Here's the confusion.  Each log named __name__ is under the root logger..
    >   If you want them all, then catch them all with the root logger.


    Thanks! I knew I was missing something obvious. Between you and Jean-
    Michael Pichavant I've figured out what I need to do here.


    On Jan 11, 9:34 pm, Roy Smith <> wrote:
    > What I would do is log to syslog (logging.handlers.SysLogHandler) and
    > let syslog worry about rotating log files. Why reinvent the wheel?


    I've also worked out what I need to reset file handles, although it
    took a lot of reading in the various logging.Handler subclasses. What
    I needed isn't explicitly documented anywhere, but it turns out that
    calling the close() method on a FileHandler instance does what I
    need. There's no method to re-open the handler, but the next call to
    emit() will automatically re-open the file if it isn't already open.

    The upshot is that this does the expected thing if you rename its log
    file and then send the running script a HUP signal.

    #!/usr/bin/env python

    import logging
    import signal
    import time

    logger = logging.getLogger()
    lh = logging.FileHandler('./foo.log')
    lh.setFormatter(logging.Formatter('%(asctime)s %(name)s: %(message)s',
    '%T'))
    logger.addHandler(lh)

    def sighup(signum, frame):
    lh.close()
    logger.error("handled {0}: {1}".format(signum, frame))

    def main():
    signal.signal(signal.SIGHUP, sighup)
    while 1:
    time.sleep(1)
    logger.error('a message')

    if __name__ == '__main__':
    main()
    Matthew Pounsett, Jan 15, 2012
    #6
  7. Matthew Pounsett

    Vinay Sajip Guest

    On Jan 12, 2:19 am, Matthew Pounsett <> wrote:

    > First, I'd like to be able to permit users to do more typical log
    > rotation, based on their OS's log rotation handler, rather than
    > rotating logs from inside an application.  This is usually handled by
    > signalling an application with a HUP, whereupon it closes and then re-
    > opens all of its logs, getting new file handles (and new inodes).  I
    > don't see anything in the Handler methods (or anywhere else) that
    > would let me tell a logger object to refresh the file handles on a log
    > file.  Is there some standard way to deal with this?


    There's the WatchedFileHandler, which checks to see if a file's device
    or inode has changed (which happens when the external rotator does
    rotation) and if so, closes and reopens the file. (This handler is for
    Unix/Linux/OS X - it doesn't work on Windows).

    See

    http://docs.python.org/library/logging.handlers.html#watchedfilehandler

    for more information.

    Regards,

    Vinay Sajip
    Vinay Sajip, Jan 16, 2012
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Stefan Siegl
    Replies:
    0
    Views:
    941
    Stefan Siegl
    Aug 27, 2003
  2. janne
    Replies:
    0
    Views:
    9,448
    janne
    Sep 10, 2004
  3. Christoph Haas
    Replies:
    0
    Views:
    431
    Christoph Haas
    Jun 12, 2006
  4. Christoph Haas
    Replies:
    1
    Views:
    441
    Vinay Sajip
    Jun 14, 2006
  5. GenxLogic
    Replies:
    3
    Views:
    1,243
    andrewmcdonagh
    Dec 6, 2006
Loading...

Share This Page