Two questions about logging

[Matthew Pounsett]

I'm trying to figure out a couple of things with the logging module,
and I'm hoping someone can provide some pointers. I've read through
the module docs on python.org, the basic and advanced tutorials, and
the cookbook post, but a couple of things still elude me.

First, I'd like to be able to permit users to do more typical log
rotation, based on their OS's log rotation handler, rather than
rotating logs from inside an application. This is usually handled by
signalling an application with a HUP, whereupon it closes and then re-
opens all of its logs, getting new file handles (and new inodes). I
don't see anything in the Handler methods (or anywhere else) that
would let me tell a logger object to refresh the file handles on a log
file. Is there some standard way to deal with this?

Second, I'm trying to get a handle on how libraries are meant to
integrate with the applications that use them. The naming advice in
the advanced tutorial is to use __name__ to name loggers, and to allow
log messages to pass back up to the using application's logger for
processing, but these two pieces of advice seem contradictory.. since
log messages only pass back up to the root if the loggers are named
hierarchically.

cat foo.py

#!/usr/bin/env python

import logging
import bar

logger = logging.getLogger(__name__)
fh = logging.FileHandler('foo.log')
ch = logging.StreamHandler()
logger.addHandler(fh)
logger.addHandler(ch)
logger.warning('from the first')
bar.funct()

cat bar.py

#!/usr/bin/env python

import logging

def funct():
logger = logging.getLogger(__name__)
logger.warning('from the second')

./foo.py

from the first
No handlers could be found for logger "bar"

So, given this state of affairs, how is a library author to use
loggers, given that he or she can't know how authors who use the
library will name their logger objects? In the above example, what
would the author of bar.py do to hook up bar's logger with foo's,
without knowing in advance what foo's logger will be named?


Thanks very much for any suggestions, or pointers to documentation
that I've missed.

 

[Roy Smith]

First, I'd like to be able to permit users to do more typical log
rotation, based on their OS's log rotation handler, rather than
rotating logs from inside an application. This is usually handled by
signalling an application with a HUP, whereupon it closes and then re-
opens all of its logs, getting new file handles (and new inodes). I
don't see anything in the Handler methods (or anywhere else) that
would let me tell a logger object to refresh the file handles on a log
file. Is there some standard way to deal with this?

What I would do is log to syslog (logging.handlers.SysLogHandler) and
let syslog worry about rotating log files. Why reinvent the wheel?

 

[Matthew Pounsett]

What I would do is log to syslog (logging.handlers.SysLogHandler) and
let syslog worry about rotating log files.  Why reinvent the wheel?

Syslog is fine for an application run by an administrator, but isn't
an option for a user.

 

[Jean-Michel Pichavant]

[snip]

Second, I'm trying to get a handle on how libraries are meant to
integrate with the applications that use them. The naming advice in
the advanced tutorial is to use __name__ to name loggers, and to allow
log messages to pass back up to the using application's logger for
processing, but these two pieces of advice seem contradictory.. since
log messages only pass back up to the root if the loggers are named
hierarchically.

[snip]

So, given this state of affairs, how is a library author to use
loggers, given that he or she can't know how authors who use the
library will name their logger objects? In the above example, what
would the author of bar.py do to hook up bar's logger with foo's,
without knowing in advance what foo's logger will be named?


Thanks very much for any suggestions, or pointers to documentation
that I've missed.

All loggers share at least one parent : the root logger. In the standard
way of using the loggers, the root logger is *the* logger responsible of
processing logs.


Considering that extlib is an external library from anApp, and intlib an
internal library to the anApp application:

People may expect logs that way:

anApp.extlib : INFO : a log message
anApp.intlib : INFO : another message

However, what will happen is :

extlib : INFO : message from the external lib
anApp.intlib : INFO : message from the internal lib

As you mentioned in your post, there is no way to 'attach' an external
lib to your application since the external lib knows nothing about the
application, but there is no need to attach it actually.


JM

 

[K Richard Pixley]

Second, I'm trying to get a handle on how libraries are meant to
integrate with the applications that use them. The naming advice in
the advanced tutorial is to use __name__ to name loggers, and to allow
log messages to pass back up to the using application's logger for
processing, but these two pieces of advice seem contradictory.. since
log messages only pass back up to the root if the loggers are named
hierarchically.

Here's the confusion. Each log named __name__ is under the root logger.
If you want them all, then catch them all with the root logger.

In foo.py, change getLogger(__name__) to getLogger(''). Only the
included modules need __name__. Or use two - one logger for setting
handlers based on the root logger, and another based on __name__ for
logging from the top level.

--rich

 

[Matthew Pounsett]

Here's the confusion.  Each log named __name__ is under the root logger..
  If you want them all, then catch them all with the root logger.

Thanks! I knew I was missing something obvious. Between you and Jean-
Michael Pichavant I've figured out what I need to do here.

What I would do is log to syslog (logging.handlers.SysLogHandler) and
let syslog worry about rotating log files. Why reinvent the wheel?

I've also worked out what I need to reset file handles, although it
took a lot of reading in the various logging.Handler subclasses. What
I needed isn't explicitly documented anywhere, but it turns out that
calling the close() method on a FileHandler instance does what I
need. There's no method to re-open the handler, but the next call to
emit() will automatically re-open the file if it isn't already open.

The upshot is that this does the expected thing if you rename its log
file and then send the running script a HUP signal.

#!/usr/bin/env python

import logging
import signal
import time

logger = logging.getLogger()
lh = logging.FileHandler('./foo.log')
lh.setFormatter(logging.Formatter('%(asctime)s %(name)s: %(message)s',
'%T'))
logger.addHandler(lh)

def sighup(signum, frame):
lh.close()
logger.error("handled {0}: {1}".format(signum, frame))

def main():
signal.signal(signal.SIGHUP, sighup)
while 1:
time.sleep(1)
logger.error('a message')

if __name__ == '__main__':
main()

 

[Vinay Sajip]

First, I'd like to be able to permit users to do more typical log
rotation, based on their OS's log rotation handler, rather than
rotating logs from inside an application.  This is usually handled by
signalling an application with a HUP, whereupon it closes and then re-
opens all of its logs, getting new file handles (and new inodes).  I
don't see anything in the Handler methods (or anywhere else) that
would let me tell a logger object to refresh the file handles on a log
file.  Is there some standard way to deal with this?

There's the WatchedFileHandler, which checks to see if a file's device
or inode has changed (which happens when the external rotator does
rotation) and if so, closes and reopens the file. (This handler is for
Unix/Linux/OS X - it doesn't work on Windows).

See

http://docs.python.org/library/logging.handlers.html#watchedfilehandler

for more information.

Regards,

Vinay Sajip