ANN: Logging Package v0.4.9.4 released

[Vinay Sajip]

A new version of the Python logging package has been released. If you
are using version 2.3 of later of Python, this release will not mean
much to you - it is the same as recent checkins into Python CVS.
However, this release fixes numerous bugs reported since the last
independent release (0.4.9.2), and may be of interest to people using
earlier versions of Python.

What Does It Do?
================
The logging package offers the ability for any Python program to log
events which occur during program execution. It's typically used to
provide application diagnostics, warnings and error messages. In
addition to capturing "what happened", "where it happened", "when it
happened", "how important it is" and event specific data, you can
configure, without changing the application source code, both the
verbosity of logging and the routing of events to different
destinations such as console, disk files, sockets, email addresses,
Web servers, SOAP servers etc. etc. You can even change the
configuration of a running program without stopping and restarting it!

You can use the logging package in exception handlers, and it will
include traceback information in the log. It's thread-safe, and very
easy to use - just "import logging" and log away! Classes provided
include:

Logger - used to log messages for a particular area of an application.
You can instantiate these wherever you want, there's no need to pass
references to them around your application. You can log events based
on importance levels of DEBUG (least important), INFO, WARN, ERROR,
and CRITICAL (most important). You can define your own levels if the
default levels don't meet your requirements.

Handler - used to route events to particular destinations. Handlers
included are:
StreamHandler (for generalized streams, including console)
FileHandler (for disk files - including log file rotation with size
limits for log files)
SocketHandler (for sending events to a TCP socket)
DatagramHandler (for sending events to a UDP socket - faster, but
less reliable than TCP)
SMTPHandler (send events to arbitrary email addresses)
HTTPHandler (send events to Web servers)
SysLogHandler (send events to Unix syslog)
MemoryHandler (batch up events and process them several at a time)
NTEventLogHandler (send events to Windows NT event logs)

There are also examples of XMLHandler and SOAPHandler in the
distribution.

Formatter - used to format events as text strings. Flexible "msg %
arg_tuple" formatting is the basis for formatting, with flexible
date/time formatting including ISO8601 and any strftime-based formats.

Filter - used when filtering based on importance
(DEBUG/INFO/WARNING/ERROR/CRITICAL) is not sufficient. The
distribution includes examples of filters based on class matching,
regular expression matching, value matching, and logger matching.

In the unlikely event that you're daunted by all the apparent
complexity, fear not. The package offers a simple function-based
interface to allow very simple, almost casual use of the underlying
features.

In addition to the core logging functionality, you get the ability to
configure logging using a ConfigParser-based text file format, a
Tkinter-based GUI configurator which creates configuration files for
you, and a simple network-based event receiver which receives events
on TCP, UDP, HTTP and SOAP ports. This is suitable for testing and
might perhaps serve as a model for your own event receivers. Also
included are over 20 test scripts which serve both as test harnesses
and examples of how to use the logging package.

You can get more information from

http://www.red-dove.com/python_logging.html

There are "download" and "recent changes" links at the top of that
page. API documentation is available at

http://www.red-dove.com/logging/index.html

As always, your feedback is most welcome (especially bug reports,
patches and suggestions for improvement). Enjoy!

Cheers

Vinay Sajip
Red Dove Consultants Ltd.

Changes since the last independent release:
===========================================
Added getLoggerClass() (thanks to Dave Wilson).
Added exception handling in shutdown().
Sort globbed files in doRollover() in TimedRotatingFileHandler.
Date formatting for SMTPHandler now uses email package where
available.
fileConfig() exception handling added for handler instantiation.
Changed basicConfig() to add keyword arguments. Changes are
backward-compatible.
Refactored RotatingFileHandler to create a base class for rotating
handlers.
Added TimedRotatingFileHandler (thanks to Mark Davidson - minor
changes have been made to the patch he supplied).
Added error checking to log() to check that level is an integer, and
raise a TypeError if not (as long as raiseExceptions is set).
Fixed a bug in DatagramHandler.send() (thanks to Mario Aleppo and
Enrico Sirola for pointing it out).
Minor documentation corrections.

 

[Michele Simionato]

A new version of the Python logging package has been released. If you
are using version 2.3 of later of Python, this release will not mean
much to you - it is the same as recent checkins into Python CVS.
However, this release fixes numerous bugs reported since the last
independent release (0.4.9.2), and may be of interest to people using
earlier versions of Python.
<snip>

The logging package is pretty good and pretty powerful. The problem is that
it is *too* powerful. I found out that in most of the situations I need
a very tiny subset of its functionalities. Months ago I studied it,
but now I have completely forgot it; I have discovered that
it takes me less time to write my own logging routines than to use the
logging module :-(
Is there some plan to solve this "issue"? I mean, an easy to use interface
for the ones who just needs minimal logging facilities? Something easy to
remember? I know for sure that there is at least a Sourceforge project
about easy logging (I could found it googling on this newsgroup) but I
would like something standard included in the batteries.
Am I alone experiencing these difficulties?

Michele Simionato

 

[Vinay Sajip]

The logging package is pretty good and pretty powerful. The problem is that

it is *too* powerful. I found out that in most of the situations I need
a very tiny subset of its functionalities. Months ago I studied it,
but now I have completely forgot it; I have discovered that
it takes me less time to write my own logging routines than to use the
logging module :-(

What exactly is hard to use? An example of the minimalist way of
logging:

import logging

logging.warn("This is your %s warning", "first")
#similarly you can call logging.debug(), logging.info() etc.

Is there some plan to solve this "issue"? I mean, an easy to use interface
for the ones who just needs minimal logging facilities? Something easy to
remember? I know for sure that there is at least a Sourceforge project
about easy logging (I could found it googling on this newsgroup) but I
would like something standard included in the batteries.
Am I alone experiencing these difficulties?

Perhaps if you can explain what it is about the minimal interface
above which is hard to use or hard to remember, I can give you a more
focused answer.

Best regards,


Vinay Sajip

 

[Michael Hoffman]

What exactly is hard to use? An example of the minimalist way of
logging:

import logging

logging.warn("This is your %s warning", "first")
#similarly you can call logging.debug(), logging.info() etc.

Wow. I never realized you could do that. Apparently I'm not the only one.

I think it is mainly a problem with the documentation. My personal
experience: I found that on the top-level logging module documentation
there's about 2.5 pages of text that seems to go on and on in much more
detail than I wanted for a simple use case. So I look for an example.
Aha! There's a "basic example" page. But the "basic example" given in
the docs has 7 lines of setup before I can even add something to the
log. I always assumed that most of that was necessary.

I personally have found the logging module very useful when I have used
it. Now that I know how easy it is to get access to the root logger I'll
use it a lot more!

 

[Peter Hansen]

What exactly is hard to use? An example of the minimalist way of
logging:

import logging

logging.warn("This is your %s warning", "first")
#similarly you can call logging.debug(), logging.info() etc.

The above shows clearly that one can use logging in a simple
way, but as Michael points out, it's not well documented how
simple it can be. Even having read this thread and knowing
in advance that I could do that, I wasn't able to find the
relevant information easily in the documentation with
several minutes of reading (and several minutes covers a
lot of territory normally). Even when I found the comments
about the shortcut functions, I wouldn't have been certain
how to use them (as above) because an example is missing.

Unlike Michele, however, I find the logging package to be
fairly straightforward and easy to use (though admittedly
after a few months of disuse maybe I will not... but in
such cases I tend to cut and paste the setup code from
past projects and don't care).

What I did find, however, were several limitations, at least
one of which I think is easily resolved.

1) I wanted to log an except from a sys.excepthook handler.
You can't, near as I can tell, without doing something like
the following incantation:

def excepthook(*args, **kwargs):
root = logging.getLogger()
fn, lno = root.findCaller()
root.handle(root.makeRecord(root.name, logging.ERROR, fn, lno,
'uncaught exception', (), exc_info=args))

Yuck! Why is it so awkward? Well, it appears that it is
impossible to pass exception info to a Logger, because the
special "exc_info" argument is merely a boolean flag that
tells it to retrieve the info from sys.exc_info().

What if sys.exc_info() doesn't have the data, as in the above
real-world example, and others I could make up? Well, in that
case you just create a Formatter() and use formatException()...
except that there doesn't appear to be a (documented) way of
getting the formatter out of the root Logger... I'm using
fileConfig() to set things up, so I don't have direct access
to any of the objects and don't see an obvious way past the
above ugly code (which is hoisted out of Logger._log()).

2) I wanted my log files written to the application's own
directory, rather than to the current directory. Using
the fileConfig capability, you can't, at least not without
using absolute paths (near as I can tell).

My hack workaround was to prepopulate the logging module's
vars() with one call _app_path, and then to reference that
in the config file in the following way. This works only
because (a) eval is used on the args, and (b) the os module
has already been imported by logging.__init__:

(in the main code
logging._app_path = MY_APP_PATH
logging.config.fileConfig(MY_CONFIG_FILE)

(in the config file
[handler_hand01]
class=handlers.RotatingFileHandler
level=NOTSET
formatter=form01
args=(os.path.join(_app_path, 'bod.log'), 'a', 10000000, 9)


Now I can sort of see why the second issue might be awkward
to resolve (since it is in effect a problem that appears in
various other ways in the Python world, with no standard
solution, mainly because there's no simple "sys.getappdir()"
or whatever it would be called).

For the first problem, I think the exc_info "flag" should be
changed so that if it contains actual exception info, or
maybe just a tuple of three items, then *that* information
is logged instead of a call to sys.exc_info() being made.

But these are just warts, and I'm sure there are others, but
I do find the logging facility to be fairly elegant, all
things considered, and not very hard to figure out. (I'd
never used it before this thread came up, but the timing
was right and I just had a need for it, finally.)

-Peter

 

[Vinay Sajip]

The above shows clearly that one can use logging in a simple

way, but as Michael points out, it's not well documented how
simple it can be. Even having read this thread and knowing
in advance that I could do that, I wasn't able to find the
relevant information easily in the documentation with
several minutes of reading (and several minutes covers a
lot of territory normally). Even when I found the comments
about the shortcut functions, I wouldn't have been certain
how to use them (as above) because an example is missing.

The following documentation, which is in CVS for Python and will be
part of the next release, does give information for simple usage:

http://www.python.org/dev/doc/devel/lib/minimal-example.html

2) I wanted my log files written to the application's own
directory, rather than to the current directory. Using
the fileConfig capability, you can't, at least not without
using absolute paths (near as I can tell).

My hack workaround was to prepopulate the logging module's
vars() with one call _app_path, and then to reference that
in the config file in the following way. This works only
because (a) eval is used on the args, and (b) the os module
has already been imported by logging.__init__:

(in the main code
logging._app_path = MY_APP_PATH
logging.config.fileConfig(MY_CONFIG_FILE)

(in the config file
[handler_hand01]
class=handlers.RotatingFileHandler
level=NOTSET
formatter=form01
args=(os.path.join(_app_path, 'bod.log'), 'a', 10000000, 9)


Now I can sort of see why the second issue might be awkward
to resolve (since it is in effect a problem that appears in
various other ways in the Python world, with no standard
solution, mainly because there's no simple "sys.getappdir()"
or whatever it would be called).

You can simply say something like

logging.mymodule = mymodule

and in the config file, the args for the file handler can, instead of
specifying the file name, reference a function such as
mymodule.getpathname('myapp.log'), where getpathname prepends the
appropriate path.

In the same way, you can reference handlers defined in your own module
in the
config file.

For the first problem, I think the exc_info "flag" should be
changed so that if it contains actual exception info, or
maybe just a tuple of three items, then *that* information
is logged instead of a call to sys.exc_info() being made.

Not quite true, although the documentation could be a little clearer.
Originally, the exc_info flag was checked for true/false, and this
check still holds for deciding whether exception info is logged.
However, in recent versions, if an exception tuple (in the same format
as returned by sys.exc_info()) is passed, sys.exc_info() is not called
and the passed tuple is used instead. I aim to update the
documentation to reflect this.

Regards,


Vinay Sajip

 

[Michael Hoffman]

The following documentation, which is in CVS for Python and will be
part of the next release, does give information for simple usage:

http://www.python.org/dev/doc/devel/lib/minimal-example.html

+1! I think that makes the example much less intimidating.

I don't understand why it is in the middle of the table of contents
instead of at the beginning or end. And personally I would take some of
the stuff off the module root page and put it on a subpage so the table
of contents can be seen immediately... but that is just me. <wink>

Thanks for this great module!

 

[Daniel Ellison]

+1! I think that makes the example much less intimidating.

I don't understand why it is in the middle of the table of contents
instead of at the beginning or end. And personally I would take some of
the stuff off the module root page and put it on a subpage so the table
of contents can be seen immediately... but that is just me. <wink>

Thanks for this great module!

Excellent documentation, but it DWOMM (doesn't work on my machine(s)).
Python 2.3.3 on Debian (Linux kernel 2.4.26) and Cygwin (Win2kPro) both
claim that "basicConfig() takes no arguments (4 given)".

Has the API changed since the docs at that link?

Dan

 

[Vinay Sajip]

Excellent documentation, but it DWOMM (doesn't work on my machine(s)).

Python 2.3.3 on Debian (Linux kernel 2.4.26) and Cygwin (Win2kPro) both
claim that "basicConfig() takes no arguments (4 given)".

Has the API changed since the docs at that link?

The link points to "in-development" documentation, which is consistent
with Python in CVS. The basicConfig() API was changed to add the
optional keyword arguments documented in the link - the 2.4 release of
Python will have this API, which is backwards-compatible. If you want
to use it now, try downloading the logging package from CVS.

Regards,


Vinay Sajip

 

[Peter L Hansen]

Not quite true, although the documentation could be a little clearer.
Originally, the exc_info flag was checked for true/false, and this
check still holds for deciding whether exception info is logged.
However, in recent versions, if an exception tuple (in the same format
as returned by sys.exc_info()) is passed, sys.exc_info() is not called
and the passed tuple is used instead. I aim to update the
documentation to reflect this.

I take it this is yet another of the changes that was made
only in post-Python-2.3, because in 2.3.4 it does not behave
that way. If exc_info is set to anything, it is overwritten
with sys.exc_info().

-Peter

 

[Daniel Ellison]

The link points to "in-development" documentation, which is consistent
with Python in CVS. The basicConfig() API was changed to add the
optional keyword arguments documented in the link - the 2.4 release of
Python will have this API, which is backwards-compatible. If you want
to use it now, try downloading the logging package from CVS.

Regards,


Vinay Sajip

Thanks, Vinay! Perhaps I should learn to read a little gooder. In my
defense, the "6.28 logging" page at the same location says "New in
version 2.3", which I used to make sure it was supposed to be working as
described.

Of course, if I had read your post more carefully...

Dan

 

[Vinay Sajip]

I take it this is yet another of the changes that was made

only in post-Python-2.3, because in 2.3.4 it does not behave
that way. If exc_info is set to anything, it is overwritten
with sys.exc_info().

Yes. The code in CVS does this:

if exc_info:
if type(exc_info) != types.TupleType:
exc_info = sys.exc_info()


Regards,

Vinay