Got some problems when using logging Filter

[sword]

The logging cookbook gives an Filter example, explainning how to add
contextural info to log. I can't figure out how to filter log from it.

Suppose I have 3 file, a.py, b.py and main.py
#file: a.py
import logging

logger=logging.getLogger(__name__)
def print_log():
logger.debug("I'm module a")

#file: b.py just like a.py
import logging
logger=logging.getLogger(__name__)
def print_log():
logger.debug("I'm module b")

#file: main.py
import logging
from logging import Filter
logging.basicConfig(level=logging.DEBUG)
logger=logging.getLogger("main")
logger.debug("This is main process")
logger.addFilter(Filter("a"))

And I expected that the console output would contain main and b module
log only. But it turned out that all logs there. Is it the problem of
root logger?

 

[Jean-Michel Pichavant]

The logging cookbook gives an Filter example, explainning how to add
contextural info to log. I can't figure out how to filter log from it.

Suppose I have 3 file, a.py, b.py and main.py
#file: a.py
import logging

logger=logging.getLogger(__name__)
def print_log():
logger.debug("I'm module a")

#file: b.py just like a.py
import logging
logger=logging.getLogger(__name__)
def print_log():
logger.debug("I'm module b")

#file: main.py
import logging
from logging import Filter
logging.basicConfig(level=logging.DEBUG)
logger=logging.getLogger("main")
logger.debug("This is main process")
logger.addFilter(Filter("a"))

And I expected that the console output would contain main and b module
log only. But it turned out that all logs there. Is it the problem of
root logger?

Hi,

First of all, in the code you provided we can't see where you import a &
b, and when you call their respective print_log method.
Secondly,Filter("a") would allow only the "a" log events, not forbid
them. quoting the docs: "if name is specified, it names a logger which,
together with its children, will have its events allowed through the
filter."

As for your problem it may come from the fact that you applied the
filter to the 'main' logger, while you probably want to add the filter
to the *root* logger. Your current hierarchy is

root
- main
- a
- b

events fired from 'a' will be handled by the root logger, not the main.
root = logging.getLogger()
root.addFilter('main')
root.addFilter('a')
root.addFilter('b')

JM

 

[sword]

Hi,

First of all, in the code you provided we can't see where you import a &
b, and when you call their respective print_log method.
Secondly,Filter("a") would allow only the "a" log events, not forbid
them. quoting the docs: "if name is specified, it names a logger which,
together with its children, will have its events allowed through the
filter."

As for your problem it may come from the fact that you applied the
filter to the 'main' logger, while you probably want to add the filter
to the *root* logger. Your current hierarchy is

root
  - main
  - a
  - b

events fired from 'a' will be handled by the root logger, not the main.
root = logging.getLogger()
root.addFilter('main')
root.addFilter('a')
root.addFilter('b')

JM

Thanks for your reply. I tried to edit the source a bit, now the
main.py looks like this:
#main.py
import logging
from logging import Filter
import a
import b

logging.basicConfig(level=logging.DEBUG)
root = logging.getLogger()
root.addFilter(Filter("GoneWithTheWind")) #so I suppose no log msg
would pass this filter

logger = logging.getLogger("main")
logger.debug("main process")
a.print_log()
b.print_log()

####
And It still prints out all the log msg.

 

[Peter Otten]

Thanks for your reply. I tried to edit the source a bit, now the
main.py looks like this:
#main.py
import logging
from logging import Filter
import a
import b

logging.basicConfig(level=logging.DEBUG)
root = logging.getLogger()
root.addFilter(Filter("GoneWithTheWind")) #so I suppose no log msg
would pass this filter

logger = logging.getLogger("main")
logger.debug("main process")
a.print_log()
b.print_log()

####
And It still prints out all the log msg.

Here's a little demo to explore how filtering works:

$ cat demo.py
import logging
class Filter(logging.Filter):
def filter(self, record):
print "applying filter", self.name
return True

logging.basicConfig()

loggers = [logging.getLogger(path) for path in ["", "a", "a.b"]]
for logger in loggers:
logger.addFilter(Filter("filter@" + logger.name))

[handler] = logging.getLogger().handlers
handler.addFilter(Filter("filter@handler"))

for logger in loggers:
logger.critical("whatever")
$ python demo.py
applying filter filter@root
applying filter filter@handler
CRITICAL:root:whatever
applying filter filter@a
applying filter filter@handler
CRITICAL:a:whatever
applying filter (e-mail address removed)
applying filter filter@handler
CRITICAL:a.b:whatever
$

As you can infer from the output only the filter(s) of the original logger
and of the handler(s) are applied.

 

[sword]

Thanks for your reply. I tried to edit the source a bit, now the
main.py looks like this:
#main.py
import logging
from logging import Filter
import a
import b

logging.basicConfig(level=logging.DEBUG)
root = logging.getLogger()
root.addFilter(Filter("GoneWithTheWind")) #so I suppose no log msg
would pass this filter

logger = logging.getLogger("main")
logger.debug("main process")
a.print_log()
b.print_log()

####
And It still prints out all the log msg.

Here's a little demo to explore how filtering works:

$ cat demo.py
import logging
class Filter(logging.Filter):
    def filter(self, record):
        print "applying filter", self.name
        return True

logging.basicConfig()

loggers = [logging.getLogger(path) for path in ["", "a", "a.b"]]
for logger in loggers:
    logger.addFilter(Filter("filter@" + logger.name))

[handler] = logging.getLogger().handlers
handler.addFilter(Filter("filter@handler"))

for logger in loggers:
    logger.critical("whatever")
$ python demo.py
applying filter filter@root
applying filter filter@handler
CRITICAL:root:whatever
applying filter filter@a
applying filter filter@handler
CRITICAL:a:whatever
applying filter (e-mail address removed)
applying filter filter@handler
CRITICAL:a.b:whatever
$

As you can infer from the output only the filter(s) of the original logger
and of the handler(s) are applied.

Thanks, so if I want to see my own log out of all logs produced by
different module in the project, I should addFilter to each
corresponding logger. I thought I could add Filter in root and filter
out only the interested info from it before.

 

[Vinay Sajip]

Here's a little demo to explore how filtering works:

$ cat demo.py
importlogging
class Filter(logging.Filter):
    def filter(self, record):
        print "applying filter", self.name
        return True

logging.basicConfig()

loggers = [logging.getLogger(path) for path in ["", "a", "a.b"]]
for logger in loggers:
    logger.addFilter(Filter("filter@" + logger.name))

[handler] =logging.getLogger().handlers
handler.addFilter(Filter("filter@handler"))

for logger in loggers:
    logger.critical("whatever")
$ python demo.py
applying filter filter@root
applying filter filter@handler
CRITICAL:root:whatever
applying filter filter@a
applying filter filter@handler
CRITICAL:a:whatever
applying filter (e-mail address removed)
applying filter filter@handler
CRITICAL:a.b:whatever
$

As you can infer from the output only the filter(s) of the original logger
and of the handler(s) are applied.

Thanks, so if I want to see my own log out of all logs produced by
different module in the project, I should addFilter to each
corresponding logger. I thought I could add Filter in root and filter
out only the interested info from it before.

Or you can add a filter to the handler (but then you can't use
basicConfig() to configure it - you need to do it explicitly).

Regards,

Vinay Sajip

 

[sword]

sword wrote:
Thanks for your reply. I tried to edit the source a bit, now the
main.py looks like this:
#main.py
importlogging
fromloggingimport Filter
import a
import b
logging.basicConfig(level=logging.DEBUG)
root =logging.getLogger()
root.addFilter(Filter("GoneWithTheWind")) #so I suppose no log msg
would pass this filter
logger =logging.getLogger("main")
logger.debug("main process")
a.print_log()
b.print_log()
####
And It still prints out all the log msg.
Here's a little demo to explore how filtering works:
$ cat demo.py
importlogging
class Filter(logging.Filter):
    def filter(self, record):
        print "applying filter", self.name
        return True
logging.basicConfig()
loggers = [logging.getLogger(path) for path in ["", "a", "a.b"]]
for logger in loggers:
    logger.addFilter(Filter("filter@" + logger.name))
[handler] =logging.getLogger().handlers
handler.addFilter(Filter("filter@handler"))
for logger in loggers:
    logger.critical("whatever")
$ python demo.py
applying filter filter@root
applying filter filter@handler
CRITICAL:root:whatever
applying filter filter@a
applying filter filter@handler
CRITICAL:a:whatever
applying filter (e-mail address removed)
applying filter filter@handler
CRITICAL:a.b:whatever
$
As you can infer from the output only the filter(s) of the original logger
and of the handler(s) are applied.

Thanks, so if I want to see my own log out of all logs produced by
different module in the project, I should addFilter to each
corresponding logger. I thought I could add Filter in root and filter
out only the interested info from it before.

Or you can add a filter to the handler (but then you can't use
basicConfig() to configure it - you need to do it explicitly).

Regards,

Vinay Sajip

Thank you! Maybe I should find out another way to manipulate the log,
like wrap the getLogger function and add the filter at the first
time

 

[Vinay Sajip]

Thank you! Maybe I should find out another way to manipulate the log,
like wrap the getLogger function and add the filter at the first
time

If you are using Python 2.7, 3.2 or later, you can use dictionary-
based configuration - it's fairly painless.

http://docs.python.org/library/logging.config.html#logging.config.dictConfig

If you are using an earlier version of Python, the logutils project
includes the same dictionary-based configuration logic.

http://code.google.com/p/logutils/

Regards,

Vinay Sajip

 

[Jean-Michel Pichavant]

Thanks for your reply. I tried to edit the source a bit, now the
main.py looks like this:
#main.py
import logging
from logging import Filter
import a
import b

logging.basicConfig(level=logging.DEBUG)
root = logging.getLogger()
root.addFilter(Filter("GoneWithTheWind")) #so I suppose no log msg
would pass this filter

logger = logging.getLogger("main")
logger.debug("main process")
a.print_log()
b.print_log()

####
And It still prints out all the log msg.

You need to add you filter to the handler(s) of the root logger, not the
root logger itself.

Filters to loggger object are applied only when the log event is raised,
i.e. when one of the logging method error, info, warning, debug is called.
In your case, log events are raised by other loggers than the root
logger, so its filter will not apply.

However any filter applied to the root *handlers* will be applied to any
log processed by the handler, including log event raised by sub-logger.

root.handlers[-1].addFilter(Filter("GoneWithTheWind")) #so I suppose no log msg

JM

Quoting http://docs.python.org/library/logging.html#filter-objects

"Note that filters attached to handlers are consulted whenever an event is emitted by the handler, whereas filters attached to loggers are consulted whenever an event is logged to the handler (using debug(), info(), etc.) This means that events which have been generated by descendant loggers will not be filtered by a logger’s filter setting, unless the filter has also been applied to those descendant loggers."