Making `logging.basicConfig` log to *both* `sys.stderr` and `sys.stdout`?

[Michel Albert]

Hi,

I use python oftentimes to write automation scripts on Linux servers.
And there's a big pattern in my scripts:

- I *always* use `logging` instead of `print` statements.
- I *always* create two stream handlers. One for `sys.stdout` with
level `INFO` and one for `sys.stderr` with level `WARN`

Well, the levels may variate occasionally, but that's only the rare
exception.

The reason I do this is simple: Most automation tasks are run via
cron. With this setup, I can redirect `stdout` to `/dev/null` and
still receive e-mails if things go wrong.

And having two handlers gives me more flexibility in my scripts. In
one case, I used a different color for error messages for example as
this script is run primarily from the shell and having errors stand
out has proven to be a good thing.

Unfortunately this setup makes `logging.basicConfig` pretty useless.
However, I believe that this is something that more people could
benefit from. I also believe, that it just "makes sense" to send
warnings (and above) to `stderr`, the rest to `stdout`.

So I was thinking: "Why does `logging.basicConfig` not behave that
way".

Naturally, I was thinking of writing a patch against the python
codebase and submit it as a suggestion. But before doing so, I would
like to hear your thoughts on this. Does it make sense to you too or
am I on the wrong track? Are there any downsides I am missing?

 

[Peter Otten]

I use python oftentimes to write automation scripts on Linux servers.
And there's a big pattern in my scripts:

- I *always* use `logging` instead of `print` statements.
- I *always* create two stream handlers. One for `sys.stdout` with
level `INFO` and one for `sys.stderr` with level `WARN`

Well, the levels may variate occasionally, but that's only the rare
exception.

How would a call to basicConfig() look like that produces this setup?

 

[Michel Albert]

How would a call to basicConfig() look like that produces this setup?

I personally see this happen by default (i.e. no additional
parameters). And in case the `stream` parameter is set, /then/ you
would send all to that stream only.

In my point of view, the call to `basicConfig` is either something
used in only the most mundane usages of the logging package, or it's
mainly used by people that have not yet grokked the logging package
(correct me if I'm wrong). In both cases, I find it useful to redirect
warnings and errors to `stderr` by default.

However, this would also mean that existing code calling this method
would result in different behavior. But only /slightly/ different.
Meaning, you still see the output on the console as expected. But it
gives you the possibility to use standard shell redirection in a way
that "makes sense".

 

[Vinay Sajip]

Unfortunately this setup makes `logging.basicConfig` pretty useless.
However, I believe that this is something that more people could
benefit from. I also believe, that it just "makes sense" to send
warnings (and above) to `stderr`, the rest to `stdout`.

So I was thinking: "Why does `logging.basicConfig` not behave that
way".

Because what seems entirely natural and obvious to you might not seem
so for someone else. The API in the stdlib tries to provide baseline
functionality which others can build on. For example, if you always
have a particular pattern which you use, you can always write a
utility function to set things up exactly how you like, and others who
want to set things up differently (for whatever reason) can do the
same thing, without having to come into conflict (if that's not too
strong a word) with views different from their own.

Naturally, I was thinking of writing a patch against the python
codebase and submit it as a suggestion. But before doing so, I would
like to hear your thoughts on this. Does it make sense to you too or
am I on the wrong track? Are there any downsides I am missing?

Python 2.x is closed to feature changes, and Python 2.7 and Python 3.2
already support flexible configuration using dictConfig() - see

http://docs.python.org/library/logging.config.html#logging.config.dictConfig

Also, Python 3.3 will support passing a list of handlers to
basicConfig(): see

http://plumberjack.blogspot.com/2011/04/added-functionality-for-basicconfig-in.html

which will allow you to do what you want quite easily.

Regards,

Vinay Sajip

 

[Daniel Dehennin]

Hello,

Unfortunately this setup makes `logging.basicConfig` pretty useless.
However, I believe that this is something that more people could
benefit from. I also believe, that it just "makes sense" to send
warnings (and above) to `stderr`, the rest to `stdout`.

[...]

Python 2.x is closed to feature changes, and Python 2.7 and Python 3.2
already support flexible configuration using dictConfig() - see

http://docs.python.org/library/logging.config.html#logging.config.dictConfig

Also, Python 3.3 will support passing a list of handlers to
basicConfig(): see

http://plumberjack.blogspot.com/2011/04/added-functionality-for-basicconfig=
-in.html

which will allow you to do what you want quite easily.

I tried to setup the same for my scripts with
"logging.config.dictConfig()" without much success, I want to limit
output to stdout to INFO, everything bellow INFO should be sent to
stderr instead.

Any hints?

Here is my configuration:

#+begin_src python
def init_logging(options):
# TODO: color stderr messages by level if sys.stderr.isatty()
config = { 'version' : 1,
'formatters' : { 'stdout' : { 'format' : '%(message)s',
'datefmt' : '', },
'stderr' : { 'format' : '%(message)s',
'datefmt' : '', },
},
'handlers' : { 'stdout' : { 'class' : 'logging.StreamHandler',
'stream' : 'ext://sys.stdout',
'level' : 'INFO',
'formatter' : 'stdout', },
'stderr' : { 'class' : 'logging.StreamHandler',
'stream' : 'ext://sys.stderr',
'level' : 'WARNING',
'formatter' : 'stderr', }
},
'root' : { 'level' : options.log_level.upper(),
'handlers' : ['stdout', 'stderr'],
},
}

logging.config.dictConfig( config )
return logging.getLogger()
#+end_src

Regards.
--
Daniel Dehennin
EOLE

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREKAAYFAlCGeckACgkQWjgIUPVihwxLlgD+Jo0yvR7ut2hyYC/nVUm0pJLB
h0JEHS9p65Twe3MdZhABALcjv70okv0tnfa5fvnXqkTfUaFzlQY+uIq89MjPclnp
=GFdD
-----END PGP SIGNATURE-----

 

[Daniel Dehennin]

Hello,
Hi


[...]

I tried to setup the same for my scripts with
"logging.config.dictConfig()" without much success, I want to limit
output to stdout to INFO, everything bellow INFO should be sent to
stderr instead.

Any hints?

I finally find a solution for my script, I added a filter.

Regards.

#+begin_src python
class UniqLevelFilter(logging.Filter):
def __init__(self, level):
self._level = level

def filter(self, rec):
return rec.levelno == self._level

def init_logging(options):
# TODO: color stderr messages by level if sys.stderr.isatty()
# http://stackoverflow.com/questions/384076/how-can-i-color-python-logging-output
# http://plumberjack.blogspot.fr/2010/12/colorizing-logging-output-in-terminals.html
config = { 'version' : 1,
'filters' : { 'stdout' : { '()' : '__main__.UniqLevelFilter',
'level' : logging.INFO, },
},
'formatters' : { 'stdout' : { 'format' : '%(message)s',
'datefmt' : '', },
'stderr' : { 'format' : '%(message)s',
'datefmt' : '', },
},
'handlers' : { 'stdout' : { 'class' : 'logging.StreamHandler',
'stream' : 'ext://sys.stdout',
'level' : 'INFO',
'filters' : ['stdout'],
'formatter' : 'stdout', },
'stderr' : { 'class' : 'logging.StreamHandler',
'stream' : 'ext://sys.stderr',
'level' : 'WARNING',
'formatter' : 'stderr', }
},
'root' : { 'level' : options.log_level.upper(),
'handlers' : ['stdout', 'stderr'],
},
}

logging.config.dictConfig( config )
return logging.getLogger()

#+end_src

--
Daniel Dehennin
EOLE

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREKAAYFAlCHta4ACgkQWjgIUPVihwyu7QEA23jMuq6UTjtn2BBwCD4m76Pu
/NR3EOHrQkBmIsmG4xwA/jqtgRGdb6MxBotUlQUEqexcBnly67b3aXnV4mkd63V+
=LoAd
-----END PGP SIGNATURE-----