Making `logging.basicConfig` log to *both* `sys.stderr` and `sys.stdout`?

Discussion in 'Python' started by Michel Albert, Aug 30, 2011.

  1. Hi,

    I use python oftentimes to write automation scripts on Linux servers.
    And there's a big pattern in my scripts:

    - I *always* use `logging` instead of `print` statements.
    - I *always* create two stream handlers. One for `sys.stdout` with
    level `INFO` and one for `sys.stderr` with level `WARN`

    Well, the levels may variate occasionally, but that's only the rare
    exception.

    The reason I do this is simple: Most automation tasks are run via
    cron. With this setup, I can redirect `stdout` to `/dev/null` and
    still receive e-mails if things go wrong.

    And having two handlers gives me more flexibility in my scripts. In
    one case, I used a different color for error messages for example as
    this script is run primarily from the shell and having errors stand
    out has proven to be a good thing.

    Unfortunately this setup makes `logging.basicConfig` pretty useless.
    However, I believe that this is something that more people could
    benefit from. I also believe, that it just "makes sense" to send
    warnings (and above) to `stderr`, the rest to `stdout`.

    So I was thinking: "Why does `logging.basicConfig` not behave that
    way".

    Naturally, I was thinking of writing a patch against the python
    codebase and submit it as a suggestion. But before doing so, I would
    like to hear your thoughts on this. Does it make sense to you too or
    am I on the wrong track? Are there any downsides I am missing?
     
    Michel Albert, Aug 30, 2011
    #1
    1. Advertising

  2. Michel Albert

    Peter Otten Guest

    Michel Albert wrote:

    > I use python oftentimes to write automation scripts on Linux servers.
    > And there's a big pattern in my scripts:
    >
    > - I *always* use `logging` instead of `print` statements.
    > - I *always* create two stream handlers. One for `sys.stdout` with
    > level `INFO` and one for `sys.stderr` with level `WARN`
    >
    > Well, the levels may variate occasionally, but that's only the rare
    > exception.


    How would a call to basicConfig() look like that produces this setup?
     
    Peter Otten, Aug 30, 2011
    #2
    1. Advertising

  3. On Aug 30, 11:45 am, Peter Otten <> wrote:
    > Michel Albert wrote:
    > > I use python oftentimes to write automation scripts on Linux servers.
    > > And there's a big pattern in my scripts:

    >
    > > - I *always* use `logging` instead of `print` statements.
    > > - I *always* create two stream handlers. One for `sys.stdout` with
    > > level `INFO` and one for `sys.stderr` with level `WARN`

    >
    > > Well, the levels may variate occasionally, but that's only the rare
    > > exception.

    >
    > How would a call to basicConfig() look like that produces this setup?


    I personally see this happen by default (i.e. no additional
    parameters). And in case the `stream` parameter is set, /then/ you
    would send all to that stream only.

    In my point of view, the call to `basicConfig` is either something
    used in only the most mundane usages of the logging package, or it's
    mainly used by people that have not yet grokked the logging package
    (correct me if I'm wrong). In both cases, I find it useful to redirect
    warnings and errors to `stderr` by default.

    However, this would also mean that existing code calling this method
    would result in different behavior. But only /slightly/ different.
    Meaning, you still see the output on the console as expected. But it
    gives you the possibility to use standard shell redirection in a way
    that "makes sense".
     
    Michel Albert, Aug 30, 2011
    #3
  4. Michel Albert

    Vinay Sajip Guest

    On Aug 30, 9:53 am, Michel Albert <> wrote:


    > Unfortunately this setup makes `logging.basicConfig` pretty useless.
    > However, I believe that this is something that more people could
    > benefit from. I also believe, that it just "makes sense" to send
    > warnings (and above) to `stderr`, the rest to `stdout`.
    >
    > So I was thinking: "Why does `logging.basicConfig` not behave that
    > way".


    Because what seems entirely natural and obvious to you might not seem
    so for someone else. The API in the stdlib tries to provide baseline
    functionality which others can build on. For example, if you always
    have a particular pattern which you use, you can always write a
    utility function to set things up exactly how you like, and others who
    want to set things up differently (for whatever reason) can do the
    same thing, without having to come into conflict (if that's not too
    strong a word) with views different from their own.

    > Naturally, I was thinking of writing a patch against the python
    > codebase and submit it as a suggestion. But before doing so, I would
    > like to hear your thoughts on this. Does it make sense to you too or
    > am I on the wrong track? Are there any downsides I am missing?


    Python 2.x is closed to feature changes, and Python 2.7 and Python 3.2
    already support flexible configuration using dictConfig() - see

    http://docs.python.org/library/logging.config.html#logging.config.dictConfig

    Also, Python 3.3 will support passing a list of handlers to
    basicConfig(): see

    http://plumberjack.blogspot.com/2011/04/added-functionality-for-basicconfig-in.html

    which will allow you to do what you want quite easily.

    Regards,

    Vinay Sajip
     
    Vinay Sajip, Sep 2, 2011
    #4
  5. Re: Making `logging.basicConfig` log to *both* `sys.stderr` and`sys.stdout`?

    Hello,

    >> Unfortunately this setup makes `logging.basicConfig` pretty useless.
    >> However, I believe that this is something that more people could
    >> benefit from. I also believe, that it just "makes sense" to send
    >> warnings (and above) to `stderr`, the rest to `stdout`.


    [...]

    > Python 2.x is closed to feature changes, and Python 2.7 and Python 3.2
    > already support flexible configuration using dictConfig() - see
    >
    > http://docs.python.org/library/logging.config.html#logging.config.dictConfig
    >
    > Also, Python 3.3 will support passing a list of handlers to
    > basicConfig(): see
    >
    > http://plumberjack.blogspot.com/2011/04/added-functionality-for-basicconfig=
    > -in.html
    >
    > which will allow you to do what you want quite easily.


    I tried to setup the same for my scripts with
    "logging.config.dictConfig()" without much success, I want to limit
    output to stdout to INFO, everything bellow INFO should be sent to
    stderr instead.

    Any hints?

    Here is my configuration:

    #+begin_src python
    def init_logging(options):
    # TODO: color stderr messages by level if sys.stderr.isatty()
    config = { 'version' : 1,
    'formatters' : { 'stdout' : { 'format' : '%(message)s',
    'datefmt' : '', },
    'stderr' : { 'format' : '%(message)s',
    'datefmt' : '', },
    },
    'handlers' : { 'stdout' : { 'class' : 'logging.StreamHandler',
    'stream' : 'ext://sys.stdout',
    'level' : 'INFO',
    'formatter' : 'stdout', },
    'stderr' : { 'class' : 'logging.StreamHandler',
    'stream' : 'ext://sys.stderr',
    'level' : 'WARNING',
    'formatter' : 'stderr', }
    },
    'root' : { 'level' : options.log_level.upper(),
    'handlers' : ['stdout', 'stderr'],
    },
    }

    logging.config.dictConfig( config )
    return logging.getLogger()
    #+end_src

    Regards.
    --
    Daniel Dehennin
    EOLE

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v2.0.19 (GNU/Linux)

    iF4EAREKAAYFAlCGeckACgkQWjgIUPVihwxLlgD+Jo0yvR7ut2hyYC/nVUm0pJLB
    h0JEHS9p65Twe3MdZhABALcjv70okv0tnfa5fvnXqkTfUaFzlQY+uIq89MjPclnp
    =GFdD
    -----END PGP SIGNATURE-----
     
    Daniel Dehennin, Oct 23, 2012
    #5
  6. [SOLVED] Re: Making `logging.basicConfig` log to *both* `sys.stderr`and `sys.stdout`?

    Daniel Dehennin <> writes:

    > Hello,


    Hi


    [...]

    > I tried to setup the same for my scripts with
    > "logging.config.dictConfig()" without much success, I want to limit
    > output to stdout to INFO, everything bellow INFO should be sent to
    > stderr instead.
    >
    > Any hints?


    I finally find a solution for my script, I added a filter.

    Regards.

    #+begin_src python
    class UniqLevelFilter(logging.Filter):
    def __init__(self, level):
    self._level = level

    def filter(self, rec):
    return rec.levelno == self._level

    def init_logging(options):
    # TODO: color stderr messages by level if sys.stderr.isatty()
    # http://stackoverflow.com/questions/384076/how-can-i-color-python-logging-output
    # http://plumberjack.blogspot.fr/2010/12/colorizing-logging-output-in-terminals.html
    config = { 'version' : 1,
    'filters' : { 'stdout' : { '()' : '__main__.UniqLevelFilter',
    'level' : logging.INFO, },
    },
    'formatters' : { 'stdout' : { 'format' : '%(message)s',
    'datefmt' : '', },
    'stderr' : { 'format' : '%(message)s',
    'datefmt' : '', },
    },
    'handlers' : { 'stdout' : { 'class' : 'logging.StreamHandler',
    'stream' : 'ext://sys.stdout',
    'level' : 'INFO',
    'filters' : ['stdout'],
    'formatter' : 'stdout', },
    'stderr' : { 'class' : 'logging.StreamHandler',
    'stream' : 'ext://sys.stderr',
    'level' : 'WARNING',
    'formatter' : 'stderr', }
    },
    'root' : { 'level' : options.log_level.upper(),
    'handlers' : ['stdout', 'stderr'],
    },
    }

    logging.config.dictConfig( config )
    return logging.getLogger()

    #+end_src

    --
    Daniel Dehennin
    EOLE

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v2.0.19 (GNU/Linux)

    iF4EAREKAAYFAlCHta4ACgkQWjgIUPVihwyu7QEA23jMuq6UTjtn2BBwCD4m76Pu
    /NR3EOHrQkBmIsmG4xwA/jqtgRGdb6MxBotUlQUEqexcBnly67b3aXnV4mkd63V+
    =LoAd
    -----END PGP SIGNATURE-----
     
    Daniel Dehennin, Oct 24, 2012
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Roman Neuhauser
    Replies:
    0
    Views:
    661
    Roman Neuhauser
    Apr 4, 2005
  2. Guillaume Dargaud

    freopen on both stdout and stderr

    Guillaume Dargaud, Nov 23, 2007, in forum: C Programming
    Replies:
    6
    Views:
    2,127
    Charlie Gordon
    Nov 26, 2007
  3. Mitchell L Model

    sys.stdout vs. sys.stderr

    Mitchell L Model, Jan 11, 2010, in forum: Python
    Replies:
    2
    Views:
    569
    Nobody
    Jan 11, 2010
  4. Geoff Bache
    Replies:
    4
    Views:
    558
  5. Ed Hames
    Replies:
    0
    Views:
    378
    Ed Hames
    Apr 16, 2008
Loading...

Share This Page