SMTPHandler and Unicode

[norbert]

Hello,

I want to send error messages with SMTPHandler logging. But
SMTPHandler does not seem to be unicode aware. Is there something
doable without playing with sys.setdefaultencoding ?

import logging,logging.handlers
smtpHandler =
logging.handlers.SMTPHandler(mailhost=("smtp.example.com",25),
fromaddr="(e-mail address removed)", toaddrs="(e-mail address removed)",
subject=u"error message")

LOG = logging.getLogger()
LOG.addHandler(smtpHandler)

LOG.error(u"sans accent")
LOG.error(u"accentu\u00E9")

gives : UnicodeEncodeError: 'ascii' codec can't encode character
u'\xe9' in position 117: ordinal not in range(128)

Thank you !

 

[Chris Withers]

I want to send error messages with SMTPHandler logging. But
SMTPHandler does not seem to be unicode aware. Is there something
doable without playing with sys.setdefaultencoding ?

try MailingLogger:

http://www.simplistix.co.uk/software/python/mailinglogger

If you have unicode problems with that, I'd be interested in fixing them!

cheers,

Chris

 

[norbert]

try MailingLogger:

If you have unicode problems with that, I'd be interested in fixing them!

Your package has the same unicode problem :
import logging,logging.handlers
from mailinglogger.MailingLogger import MailingLogger
mailingLogger = MailingLogger(mailhost=('smtp.example.com',
25),fromaddr='(e-mail address removed)',toaddrs=('(e-mail address removed)',))
LOG = logging.getLogger()
LOG.addHandler(mailingLogger)
LOG.error(u"sans accent")
LOG.error(u"accentu\u00E9")

--> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9'
in position 7: ordinal not in range(128)

 

[Chris Withers]

Your package has the same unicode problem :
import logging,logging.handlers
from mailinglogger.MailingLogger import MailingLogger
mailingLogger = MailingLogger(mailhost=('smtp.example.com',
25),fromaddr='(e-mail address removed)',toaddrs=('(e-mail address removed)',))
LOG = logging.getLogger()
LOG.addHandler(mailingLogger)
LOG.error(u"sans accent")
LOG.error(u"accentu\u00E9")

--> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9'
in position 7: ordinal not in range(128)

Interesting, I don't know what the logging framework's position is on
unicode...

What happens when you try the same logging with just a FileHandler
registered? What encoding does the log file use?

cheers,

Chris

 

[norbert]

Interesting, I don't know what the logging framework's position is on
unicode...

What happens when you try the same logging with just a FileHandler
registered? What encoding does the log file use?

a FileHandler works as expected, the log file being UTF-8 encoded. The
SMTPHandler is the only logger I know with this problem, maybe
connected to SMTPLib implementation ?

 

[Antoine Pitrou]

a FileHandler works as expected, the log file being UTF-8 encoded.

Ouch. Implicit encoding sounds like a bad behaviour.

The
SMTPHandler is the only logger I know with this problem, maybe
connected to SMTPLib implementation ?

I suggest you report an issue on http://bugs.python.org

 

[Chris Withers]

Ouch. Implicit encoding sounds like a bad behaviour.

Yes indeed, hence my question on python-dev...

Chris

 

[norbert]

Ouch. Implicit encoding sounds like a bad behaviour.

Looking at the FileHandler source (
http://svn.python.org/view/python/trunk/Lib/logging/__init__.py?view=markup
) : the utf-8 encoding is a fallback. But *FileHandler family let you
specify the encoding you want, so that's OK I think.

But SMTPHandler does not have such a thing it sends its email with :
msg = "From: %s\r\nTo: %s\r\nSubject: %s\r\nDate: %s\r\n\r\n%s" % (
self.fromaddr,
",".join(self.toaddrs),
self.getSubject(record),
formatdate(), msg)
....
smtp.sendmail(from,to,msg)

And there is no encoding in all this.

It seems pretty dangerous to me (so my first post) because your
application will work without any problem with a FileHandler and the
day you'll decide to send email in case of serious problem, it will
crash with a UnicodeError. I can't see any workaround, except by
subclassing SMTPHandler's emit method to be unicode-aware or at least
URF-8 aware.

 

[Vinay Sajip]

Ouch. Implicit encoding sounds like a bad behaviour.

UTF-8 is only used as a fallback in an exception handler, you can use
any supported encoding using the encoding= keyword argument to a
FileHandler.

I suggest you report an issue onhttp://bugs.python.org

Yes, please do that and I will investigate.

Regards,

Vinay Sajip

 

[Vinay Sajip]

crash with a UnicodeError. I can't see any workaround, except by
subclassing SMTPHandler's emit method to be unicode-aware or at least
URF-8 aware.

Well, you could use an approach like the one suggested here:

http://plumberjack.blogspot.com/2010/07/using-custom-formatter-to-deal-with.html

 

[norbert]

Well, you could use an approach like the one suggested here:

http://plumberjack.blogspot.com/2010/07/using-custom-formatter-to-dea...

That's nice, thanks. I'll use something like this. Just a thought : I
will use "errors=replace" in the call to the encode method to be sure
that the logger does not raise any exception.

 

[norbert]

I suggest you report an issue onhttp://bugs.python.org

done : http://bugs.python.org/issue9208

 

[Chris Withers]

Hi Norbert,

Your package has the same unicode problem :
import logging,logging.handlers
from mailinglogger.MailingLogger import MailingLogger
mailingLogger = MailingLogger(mailhost=('smtp.example.com',
25),fromaddr='(e-mail address removed)',toaddrs=('(e-mail address removed)',))
LOG = logging.getLogger()
LOG.addHandler(mailingLogger)
LOG.error(u"sans accent")
LOG.error(u"accentu\u00E9")

--> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9'
in position 7: ordinal not in range(128)

It's taken a ridiculously long amount of time (sadly due to no-one else
complaining) but this is now fixed in the source control for
mailinglogger, and will be in the next release:

https://github.com/Simplistix/testfixtures

cheers,

Chris

 

[Chris Withers]

It's taken a ridiculously long amount of time (sadly due to no-one else
complaining) but this is now fixed in the source control for
mailinglogger, and will be in the next release:

https://github.com/Simplistix/testfixtures

Sorry, I meant this:

https://github.com/Simplistix/mailinglogger/commit/1c95f532c65ab18e1dd8513e1aa1ae328a19d249

cheers,

Chris