Trouble with UnicodeEncodeError and email

  • Thread starter Florian Lindner
  • Start date


Florian Lindner


I've written some tiny script using Python 3 and it used to work perfectly. Then I realized it needs to run on my Debian Stable server too, which offers only Python 2. Ok, most backporting was a matter of minutes, but I'm becoming desperate on some Unicode error...

i use scikit-learn to train a filter on a set of email messages:

vectorizer = CountVectorizer(input='filename', decode_error='replace', strip_accents='unicode',
preprocessor=self.mail_preprocessor, stop_words='english')

The vectorizer gets a list of filenames, reads them and passes them to the preprocessor:

def mail_preprocessor(self, message):
# Filter POPFile cruft by matching date string at the beginning.
print("Type:", type(message)) # imported from __future__
pop_reg = re.compile(r"^[0-9]{4}/[0-1][1-9]/[0-3]?[0-9]")
message = [line for line in message.splitlines(True) if not pop_reg.match(line)]
xxx = "".join(message)
msg = email.message_from_string(xxx) # <-- CRASH here

msg_body = ""

for part in msg.walk():
if part.get_content_type() in ["text/plain", "text/html"]:
body = part.get_payload(decode=True)
soup = BeautifulSoup(body)
msg_body += soup.get_text(" ", strip=True)

if "-----BEGIN PGP MESSAGE-----" in msg_body:
msg_body = ""

msg_body += " ".join(email.utils.parseaddr(msg["From"]))
msg_body += " " + msg["Subject"]
except TypeError: # Can't convert 'NoneType' object to str implicitly
msg_body = msg_body.lower()
return msg_body

Type: <type 'unicode'>

Traceback (most recent call last):
File "", line 182, in <module>
File "", line 161, in main
File "", line 73, in train
vectors = vectorizer.fit_transform(data[:,1])
File "/usr/lib/python2.7/site-packages/sklearn/feature_extraction/", line 780, in fit_transform
vocabulary, X = self._count_vocab(raw_documents, self.fixed_vocabulary)
File "/usr/lib/python2.7/site-packages/sklearn/feature_extraction/", line 715, in _count_vocab
for feature in analyze(doc):
File "/usr/lib/python2.7/site-packages/sklearn/feature_extraction/", line 229, in <lambda>
tokenize(preprocess(self.decode(doc))), stop_words)
File "", line 119, in mail_preprocessor
msg = email.message_from_string(xxx)
File "/usr/lib/python2.7/email/", line 57, in message_from_string
return Parser(*args, **kws).parsestr(s)
File "/usr/lib/python2.7/email/", line 82, in parsestr
return self.parse(StringIO(text), headersonly=headersonly)
UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 1624: ordinal not in range(128)

I've tried various modifications like encoding/decoding the message argument to utf-8.

Any help?




Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question