F
Florian Lindner
Hello!
I've written some tiny script using Python 3 and it used to work perfectly. Then I realized it needs to run on my Debian Stable server too, which offers only Python 2. Ok, most backporting was a matter of minutes, but I'm becoming desperate on some Unicode error...
i use scikit-learn to train a filter on a set of email messages:
vectorizer = CountVectorizer(input='filename', decode_error='replace', strip_accents='unicode',
preprocessor=self.mail_preprocessor, stop_words='english')
http://scikit-learn.org/stable/modu....feature_extraction.text.CountVectorizer.html
The vectorizer gets a list of filenames, reads them and passes them to the preprocessor:
def mail_preprocessor(self, message):
# Filter POPFile cruft by matching date string at the beginning.
print("Type:", type(message)) # imported from __future__
pop_reg = re.compile(r"^[0-9]{4}/[0-1][1-9]/[0-3]?[0-9]")
message = [line for line in message.splitlines(True) if not pop_reg.match(line)]
xxx = "".join(message)
msg = email.message_from_string(xxx) # <-- CRASH here
msg_body = ""
for part in msg.walk():
if part.get_content_type() in ["text/plain", "text/html"]:
body = part.get_payload(decode=True)
soup = BeautifulSoup(body)
msg_body += soup.get_text(" ", strip=True)
if "-----BEGIN PGP MESSAGE-----" in msg_body:
msg_body = ""
msg_body += " ".join(email.utils.parseaddr(msg["From"]))
try:
msg_body += " " + msg["Subject"]
except TypeError: # Can't convert 'NoneType' object to str implicitly
pass
msg_body = msg_body.lower()
return msg_body
Type: <type 'unicode'>
Traceback (most recent call last):
File "flofify.py", line 182, in <module>
main()
File "flofify.py", line 161, in main
model.train()
File "flofify.py", line 73, in train
vectors = vectorizer.fit_transform(data[:,1])
File "/usr/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 780, in fit_transform
vocabulary, X = self._count_vocab(raw_documents, self.fixed_vocabulary)
File "/usr/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 715, in _count_vocab
for feature in analyze(doc):
File "/usr/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 229, in <lambda>
tokenize(preprocess(self.decode(doc))), stop_words)
File "flofify.py", line 119, in mail_preprocessor
msg = email.message_from_string(xxx)
File "/usr/lib/python2.7/email/__init__.py", line 57, in message_from_string
return Parser(*args, **kws).parsestr(s)
File "/usr/lib/python2.7/email/parser.py", line 82, in parsestr
return self.parse(StringIO(text), headersonly=headersonly)
UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 1624: ordinal not in range(128)
I've tried various modifications like encoding/decoding the message argument to utf-8.
Any help?
Thanks!
Florian
I've written some tiny script using Python 3 and it used to work perfectly. Then I realized it needs to run on my Debian Stable server too, which offers only Python 2. Ok, most backporting was a matter of minutes, but I'm becoming desperate on some Unicode error...
i use scikit-learn to train a filter on a set of email messages:
vectorizer = CountVectorizer(input='filename', decode_error='replace', strip_accents='unicode',
preprocessor=self.mail_preprocessor, stop_words='english')
http://scikit-learn.org/stable/modu....feature_extraction.text.CountVectorizer.html
The vectorizer gets a list of filenames, reads them and passes them to the preprocessor:
def mail_preprocessor(self, message):
# Filter POPFile cruft by matching date string at the beginning.
print("Type:", type(message)) # imported from __future__
pop_reg = re.compile(r"^[0-9]{4}/[0-1][1-9]/[0-3]?[0-9]")
message = [line for line in message.splitlines(True) if not pop_reg.match(line)]
xxx = "".join(message)
msg = email.message_from_string(xxx) # <-- CRASH here
msg_body = ""
for part in msg.walk():
if part.get_content_type() in ["text/plain", "text/html"]:
body = part.get_payload(decode=True)
soup = BeautifulSoup(body)
msg_body += soup.get_text(" ", strip=True)
if "-----BEGIN PGP MESSAGE-----" in msg_body:
msg_body = ""
msg_body += " ".join(email.utils.parseaddr(msg["From"]))
try:
msg_body += " " + msg["Subject"]
except TypeError: # Can't convert 'NoneType' object to str implicitly
pass
msg_body = msg_body.lower()
return msg_body
Type: <type 'unicode'>
Traceback (most recent call last):
File "flofify.py", line 182, in <module>
main()
File "flofify.py", line 161, in main
model.train()
File "flofify.py", line 73, in train
vectors = vectorizer.fit_transform(data[:,1])
File "/usr/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 780, in fit_transform
vocabulary, X = self._count_vocab(raw_documents, self.fixed_vocabulary)
File "/usr/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 715, in _count_vocab
for feature in analyze(doc):
File "/usr/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 229, in <lambda>
tokenize(preprocess(self.decode(doc))), stop_words)
File "flofify.py", line 119, in mail_preprocessor
msg = email.message_from_string(xxx)
File "/usr/lib/python2.7/email/__init__.py", line 57, in message_from_string
return Parser(*args, **kws).parsestr(s)
File "/usr/lib/python2.7/email/parser.py", line 82, in parsestr
return self.parse(StringIO(text), headersonly=headersonly)
UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 1624: ordinal not in range(128)
I've tried various modifications like encoding/decoding the message argument to utf-8.
Any help?
Thanks!
Florian