processing a large utf-8 file

[Ivan Voras]

Since the .encoding attribute of file objects are read-only, what is the
proper way to process large utf-8 text files?

I need "bulk" processing (i.e. in blocks - the file is ~ 1GB), but
reading it in fixed blocks is bound to result in partially-read utf-8
characters at block boundaries.

 

[Guest]

Since the .encoding attribute of file objects are read-only, what is the
proper way to process large utf-8 text files?

You should use codecs.open, or codecs.getreader to get a StreamReader
for UTF-8.

Regards,
Martin