UnicodeEncodeError when piping stdout, but not when printingdirectly to the console

[Adam Funk]

(I'm using Python 2.7.2+ on Ubuntu.)

When I'm running my program in an xterm, the print command with an
argument containing unicode works fine (it correctly detects my UTF-8
environment). But when I run it with a pipe or redirect to a file (|
or >), unicode strings fail with the following (for example):

UnicodeEncodeError: 'ascii' codec can't encode character u'\u0107' in position 21: ordinal not in range(128)

How can I force python (preferably within my python program, rather
than having to set something externally) to treat stdout as UTF-8?


Thanks,
Adam

 

[Peter Otten]

(I'm using Python 2.7.2+ on Ubuntu.)

When I'm running my program in an xterm, the print command with an
argument containing unicode works fine (it correctly detects my UTF-8
environment). But when I run it with a pipe or redirect to a file (|
or >), unicode strings fail with the following (for example):

UnicodeEncodeError: 'ascii' codec can't encode character u'\u0107' in
position 21: ordinal not in range(128)

How can I force python (preferably within my python program, rather
than having to set something externally) to treat stdout as UTF-8?

$ cat force_utf8.py
# -*- coding: utf-8 -*-
import sys

if sys.stdout.encoding is None:
import codecs
writer = codecs.getwriter("utf-8")
sys.stdout = writer(sys.stdout)

print u"Ähnlich üblich nötig"

$ python force_utf8.py
Ähnlich üblich nötig

$ python force_utf8.py | cat
Ähnlich üblich nötig

 

[Adam Funk]

Adam Funk wrote:


$ cat force_utf8.py
# -*- coding: utf-8 -*-
import sys

if sys.stdout.encoding is None:
import codecs
writer = codecs.getwriter("utf-8")
sys.stdout = writer(sys.stdout)

print u"Ähnlich üblich nötig"

That's great, thanks!

I guess issues like this will magically go away when I eventually move
to Python 3?

 

[Peter Otten]

That's great, thanks!

I guess issues like this will magically go away when I eventually move
to Python 3?

Not "magically", but UTF-8 has become the default encoding...

 

[Adam Funk]

Not "magically", but UTF-8 has become the default encoding...

Close enough!