UnicodeEncodeError when piping stdout, but not when printingdirectly to the console

Discussion in 'Python' started by Adam Funk, Jan 4, 2012.

  1. Adam Funk

    Adam Funk Guest

    (I'm using Python 2.7.2+ on Ubuntu.)

    When I'm running my program in an xterm, the print command with an
    argument containing unicode works fine (it correctly detects my UTF-8
    environment). But when I run it with a pipe or redirect to a file (|
    or >), unicode strings fail with the following (for example):

    UnicodeEncodeError: 'ascii' codec can't encode character u'\u0107' in position 21: ordinal not in range(128)

    How can I force python (preferably within my python program, rather
    than having to set something externally) to treat stdout as UTF-8?


    Thanks,
    Adam


    --
    Nam Sibbyllam quidem Cumis ego ipse oculis meis vidi in ampulla
    pendere, et cum illi pueri dicerent: beable beable beable; respondebat
    illa: doidy doidy doidy. [plorkwort]
     
    Adam Funk, Jan 4, 2012
    #1
    1. Advertising

  2. Adam Funk

    Peter Otten Guest

    Re: UnicodeEncodeError when piping stdout,but not when printing directly to the console

    Adam Funk wrote:

    > (I'm using Python 2.7.2+ on Ubuntu.)
    >
    > When I'm running my program in an xterm, the print command with an
    > argument containing unicode works fine (it correctly detects my UTF-8
    > environment). But when I run it with a pipe or redirect to a file (|
    > or >), unicode strings fail with the following (for example):
    >
    > UnicodeEncodeError: 'ascii' codec can't encode character u'\u0107' in
    > position 21: ordinal not in range(128)
    >
    > How can I force python (preferably within my python program, rather
    > than having to set something externally) to treat stdout as UTF-8?



    $ cat force_utf8.py
    # -*- coding: utf-8 -*-
    import sys

    if sys.stdout.encoding is None:
    import codecs
    writer = codecs.getwriter("utf-8")
    sys.stdout = writer(sys.stdout)

    print u"Ähnlich üblich nötig"

    $ python force_utf8.py
    Ähnlich üblich nötig

    $ python force_utf8.py | cat
    Ähnlich üblich nötig
     
    Peter Otten, Jan 4, 2012
    #2
    1. Advertising

  3. Adam Funk

    Adam Funk Guest

    On 2012-01-04, Peter Otten wrote:

    > Adam Funk wrote:


    >> How can I force python (preferably within my python program, rather
    >> than having to set something externally) to treat stdout as UTF-8?

    >
    >
    > $ cat force_utf8.py
    > # -*- coding: utf-8 -*-
    > import sys
    >
    > if sys.stdout.encoding is None:
    > import codecs
    > writer = codecs.getwriter("utf-8")
    > sys.stdout = writer(sys.stdout)
    >
    > print u"Ähnlich üblich nötig"


    That's great, thanks!

    I guess issues like this will magically go away when I eventually move
    to Python 3?


    --
    Physics is like sex. Sure, it may give some practical results,
    but that's not why we do it. [Richard Feynman]
     
    Adam Funk, Jan 6, 2012
    #3
  4. Adam Funk

    Peter Otten Guest

    Re: UnicodeEncodeError when piping stdout,but not when printing directly to the console

    Adam Funk wrote:

    > On 2012-01-04, Peter Otten wrote:
    >
    >> Adam Funk wrote:

    >
    >>> How can I force python (preferably within my python program, rather
    >>> than having to set something externally) to treat stdout as UTF-8?

    >>
    >>
    >> $ cat force_utf8.py
    >> # -*- coding: utf-8 -*-
    >> import sys
    >>
    >> if sys.stdout.encoding is None:
    >> import codecs
    >> writer = codecs.getwriter("utf-8")
    >> sys.stdout = writer(sys.stdout)
    >>
    >> print u"Ähnlich üblich nötig"

    >
    > That's great, thanks!
    >
    > I guess issues like this will magically go away when I eventually move
    > to Python 3?


    Not "magically", but UTF-8 has become the default encoding...
     
    Peter Otten, Jan 6, 2012
    #4
  5. Adam Funk

    Adam Funk Guest

    On 2012-01-06, Peter Otten wrote:

    > Adam Funk wrote:
    >
    >> On 2012-01-04, Peter Otten wrote:
    >>
    >>> Adam Funk wrote:

    >>
    >>>> How can I force python (preferably within my python program, rather
    >>>> than having to set something externally) to treat stdout as UTF-8?
    >>>
    >>>
    >>> $ cat force_utf8.py
    >>> # -*- coding: utf-8 -*-
    >>> import sys
    >>>
    >>> if sys.stdout.encoding is None:
    >>> import codecs
    >>> writer = codecs.getwriter("utf-8")
    >>> sys.stdout = writer(sys.stdout)
    >>>
    >>> print u"Ähnlich üblich nötig"

    >>
    >> That's great, thanks!
    >>
    >> I guess issues like this will magically go away when I eventually move
    >> to Python 3?

    >
    > Not "magically", but UTF-8 has become the default encoding...


    Close enough!



    --
    When Elaine turned 11, her mother sent her to train under
    Donald Knuth in his mountain hideaway. [XKCD 342]
     
    Adam Funk, Jan 6, 2012
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Edward Diener

    Piping stdout to Python callable

    Edward Diener, Aug 17, 2004, in forum: Python
    Replies:
    7
    Views:
    462
    Antoon Pardon
    Aug 19, 2004
  2. Francach
    Replies:
    2
    Views:
    36,518
    Diez B. Roggisch
    Nov 6, 2005
  3. James McGill
    Replies:
    1
    Views:
    311
    Fredrik Lundh
    Sep 8, 2008
  4. Magnus Pettersson
    Replies:
    17
    Views:
    317
    Magnus Pettersson
    Feb 13, 2013
  5. Luca Cerone
    Replies:
    13
    Views:
    148
    Tobiah
    Aug 5, 2013
Loading...

Share This Page