Program works great, except under less, cron or execl (Unicode?)

Discussion in 'Python' started by Sam, Sep 18, 2008.

  1. Sam

    Sam Guest

    I have a program which works great when run from the command line.

    But when I run it combined with something else such as:
    - piping it through less
    - cron
    - execl (i.e. calling it from another python program)

    it gives me a unicode error

    File "../myparser.py", line 261, in set_attributes
    print "self.atd['Name'] is: ", self.atd['Name']
    UnicodeEncodeError: 'ascii' codec can't encode character u'\xeb' in
    position 7: ordinal not in range(128)

    I'd post the whole program here, except it involves weird Unicode
    strings.

    I could probably change the program to get it working under less/cron/
    etc.

    But I'd rather understand exactly what the issue is. Why does it work
    fine when run directly from the command line, but not otherwise?
     
    Sam, Sep 18, 2008
    #1
    1. Advertising

  2. Sam wrote:

    > I have a program which works great when run from the command line.
    >
    > But when I run it combined with something else such as:
    > - piping it through less
    > - cron
    > - execl (i.e. calling it from another python program)
    >
    > it gives me a unicode error
    >
    > File "../myparser.py", line 261, in set_attributes
    > print "self.atd['Name'] is: ", self.atd['Name']
    > UnicodeEncodeError: 'ascii' codec can't encode character u'\xeb' in
    > position 7: ordinal not in range(128)
    >
    > I'd post the whole program here, except it involves weird Unicode
    > strings.
    >
    > I could probably change the program to get it working under less/cron/
    > etc.
    >
    > But I'd rather understand exactly what the issue is. Why does it work
    > fine when run directly from the command line, but not otherwise?


    Most probably because when to running directly inside a terminal, it gets
    it's stdin/stdout as pipes - and python can't attempt to guess the proper
    encoding on that, as it does on a terminal.

    And thus, when you print unicode to the pipe, it can't decide which encoding
    to use.

    To circumvene this, try & wrap stdout into a codecs-module wrapper with a
    proper encoding applied (e.g. utf-8).

    You might make that conditionally based on the sys.stdout.encoding-variable
    being set or not, albeit I'm not 100% sure to what it actually gets set
    when used in a subprocess. But this should give you the idea where to look.



    Diez
     
    Diez B. Roggisch, Sep 18, 2008
    #2
    1. Advertising

  3. > Most probably because when to running directly inside a terminal, it gets

    That was of course meant to be "not running directly inside a terminal".

    > it's stdin/stdout as pipes - and python can't attempt to guess the proper
    > encoding on that, as it does on a terminal.


    Diez
     
    Diez B. Roggisch, Sep 18, 2008
    #3
  4. Sam

    Sam Guest

    Diez for the win... :)

    sys.stdout.encoding does indeed have the proper value when called from
    the command line of UTF-8.

    But when piped into anything or called from anywhere it's None.

    Just for completeness, here's my test program:
    #!/usr/bin/env python
    import sys
    print sys.stdout.encoding

    And here are the results:
    $ ./encoding.py
    UTF-8
    $ ./encoding.py | cat
    None

    Really, really annoying!

    So how can I set sys.stdout.encoding so it's UTF-8 when piped through
    cat (or anything else).

    I tried assigning to it, but no dice.

    On Sep 18, 2:12 am, "Diez B. Roggisch" <> wrote:
    > Sam wrote:
    > > I have a program which works great when run from the command line.

    >
    > > But when I run it combined with something else such as:
    > > - piping it through less
    > > - cron
    > > - execl (i.e. calling it from another python program)

    >
    > > it gives me a unicode error

    >
    > >  File "../myparser.py", line 261, in set_attributes
    > >     print "self.atd['Name'] is: ", self.atd['Name']
    > > UnicodeEncodeError: 'ascii' codec can't encode character u'\xeb' in
    > > position 7: ordinal not in range(128)

    >
    > > I'd post the whole program here, except it involves weird Unicode
    > > strings.

    >
    > > I could probably change the program to get it working under less/cron/
    > > etc.

    >
    > > But I'd rather understand exactly what the issue is.  Why does it work
    > > fine when run directly from the command line, but not otherwise?

    >
    > Most probably because when to running directly inside a terminal, it gets
    > it's stdin/stdout as pipes - and python can't attempt to guess the proper
    > encoding on that, as it does on a terminal.
    >
    > And thus, when you print unicode to the pipe, it can't decide which encoding
    > to use.
    >
    > To circumvene this, try & wrap stdout into a codecs-module wrapper with a
    > proper encoding applied (e.g. utf-8).
    >
    > You might make that conditionally based on the sys.stdout.encoding-variable
    > being set or not, albeit I'm not 100% sure to what it actually gets set
    > when used in a subprocess. But this should give you the idea where to look.
    >
    > Diez
     
    Sam, Sep 19, 2008
    #4
  5. In message
    <>, Sam
    wrote:

    > So how can I set sys.stdout.encoding so it's UTF-8 when piped through
    > cat (or anything else).
    >
    > I tried assigning to it, but no dice.


    You could try wrapping it in a file object that does explicit encoding
    translation, using codecs.EncodedFile
    <http://docs.python.org/lib/module-codecs.html>.
     
    Lawrence D'Oliveiro, Sep 24, 2008
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John Salerno
    Replies:
    20
    Views:
    859
    John Salerno
    Aug 11, 2006
  2. Fabio Z Tessitore

    who is simpler? try/except/else or try/except

    Fabio Z Tessitore, Aug 12, 2007, in forum: Python
    Replies:
    5
    Views:
    378
  3. David House

    try -> except -> else -> except?

    David House, Jul 6, 2009, in forum: Python
    Replies:
    2
    Views:
    341
    Bruno Desthuilliers
    Jul 6, 2009
  4. Peng Yu
    Replies:
    1
    Views:
    522
    Steven D'Aprano
    Nov 18, 2009
  5. Eric I.
    Replies:
    0
    Views:
    264
    Eric I.
    Oct 5, 2008
Loading...

Share This Page