unicode mystery/problem

Discussion in 'Python' started by Petr Jakes, Sep 20, 2006.

  1. Petr Jakes

    Petr Jakes Guest

    Hi,
    I am using Python 2.4.3 on Fedora Core4 and "Eric3" Python IDE
    ..
    Below mentioned code works fine in the Eric3 environment. While trying
    to start it from the command line, it returns:

    Traceback (most recent call last):
    File "pokus_1.py", line 5, in ?
    print str(a)
    UnicodeEncodeError: 'ascii' codec can't encode character u'\xc1' in
    position 6: ordinal not in range(128)

    ========== 8< =============
    #!/usr/bin python
    # -*- Encoding: utf_8 -*-

    a= u'DISKOV\xc1 POLE'
    print a
    print str(a)
    ========== 8< =============

    Even it looks strange, I have to use str(a) syntax even I know the "a"
    variable is a string.
    I am trying to use ChartDirector for Python (charts for Python) and the
    method "layer.addDataSet()" needs above mentioned syntax otherwise it
    returns an Error.

    layer.addDataSet(data, colour, str(dataName))

    Thanks for your comments

    Petr Jakes
     
    Petr Jakes, Sep 20, 2006
    #1
    1. Advertising

  2. Petr Jakes

    John Machin Guest

    Petr Jakes wrote:
    > Hi,
    > I am using Python 2.4.3 on Fedora Core4 and "Eric3" Python IDE
    > .
    > Below mentioned code works fine in the Eric3 environment. While trying
    > to start it from the command line, it returns:
    >
    > Traceback (most recent call last):
    > File "pokus_1.py", line 5, in ?
    > print str(a)
    > UnicodeEncodeError: 'ascii' codec can't encode character u'\xc1' in
    > position 6: ordinal not in range(128)


    So print a works, but print str(a) crashes.

    Instead, insert this:
    import sys
    print "default", sys.getdefaultencoding()
    print "stdout", sys.stdout.encoding
    and run your script at the command line. It should print:
    default ascii
    stdout x
    here, and crash at the later use of str(a).
    Step 2: run your script under Eric3. It will print:
    default y
    stdout z
    and then should work properly. It is probable that x == y == z ==
    'utf-8'
    Step 3: see below.

    >
    > ========== 8< =============
    > #!/usr/bin python
    > # -*- Encoding: utf_8 -*-


    There is no UTF8-encoded text in this short test script. Is the above
    encoding comment merely a carry-over from your real script, or do you
    believe it is necessary or useful in this test script?

    >
    > a= u'DISKOV\xc1 POLE'
    > print a
    > print str(a)
    > ========== 8< =============
    >
    > Even it looks strange, I have to use str(a) syntax even I know the "a"
    > variable is a string.


    Some concepts you need to understand:
    (a) "a" is not a string, it is a reference to a string.
    (b) It is a reference to a unicode object (an implementation of a
    conceptual Unicode string) ...
    (c) which must be distinguished from a str object, which represents a
    conceptual string of bytes.
    (d) str(a) is trying to produce a str object from a unicode object. Not
    being told what encoding to use, it uses the default encoding
    (typically ascii) and naturally this will crash if there are non-ascii
    characters in the unicode object.

    > I am trying to use ChartDirector for Python (charts for Python) and the
    > method "layer.addDataSet()" needs above mentioned syntax otherwise it
    > returns an Error.


    Care to tell us which error???

    >
    > layer.addDataSet(data, colour, str(dataName))


    The method presumably expects a str object (8-bit string). What does
    its documentation say? Again, what error message do you get if you feed
    it a unicode object with non-ascii characters?

    [Step 3] For foo in set(['x', 'y', 'z']):
    Change str(dataName) to dataName.encode(foo). Change any debugging
    display to use repr(a) instead of str(a). Test it with both Eric3 and
    the command line.

    [Aside: it's entirely possible that your problem will go away if you
    remove the letter u from the line a= u'DISKOV\xc1 POLE' -- however if
    you want to understand what is happening generally, I suggest you don't
    do that]

    HTH,
    John
     
    John Machin, Sep 20, 2006
    #2
    1. Advertising

  3. Re[2]: unicode mystery/problem

    John, thanks for your extensive answer.
    >> Hi,
    >> I am using Python 2.4.3 on Fedora Core4 and "Eric3" Python IDE
    >> .
    >> Below mentioned code works fine in the Eric3 environment. While trying
    >> to start it from the command line, it returns:
    >>
    >> Traceback (most recent call last):
    >> File "pokus_1.py", line 5, in ?
    >> print str(a)
    >> UnicodeEncodeError: 'ascii' codec can't encode character u'\xc1' in
    >> position 6: ordinal not in range(128)


    JM> So print a works, but print str(a) crashes.

    JM> Instead, insert this:
    JM> import sys
    JM> print "default", sys.getdefaultencoding()
    JM> print "stdout", sys.stdout.encoding
    JM> and run your script at the command line. It should print:
    JM> default ascii
    JM> stdout x
    **** in the command line it prints: *****
    default ascii
    stdout UTF-8
    JM> here, and crash at the later use of str(a).
    JM> Step 2: run your script under Eric3. It will print:
    JM> default y
    JM> stdout z

    **** in the Eric3 it prints: ****
    if the # -*- Eencoding: utf_8 -*- is set than:

    default utf_8
    stdout
    unhandled AttributeError, "AsyncFile instance has no attribute
    'encoding' "

    if the encoding is not set than it prints:

    DeprecationWarning: Non-ASCII character '\xc3' in file
    /root/eric/analyza_dat_TPC/pokus_1.py on line 26, but no encoding
    declared; see http://www.python.org/peps/pep-0263.html for details execfile(sys.argv[0], self.debugMod.__dict__)

    default latin-1
    stdout
    unhandled AttributeError, "AsyncFile instance has no attribute
    'encoding' "

    JM> and then should work properly. It is probable that x == y == z ==
    JM> 'utf-8'
    JM> Step 3: see below.

    >>
    >> ========== 8< =============
    >> #!/usr/bin python
    >> # -*- Encoding: utf_8 -*-


    JM> There is no UTF8-encoded text in this short test script. Is the above
    JM> encoding comment merely a carry-over from your real script, or do you
    JM> believe it is necessary or useful in this test script?
    Generally, I am working with string like u'DISKOV\xc1 POLE' (I am
    getting it from the database)

    My intention to use >> # -*- Encoding: utf_8 -*- was to suppress
    DeprecationWarnings if I use utf_8 in the code (like u'DISKOV\xc1 POLE')

    >>
    >> a= u'DISKOV\xc1 POLE'
    >> print a
    >> print str(a)
    >> ========== 8< =============
    >>
    >> Even it looks strange, I have to use str(a) syntax even I know the "a"
    >> variable is a string.


    JM> Some concepts you need to understand:
    JM> (a) "a" is not a string, it is a reference to a string.
    JM> (b) It is a reference to a unicode object (an implementation of a
    JM> conceptual Unicode string) ...
    JM> (c) which must be distinguished from a str object, which represents a
    JM> conceptual string of bytes.
    JM> (d) str(a) is trying to produce a str object from a unicode object. Not
    JM> being told what encoding to use, it uses the default encoding
    JM> (typically ascii) and naturally this will crash if there are non-ascii
    JM> characters in the unicode object.

    >> I am trying to use ChartDirector for Python (charts for Python) and the
    >> method "layer.addDataSet()" needs above mentioned syntax otherwise it
    >> returns an Error.


    JM> Care to tell us which error???
    you can see the Error description and author comments here:
    http://tinyurl.com/ezohe

    >>
    >> layer.addDataSet(data, colour, str(dataName))

    I have try to experiment with the code a bit.
    the simplest code where I can demonstrate my problems:
    #!/usr/bin python
    import sys
    print "default", sys.getdefaultencoding()
    print "stdout", sys.stdout.encoding

    a=['P\xc5\x99\xc3\xad','Petr Jake\xc5\xa1']
    b="my nice try %s" % ''.join(a).encode("utf-8")
    print b

    When I run it from the command line i am getting:
    sys:1: DeprecationWarning: Non-ASCII character '\xc3' in file pokus_1.py on line 26, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

    default ascii
    stdout UTF-8

    Traceback (most recent call last):
    File "pokus_1.py", line 8, in ?
    b="my nice try %s" % ''.join(a).encode("utf-8")
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 1: ordinal not in range(128)



    JM> The method presumably expects a str object (8-bit string). What does
    JM> its documentation say? Again, what error message do you get if you feed
    JM> it a unicode object with non-ascii characters?

    JM> [Step 3] For foo in set(['x', 'y', 'z']):
    JM> Change str(dataName) to dataName.encode(foo). Change any debugging
    JM> display to use repr(a) instead of str(a). Test it with both Eric3 and
    JM> the command line.

    JM> [Aside: it's entirely possible that your problem will go away if you
    JM> remove the letter u from the line a= u'DISKOV\xc1 POLE' -- however if
    JM> you want to understand what is happening generally, I suggest you don't
    JM> do that]

    JM> HTH,
    JM> John
     
    =?windows-1250?Q?Petr_Jake=9A?=, Sep 22, 2006
    #3
  4. Re: Re[2]: unicode mystery/problem

    In <>, Petr Jakeš
    wrote:

    > I have try to experiment with the code a bit.
    > the simplest code where I can demonstrate my problems:
    > #!/usr/bin python
    > import sys
    > print "default", sys.getdefaultencoding()
    > print "stdout", sys.stdout.encoding
    >
    > a=['P\xc5\x99\xc3\xad','Petr Jake\xc5\xa1']
    > b="my nice try %s" % ''.join(a).encode("utf-8")


    You have two byte strings in the list `a` and try to *encode* them as
    utf-8. That does not work. You can make the example even a bit simpler::

    'P\xc5\x99\xc3\xadPetr Jake\xc5\xa1'.encode('utf-8')

    You cant't *encode* byte strings, just *decode* them. What happens is
    that Python tries to make a unicode string from the byte string to encode
    that in utf-8. But it decodes as ASCII as that is the default.

    Don't mix byte strings and unicode strings. Put an encoding declaration
    at the top of your file and convert everything to unicode on the "way in"
    and to the proper encoding on the "way out" of your program.

    Ciao,
    Marc 'BlackJack' Rintsch
     
    Marc 'BlackJack' Rintsch, Sep 22, 2006
    #4
  5. Petr Jakes

    John Machin Guest

    Re: Re[2]: unicode mystery/problem

    Petr Jakeš wrote:
    > John, thanks for your extensive answer.
    > >> Hi,
    > >> I am using Python 2.4.3 on Fedora Core4 and "Eric3" Python IDE
    > >> .
    > >> Below mentioned code works fine in the Eric3 environment. While trying
    > >> to start it from the command line, it returns:
    > >>
    > >> Traceback (most recent call last):
    > >> File "pokus_1.py", line 5, in ?
    > >> print str(a)
    > >> UnicodeEncodeError: 'ascii' codec can't encode character u'\xc1' in
    > >> position 6: ordinal not in range(128)

    >
    > JM> So print a works, but print str(a) crashes.
    >
    > JM> Instead, insert this:
    > JM> import sys
    > JM> print "default", sys.getdefaultencoding()
    > JM> print "stdout", sys.stdout.encoding
    > JM> and run your script at the command line. It should print:
    > JM> default ascii
    > JM> stdout x
    > **** in the command line it prints: *****
    > default ascii
    > stdout UTF-8
    > JM> here, and crash at the later use of str(a).
    > JM> Step 2: run your script under Eric3. It will print:
    > JM> default y
    > JM> stdout z
    >
    > **** in the Eric3 it prints: ****
    > if the # -*- Eencoding: utf_8 -*- is set than:
    >
    > default utf_8
    > stdout
    > unhandled AttributeError, "AsyncFile instance has no attribute
    > 'encoding' "
    >
    > if the encoding is not set than it prints:
    >
    > DeprecationWarning: Non-ASCII character '\xc3' in file
    > /root/eric/analyza_dat_TPC/pokus_1.py on line 26, but no encoding
    > declared; see http://www.python.org/peps/pep-0263.html for details execfile(sys.argv[0], self.debugMod.__dict__)
    >
    > default latin-1
    > stdout
    > unhandled AttributeError, "AsyncFile instance has no attribute
    > 'encoding' "
    >
    > JM> and then should work properly. It is probable that x == y == z ==
    > JM> 'utf-8'
    > JM> Step 3: see below.
    >
    > >>
    > >> ========== 8< =============
    > >> #!/usr/bin python
    > >> # -*- Encoding: utf_8 -*-

    >
    > JM> There is no UTF8-encoded text in this short test script. Is the above
    > JM> encoding comment merely a carry-over from your real script, or do you
    > JM> believe it is necessary or useful in this test script?
    > Generally, I am working with string like u'DISKOV\xc1 POLE' (I am
    > getting it from the database)
    >
    > My intention to use >> # -*- Encoding: utf_8 -*- was to suppress
    > DeprecationWarnings if I use utf_8 in the code (like u'DISKOV\xc1 POLE')
    >
    > >>
    > >> a= u'DISKOV\xc1 POLE'
    > >> print a
    > >> print str(a)
    > >> ========== 8< =============
    > >>
    > >> Even it looks strange, I have to use str(a) syntax even I know the "a"
    > >> variable is a string.

    >
    > JM> Some concepts you need to understand:
    > JM> (a) "a" is not a string, it is a reference to a string.
    > JM> (b) It is a reference to a unicode object (an implementation of a
    > JM> conceptual Unicode string) ...
    > JM> (c) which must be distinguished from a str object, which represents a
    > JM> conceptual string of bytes.
    > JM> (d) str(a) is trying to produce a str object from a unicode object. Not
    > JM> being told what encoding to use, it uses the default encoding
    > JM> (typically ascii) and naturally this will crash if there are non-ascii
    > JM> characters in the unicode object.
    >
    > >> I am trying to use ChartDirector for Python (charts for Python) and the
    > >> method "layer.addDataSet()" needs above mentioned syntax otherwise it
    > >> returns an Error.

    >
    > JM> Care to tell us which error???
    > you can see the Error description and author comments here:
    > http://tinyurl.com/ezohe


    You have two different episodes on that website; adding the one we have
    been discussing gives *three* different stories:

    Episode 1:

    The error description: "TypeError: Error converting argument 1 to type
    PCc" -- you should ask him "What is type PCc???" If arg 1 is an
    arbitrary str object, which byte values could it possibly be objecting
    to?

    The author comments: "The error code usually means the filename is not
    a text string, ..." (1) Input file or output file? Is it possible that
    one or more bytes are not allowable in a filename? (2) Is it possible
    for you to give him the exact args that you are passing in (use print
    repr(arg) before the call), and for him to tell you the *exact* reason,
    not the "usual" reason?

    Episode 2: Evidently arg is a str object, but passing in str(arg) and
    just plain arg give different results??? I doubt it. print repr(arg)
    and type(arg) and see what you've actually got there.

    >
    > >>
    > >> layer.addDataSet(data, colour, str(dataName))

    > I have try to experiment with the code a bit.
    > the simplest code where I can demonstrate my problems:
    > #!/usr/bin python
    > import sys
    > print "default", sys.getdefaultencoding()
    > print "stdout", sys.stdout.encoding
    >
    > a=['P\xc5\x99\xc3\xad','Petr Jake\xc5\xa1']
    > b="my nice try %s" % ''.join(a).encode("utf-8")


    So ''.join(a) is a str object, encoded in utf-8 *already*.
    Please try to understand:
    (1) unicode_object.encode('utf-8') produces a str_object # in utf-8
    encoding
    (2) str_object.decode('utf-8') produces a unicode object # if
    str_object contains valid utf-8.
    (3) str_object.encode('anything') is a nonsense; it is the equivalent
    of str_object.decode('ascii').encode('anything') and will typically
    fail, as your next error message shows.

    What were you trying to do?? I don't understand the relationship
    between this little exercise and Episodes 1, 2, & 3.

    Try to concentrate on what your data is (u"DISKOetcetc" is a unicode
    string, but then you say that str(x) should be unnecessary because x is
    already a str object!?) and what you need to have to get it passed
    through to that package's methods.


    > print b
    >
    > When I run it from the command line i am getting:
    > sys:1: DeprecationWarning: Non-ASCII character '\xc3' in file pokus_1.py on line 26, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
    >
    > default ascii
    > stdout UTF-8
    >
    > Traceback (most recent call last):
    > File "pokus_1.py", line 8, in ?
    > b="my nice try %s" % ''.join(a).encode("utf-8")
    > UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 1: ordinal not in range(128)
    >


    As expected.

    Regards,
    John
     
    John Machin, Sep 22, 2006
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. valentin tihomirov

    another array ranges mystery

    valentin tihomirov, Jun 18, 2005, in forum: VHDL
    Replies:
    2
    Views:
    486
    Mike Treseler
    Jun 18, 2005
  2. John K. Humkey

    sed regexp mystery

    John K. Humkey, Jul 8, 2003, in forum: Perl
    Replies:
    0
    Views:
    2,053
    John K. Humkey
    Jul 8, 2003
  3. bjh

    Mystery Repost

    bjh, Jan 13, 2004, in forum: Perl
    Replies:
    0
    Views:
    736
  4. Todd Plambeck

    Mystery Error

    Todd Plambeck, Jul 15, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    517
    Victor Garcia Aprea [MVP]
    Jul 15, 2003
  5. Sean McIlroy

    unicode mystery

    Sean McIlroy, Jan 11, 2005, in forum: Python
    Replies:
    20
    Views:
    854
    Bengt Richter
    Jan 14, 2005
Loading...

Share This Page