"Decoding unicode is not supported" in unusual situation

Discussion in 'Python' started by John Nagle, Mar 7, 2012.

  1. John Nagle

    John Nagle Guest

    I'm getting

    line 79, in tounicode
    return(unicode(s, errors='replace'))
    TypeError: decoding Unicode is not supported

    from this, under Python 2.7:

    def tounicode(s) :
    if type(s) == unicode :
    return(s)
    return(unicode(s, errors='replace'))

    That would seem to be impossible. But it's not.
    "s" is generated from the "suds" SOAP client. The documentation
    for "suds" says:

    "Suds leverages python meta programming to provide an intuative API for
    consuming web services. Runtime objectification of types defined in the
    WSDL is provided without class generation."

    I think that somewhere in "suds", they subclass the "unicode" type.
    That's almost too cute.

    The proper test is

    isinstance(s,unicode)


    John Nagle
    John Nagle, Mar 7, 2012
    #1
    1. Advertising

  2. John Nagle <> writes:

    > I think that somewhere in "suds", they subclass the "unicode" type.
    > That's almost too cute.
    >
    > The proper test is
    >
    > isinstance(s,unicode)



    Woot, you finally discovered polymorphism - congratulations!

    Diez
    Diez B. Roggisch, Mar 7, 2012
    #2
    1. Advertising

  3. On Wed, 07 Mar 2012 22:18:50 +1100, Ben Finney wrote:

    > (Diez B. Roggisch) writes:
    >
    >> John Nagle <> writes:
    >>
    >> > I think that somewhere in "suds", they subclass the "unicode" type.
    >> > That's almost too cute.
    >> >
    >> > The proper test is
    >> >
    >> > isinstance(s,unicode)

    >>
    >> Woot, you finally discovered polymorphism - congratulations!

    >
    > If by “discovered†you mean “brokeâ€.
    >
    > John, polymorphism entails that it *doesn't matter* whether the object
    > inherits from any particular type; it only matters whether the object
    > behaves correctly.
    >
    > So rather than testing whether the object inherits from ‘unicode’, test
    > whether it behaves how you expect – preferably by just using it as
    > though it does behave that way.



    I must admit that I can't quite understand John Nagle's original post, so
    I could be wrong, but I *think* that both you and Diez have misunderstood
    the nature of John's complaint.

    I *think* he is complaining that some other library -- suds? -- has a
    broken test for Unicode, by using:

    if type(s) is unicode: ...

    instead of

    if isinstance(s, unicode): ...

    Consequently, when the library passes a unicode *subclass* to the
    tounicode function, the "type() is unicode" test fails. That's a bad bug.

    It's arguable that the library shouldn't even use isinstance, but that's
    an argument for another day.


    --
    Steven
    Steven D'Aprano, Mar 7, 2012
    #3
  4. John Nagle

    John Nagle Guest

    On 3/7/2012 3:42 AM, Steven D'Aprano wrote:

    > I *think* he is complaining that some other library -- suds? -- has a
    > broken test for Unicode, by using:
    >
    > if type(s) is unicode: ...
    >
    > instead of
    >
    > if isinstance(s, unicode): ...
    >
    > Consequently, when the library passes a unicode *subclass* to the
    > tounicode function, the "type() is unicode" test fails. That's a bad bug.


    No, that was my bug.

    The library bug, if any, is that you can't apply

    unicode(s, errors='replace')

    to a Unicode string. TypeError("Decoding unicode is not supported") is
    raised. However

    unicode(s)

    will accept Unicode input.

    The Python documentation
    ("http://docs.python.org/library/functions.html#unicode") does not
    mention this. It is therefore necessary to check the type before
    calling "unicode", or catch the undocumented TypeError exception
    afterward.


    John Nagle
    John Nagle, Mar 7, 2012
    #4
  5. On Thu, 08 Mar 2012 08:48:58 +1100, Ben Finney wrote:

    > John Nagle <> writes:
    >
    >> The library bug, if any, is that you can't apply
    >>
    >> unicode(s, errors='replace')
    >>
    >> to a Unicode string. TypeError("Decoding unicode is not supported") is
    >> raised. However
    >>
    >> unicode(s)
    >>
    >> will accept Unicode input.

    >
    > I think that's a Python bug. If the latter succeeds as a no-op, the
    > former should also succeed as a no-op. Neither should ever get any
    > errors when ‘s’ is a ‘unicode’ object already.


    No. The semantics of the unicode function (technically: a type
    constructor) are well-defined, and there are two distinct behaviours:

    unicode(obj)

    is analogous to str(obj), and it attempts to convert obj to a unicode
    string by calling obj.__unicode__, if it exists, or __str__ if it
    doesn't. No encoding or decoding is attempted in the event that obj is a
    unicode instance.

    unicode(obj, encoding, errors)

    is explicitly stated in the docs as decoding obj if EITHER of encoding or
    errors is given, AND that obj must be either an 8-bit string (bytes) or a
    buffer object.

    It is true that u''.decode() will succeed, in Python 2, but the fact that
    unicode objects have a decode method at all is IMO a bug. It has also
    been corrected in Python 3, where (unicode) str objects no longer have a
    decode method, and bytes objects no longer have an encode method.


    >> The Python documentation
    >> ("http://docs.python.org/library/functions.html#unicode") does not
    >> mention this.


    Yes it does. It is is the SECOND sentence, immediately after the summary
    line:

    unicode([object[, encoding[, errors]]])
    Return the Unicode string version of object using one of the
    following modes:

    If encoding and/or errors are given, unicode() will decode the object
    which can either be an 8-bit string or a character buffer using the
    codec for encoding. ...


    Admittedly, it doesn't *explicitly* state that TypeError will be raised,
    but what other exception kind would you expect when you supply an
    argument of the wrong type?


    >> It is therefore necessary to check the type before
    >> calling "unicode", or catch the undocumented TypeError exception
    >> afterward.

    >
    > Yes, this check should not be necessary; calling the ‘unicode’
    > constructor with an object that's already an instance of ‘unicode’
    > should just return the object as-is, IMO. It shouldn't matter that
    > you've specified how decoding errors are to be handled, because in that
    > case no decoding happens anyway.


    I don't believe that it is the job of unicode() to Do What I Mean, but
    only to Do What I Say. If I *explicitly* tell unicode() to decode the
    argument (by specifying either the codec or the error handler or both)
    then it should not double-guess me and ignore the extra parameters.

    End-user applications may, with care, try to be smart and DWIM, but
    library functions should be dumb and should do what they are told.



    --
    Steven
    Steven D'Aprano, Mar 7, 2012
    #5
  6. John Nagle

    Terry Reedy Guest

    On 3/7/2012 6:26 PM, Steven D'Aprano wrote:
    > On Thu, 08 Mar 2012 08:48:58 +1100, Ben Finney wrote:
    >
    >> John Nagle<> writes:
    >>
    >>> The library bug, if any, is that you can't apply
    >>>
    >>> unicode(s, errors='replace')
    >>>
    >>> to a Unicode string. TypeError("Decoding unicode is not supported") is
    >>> raised. However
    >>>
    >>> unicode(s)
    >>>
    >>> will accept Unicode input.

    >>
    >> I think that's a Python bug. If the latter succeeds as a no-op, the
    >> former should also succeed as a no-op. Neither should ever get any
    >> errors when ‘s’ is a ‘unicode’ object already.


    > No. The semantics of the unicode function (technically: a type
    > constructor) are well-defined, and there are two distinct behaviours:
    >
    > unicode(obj)
    >
    > is analogous to str(obj), and it attempts to convert obj to a unicode
    > string by calling obj.__unicode__, if it exists, or __str__ if it
    > doesn't. No encoding or decoding is attempted in the event that obj is a
    > unicode instance.
    >
    > unicode(obj, encoding, errors)
    >
    > is explicitly stated in the docs as decoding obj if EITHER of encoding or
    > errors is given, AND that obj must be either an 8-bit string (bytes) ora
    > buffer object.
    >
    > It is true that u''.decode() will succeed, in Python 2, but the fact that
    > unicode objects have a decode method at all is IMO a bug. It has also


    I believe that is because in Py 2, codecs and .encode/.decode were used
    for same type recoding like base64, uu coding. That was simplified in
    Py3 so that 'decoding' is bytes to string and 'encoding' is string to
    bytes, and base64, etc, are only done in their separate modules and not
    also duplicated in the codecs machinery.

    > been corrected in Python 3, where (unicode) str objects no longer have a
    > decode method, and bytes objects no longer have an encode method.
    >
    >
    >>> The Python documentation
    >>> ("http://docs.python.org/library/functions.html#unicode") does not
    >>> mention this.

    >
    > Yes it does. It is is the SECOND sentence, immediately after the summary
    > line:
    >
    > unicode([object[, encoding[, errors]]])
    > Return the Unicode string version of object using one of the
    > following modes:
    >
    > If encoding and/or errors are given, unicode() will decode the object
    > which can either be an 8-bit string or a character buffer using the
    > codec for encoding. ...
    >
    >
    > Admittedly, it doesn't *explicitly* state that TypeError will be raised,
    > but what other exception kind would you expect when you supply an
    > argument of the wrong type?


    What you have correctly pointed out is that there is no discrepancy
    between doc and behavior and hence no bug for the purpose of the
    tracker. Thanks.

    >>> It is therefore necessary to check the type before
    >>> calling "unicode", or catch the undocumented TypeError exception
    >>> afterward.

    >>
    >> Yes, this check should not be necessary; calling the ‘unicode’
    >> constructor with an object that's already an instance of ‘unicode’
    >> should just return the object as-is, IMO. It shouldn't matter that
    >> you've specified how decoding errors are to be handled, because in that
    >> case no decoding happens anyway.

    >
    > I don't believe that it is the job of unicode() to Do What I Mean, but
    > only to Do What I Say. If I *explicitly* tell unicode() to decode the
    > argument (by specifying either the codec or the error handler or both)
    > then it should not double-guess me and ignore the extra parameters.
    >
    > End-user applications may, with care, try to be smart and DWIM, but
    > library functions should be dumb and should do what they are told.


    --
    Terry Jan Reedy
    Terry Reedy, Mar 8, 2012
    #6
  7. John Nagle

    John Nagle Guest

    On 3/7/2012 6:18 PM, Ben Finney wrote:
    > Steven D'Aprano<> writes:
    >
    >> On Thu, 08 Mar 2012 08:48:58 +1100, Ben Finney wrote:
    >>> I think that's a Python bug. If the latter succeeds as a no-op, the
    >>> former should also succeed as a no-op. Neither should ever get any
    >>> errors when ‘s’ is a ‘unicode’ object already.

    >>
    >> No. The semantics of the unicode function (technically: a type
    >> constructor) are well-defined, and there are two distinct behaviours:


    Right. The real problem is that Python 2.7 doesn't have distinct
    "str" and "bytes" types. type(bytes() returns <type 'str'>
    "str" is assumed to be ASCII 0..127, but that's not enforced.
    "bytes" and "str" should have been distinct types, but
    that would have broken much old code. If they were distinct, then
    constructors could distinguish between string type conversion
    (which requires no encoding information) and byte stream decoding.

    So it's possible to get junk characters in a "str", and they
    won't convert to Unicode. I've had this happen with databases which
    were supposed to be ASCII, but occasionally a non-ASCII character
    would slip through.

    This is all different in Python 3.x, where "str" is Unicode and
    "bytes" really are a distinct type.

    John Nagle
    John Nagle, Mar 8, 2012
    #7
  8. John Nagle

    Neil Cerutti Guest

    On 2012-03-08, John Nagle <> wrote:
    > So it's possible to get junk characters in a "str", and they
    > won't convert to Unicode. I've had this happen with databases
    > which were supposed to be ASCII, but occasionally a non-ASCII
    > character would slip through.


    Perhaps encode and then decode, rather than try to encode a
    non-encoded str.

    --
    Neil Cerutti
    Neil Cerutti, Mar 9, 2012
    #8
  9. John Nagle

    John Nagle Guest

    On 3/8/2012 2:58 PM, Prasad, Ramit wrote:
    >> Right. The real problem is that Python 2.7 doesn't have distinct
    >> "str" and "bytes" types. type(bytes() returns<type 'str'>
    >> "str" is assumed to be ASCII 0..127, but that's not enforced.
    >> "bytes" and "str" should have been distinct types, but
    >> that would have broken much old code. If they were distinct, then
    >> constructors could distinguish between string type conversion
    >> (which requires no encoding information) and byte stream decoding.
    >>
    >> So it's possible to get junk characters in a "str", and they
    >> won't convert to Unicode. I've had this happen with databases which
    >> were supposed to be ASCII, but occasionally a non-ASCII character
    >> would slip through.

    >
    > bytes and str are just aliases for each other.


    That's true in Python 2.7, but not in 3.x. From 2.6 forward,
    "bytes" and "str" were slowly being separated. See PEP 358.
    Some of the problems in Python 2.7 come from this ambiguity.
    Logically, "unicode" of "str" should be a simple type conversion
    from ASCII to Unicode, while "unicode" of "bytes" should
    require an encoding. But because of the bytes/str ambiguity
    in Python 2.6/2.7, the behavior couldn't be type-based.

    John Nagle
    John Nagle, Mar 9, 2012
    #9
  10. On Fri, 09 Mar 2012 10:11:58 -0800, John Nagle wrote:

    > On 3/8/2012 2:58 PM, Prasad, Ramit wrote:
    >>> Right. The real problem is that Python 2.7 doesn't have distinct
    >>> "str" and "bytes" types. type(bytes() returns<type 'str'> "str" is
    >>> assumed to be ASCII 0..127, but that's not enforced. "bytes" and "str"
    >>> should have been distinct types, but that would have broken much old
    >>> code. If they were distinct, then constructors could distinguish
    >>> between string type conversion (which requires no encoding
    >>> information) and byte stream decoding.
    >>>
    >>> So it's possible to get junk characters in a "str", and they
    >>> won't convert to Unicode. I've had this happen with databases which
    >>> were supposed to be ASCII, but occasionally a non-ASCII character
    >>> would slip through.

    >>
    >> bytes and str are just aliases for each other.

    >
    > That's true in Python 2.7, but not in 3.x. From 2.6 forward,
    > "bytes" and "str" were slowly being separated. See PEP 358. Some of the
    > problems in Python 2.7 come from this ambiguity. Logically, "unicode" of
    > "str" should be a simple type conversion from ASCII to Unicode, while
    > "unicode" of "bytes" should require an encoding. But because of the
    > bytes/str ambiguity in Python 2.6/2.7, the behavior couldn't be
    > type-based.


    This demonstrates a gross confusion about both Unicode and Python. John,
    I honestly don't mean to be rude here, but if you actually believe that
    (rather than merely expressing yourself poorly), then it seems to me that
    you are desperately misinformed about Unicode and are working on the
    basis of some serious misapprehensions about the nature of strings.

    I recommend you start with this:

    http://www.joelonsoftware.com/articles/Unicode.html


    In Python 2.6/2.7, there is no ambiguity between str/bytes. The two names
    are aliases for each other. The older name, "str", is a misnomer, since
    it *actually* refers to bytes (and always has, all the way back to the
    earliest days of Python). At best, it could be read as "byte string" or
    "8-bit string", but the emphasis should always be on the *bytes*.

    str is NOT "assumed to be ASCII 0..127", and it never has been. Python's
    str prior to version 3.0 has *always* been bytes, it just never used that
    name. For example, in Python 2.4, help(chr) explicitly supports
    characters with ordinal 0...255:

    Help on built-in function chr in module __builtin__:

    chr(...)
    chr(i) -> character

    Return a string of one character with ordinal i; 0 <= i < 256.


    I can go all the way back to Python 0.9, which was so primitive it didn't
    even accept "" as string delimiters, and the str type was still based on
    bytes, with explicit support for non-ASCII values:

    steve@runes:~/Downloads/python-0.9.1$ ./python0.9.1
    >>> print 'This is *not* ASCII \xCA see the non-ASCII byte.'

    This is *not* ASCII � see the non-ASCII byte.


    Any conversion from bytes (including Python 2 strings) to Unicode is
    ALWAYS a decoding operation. It can't possibly be anything else. If you
    think that it can be, you don't understand the relationship between
    strings, Unicode and bytes.


    --
    Steven
    Steven D'Aprano, Mar 10, 2012
    #10
  11. John Nagle

    John Nagle Guest

    On 3/9/2012 4:57 PM, Steven D'Aprano wrote:
    > On Fri, 09 Mar 2012 10:11:58 -0800, John Nagle wrote:
    > This demonstrates a gross confusion about both Unicode and Python. John,
    > I honestly don't mean to be rude here, but if you actually believe that
    > (rather than merely expressing yourself poorly), then it seems to me that
    > you are desperately misinformed about Unicode and are working on the
    > basis of some serious misapprehensions about the nature of strings.
    >
    > In Python 2.6/2.7, there is no ambiguity between str/bytes. The two names
    > are aliases for each other. The older name, "str", is a misnomer, since
    > it *actually* refers to bytes (and always has, all the way back to the
    > earliest days of Python). At best, it could be read as "byte string" or
    > "8-bit string", but the emphasis should always be on the *bytes*.


    There's an inherent ambiguity in that "bytes" and "str" are really
    the same type in Python 2.6/2.7. That's a hack for backwards
    compatibility, and it goes away in 3.x. The notes for PEP 358
    admit this.

    It's implicit in allowing

    unicode(s)

    with no encoding, on type "str", that there is an implicit
    assumption that s is ASCII. Arguably, "unicode()" should
    have required an encoding in all cases.

    Or "str" and "bytes" should have been made separate types in
    Python 2.7, in which case unicode() of a str would be a safe
    ASCII to Unicode translation, and unicode() of a bytes object
    would require an encoding. But that would break too much old code.
    So we have an ambiguity and a hack.

    "While Python 2 also has a unicode string type, the fundamental
    ambiguity of the core string type, coupled with Python 2's default
    behavior of supporting automatic coercion from 8-bit strings to unicode
    objects when the two are combined, often leads to UnicodeErrors"
    - PEP 404

    John Nagle
    John Nagle, Mar 11, 2012
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Diogenes Dilone

    Funky error Situation

    Diogenes Dilone, Jan 6, 2004, in forum: ASP .Net
    Replies:
    3
    Views:
    526
    Diogenes Dilone
    Jan 7, 2004
  2. Hai Nguyen

    Have you seen this situation?

    Hai Nguyen, Feb 4, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    315
    Hermit Dave
    Feb 4, 2004
  3. .Net Newbie
    Replies:
    0
    Views:
    467
    .Net Newbie
    Jul 29, 2004
  4. VB Programmer
    Replies:
    3
    Views:
    2,850
    cbDevelopment
    Oct 17, 2004
  5. =?Utf-8?B?Q2hyaXM=?=

    situation with need to update dropdown on another page

    =?Utf-8?B?Q2hyaXM=?=, Feb 15, 2005, in forum: ASP .Net
    Replies:
    5
    Views:
    508
    =?Utf-8?B?Q2hyaXM=?=
    Mar 2, 2005
Loading...

Share This Page