"Decoding unicode is not supported" in unusual situation

[John Nagle]

I'm getting

line 79, in tounicode
return(unicode(s, errors='replace'))
TypeError: decoding Unicode is not supported

from this, under Python 2.7:

def tounicode(s) :
if type(s) == unicode :
return(s)
return(unicode(s, errors='replace'))

That would seem to be impossible. But it's not.
"s" is generated from the "suds" SOAP client. The documentation
for "suds" says:

"Suds leverages python meta programming to provide an intuative API for
consuming web services. Runtime objectification of types defined in the
WSDL is provided without class generation."

I think that somewhere in "suds", they subclass the "unicode" type.
That's almost too cute.

The proper test is

isinstance(s,unicode)


John Nagle

 

[Diez B. Roggisch]

I think that somewhere in "suds", they subclass the "unicode" type.
That's almost too cute.

The proper test is

isinstance(s,unicode)

Woot, you finally discovered polymorphism - congratulations!

Diez

 

[Steven D'Aprano]

If by “discovered†you mean “brokeâ€.

John, polymorphism entails that it *doesn't matter* whether the object
inherits from any particular type; it only matters whether the object
behaves correctly.

So rather than testing whether the object inherits from ‘unicode’, test
whether it behaves how you expect – preferably by just using it as
though it does behave that way.

I must admit that I can't quite understand John Nagle's original post, so
I could be wrong, but I *think* that both you and Diez have misunderstood
the nature of John's complaint.

I *think* he is complaining that some other library -- suds? -- has a
broken test for Unicode, by using:

if type(s) is unicode: ...

instead of

if isinstance(s, unicode): ...

Consequently, when the library passes a unicode *subclass* to the
tounicode function, the "type() is unicode" test fails. That's a bad bug.

It's arguable that the library shouldn't even use isinstance, but that's
an argument for another day.

 

[John Nagle]

I *think* he is complaining that some other library -- suds? -- has a
broken test for Unicode, by using:

if type(s) is unicode: ...

instead of

if isinstance(s, unicode): ...

Consequently, when the library passes a unicode *subclass* to the
tounicode function, the "type() is unicode" test fails. That's a bad bug.

No, that was my bug.

The library bug, if any, is that you can't apply

unicode(s, errors='replace')

to a Unicode string. TypeError("Decoding unicode is not supported") is
raised. However

unicode(s)

will accept Unicode input.

The Python documentation
("http://docs.python.org/library/functions.html#unicode") does not
mention this. It is therefore necessary to check the type before
calling "unicode", or catch the undocumented TypeError exception
afterward.


John Nagle

 

[Steven D'Aprano]

I think that's a Python bug. If the latter succeeds as a no-op, the
former should also succeed as a no-op. Neither should ever get any
errors when ‘s’ is a ‘unicode’ object already.

No. The semantics of the unicode function (technically: a type
constructor) are well-defined, and there are two distinct behaviours:

unicode(obj)

is analogous to str(obj), and it attempts to convert obj to a unicode
string by calling obj.__unicode__, if it exists, or __str__ if it
doesn't. No encoding or decoding is attempted in the event that obj is a
unicode instance.

unicode(obj, encoding, errors)

is explicitly stated in the docs as decoding obj if EITHER of encoding or
errors is given, AND that obj must be either an 8-bit string (bytes) or a
buffer object.

It is true that u''.decode() will succeed, in Python 2, but the fact that
unicode objects have a decode method at all is IMO a bug. It has also
been corrected in Python 3, where (unicode) str objects no longer have a
decode method, and bytes objects no longer have an encode method.


Yes it does. It is is the SECOND sentence, immediately after the summary
line:

unicode([object[, encoding[, errors]]])
Return the Unicode string version of object using one of the
following modes:

If encoding and/or errors are given, unicode() will decode the object
which can either be an 8-bit string or a character buffer using the
codec for encoding. ...


Admittedly, it doesn't *explicitly* state that TypeError will be raised,
but what other exception kind would you expect when you supply an
argument of the wrong type?

Yes, this check should not be necessary; calling the ‘unicode’
constructor with an object that's already an instance of ‘unicode’
should just return the object as-is, IMO. It shouldn't matter that
you've specified how decoding errors are to be handled, because in that
case no decoding happens anyway.

I don't believe that it is the job of unicode() to Do What I Mean, but
only to Do What I Say. If I *explicitly* tell unicode() to decode the
argument (by specifying either the codec or the error handler or both)
then it should not double-guess me and ignore the extra parameters.

End-user applications may, with care, try to be smart and DWIM, but
library functions should be dumb and should do what they are told.

 

[Terry Reedy]

No. The semantics of the unicode function (technically: a type
constructor) are well-defined, and there are two distinct behaviours:

unicode(obj)

is analogous to str(obj), and it attempts to convert obj to a unicode
string by calling obj.__unicode__, if it exists, or __str__ if it
doesn't. No encoding or decoding is attempted in the event that obj is a
unicode instance.

unicode(obj, encoding, errors)

is explicitly stated in the docs as decoding obj if EITHER of encoding or
errors is given, AND that obj must be either an 8-bit string (bytes) ora
buffer object.

It is true that u''.decode() will succeed, in Python 2, but the fact that
unicode objects have a decode method at all is IMO a bug. It has also

I believe that is because in Py 2, codecs and .encode/.decode were used
for same type recoding like base64, uu coding. That was simplified in
Py3 so that 'decoding' is bytes to string and 'encoding' is string to
bytes, and base64, etc, are only done in their separate modules and not
also duplicated in the codecs machinery.

been corrected in Python 3, where (unicode) str objects no longer have a
decode method, and bytes objects no longer have an encode method.


Yes it does. It is is the SECOND sentence, immediately after the summary
line:

unicode([object[, encoding[, errors]]])
Return the Unicode string version of object using one of the
following modes:

If encoding and/or errors are given, unicode() will decode the object
which can either be an 8-bit string or a character buffer using the
codec for encoding. ...


Admittedly, it doesn't *explicitly* state that TypeError will be raised,
but what other exception kind would you expect when you supply an
argument of the wrong type?

What you have correctly pointed out is that there is no discrepancy
between doc and behavior and hence no bug for the purpose of the
tracker. Thanks.

 

[John Nagle]

Right. The real problem is that Python 2.7 doesn't have distinct
"str" and "bytes" types. type(bytes() returns <type 'str'>
"str" is assumed to be ASCII 0..127, but that's not enforced.
"bytes" and "str" should have been distinct types, but
that would have broken much old code. If they were distinct, then
constructors could distinguish between string type conversion
(which requires no encoding information) and byte stream decoding.

So it's possible to get junk characters in a "str", and they
won't convert to Unicode. I've had this happen with databases which
were supposed to be ASCII, but occasionally a non-ASCII character
would slip through.

This is all different in Python 3.x, where "str" is Unicode and
"bytes" really are a distinct type.

John Nagle

 

[Neil Cerutti]

So it's possible to get junk characters in a "str", and they
won't convert to Unicode. I've had this happen with databases
which were supposed to be ASCII, but occasionally a non-ASCII
character would slip through.

Perhaps encode and then decode, rather than try to encode a
non-encoded str.

 

[John Nagle]

bytes and str are just aliases for each other.

That's true in Python 2.7, but not in 3.x. From 2.6 forward,
"bytes" and "str" were slowly being separated. See PEP 358.
Some of the problems in Python 2.7 come from this ambiguity.
Logically, "unicode" of "str" should be a simple type conversion
from ASCII to Unicode, while "unicode" of "bytes" should
require an encoding. But because of the bytes/str ambiguity
in Python 2.6/2.7, the behavior couldn't be type-based.

John Nagle

 

[Steven D'Aprano]

That's true in Python 2.7, but not in 3.x. From 2.6 forward,
"bytes" and "str" were slowly being separated. See PEP 358. Some of the
problems in Python 2.7 come from this ambiguity. Logically, "unicode" of
"str" should be a simple type conversion from ASCII to Unicode, while
"unicode" of "bytes" should require an encoding. But because of the
bytes/str ambiguity in Python 2.6/2.7, the behavior couldn't be
type-based.

This demonstrates a gross confusion about both Unicode and Python. John,
I honestly don't mean to be rude here, but if you actually believe that
(rather than merely expressing yourself poorly), then it seems to me that
you are desperately misinformed about Unicode and are working on the
basis of some serious misapprehensions about the nature of strings.

I recommend you start with this:

http://www.joelonsoftware.com/articles/Unicode.html


In Python 2.6/2.7, there is no ambiguity between str/bytes. The two names
are aliases for each other. The older name, "str", is a misnomer, since
it *actually* refers to bytes (and always has, all the way back to the
earliest days of Python). At best, it could be read as "byte string" or
"8-bit string", but the emphasis should always be on the *bytes*.

str is NOT "assumed to be ASCII 0..127", and it never has been. Python's
str prior to version 3.0 has *always* been bytes, it just never used that
name. For example, in Python 2.4, help(chr) explicitly supports
characters with ordinal 0...255:

Help on built-in function chr in module __builtin__:

chr(...)
chr(i) -> character

Return a string of one character with ordinal i; 0 <= i < 256.


I can go all the way back to Python 0.9, which was so primitive it didn't
even accept "" as string delimiters, and the str type was still based on
bytes, with explicit support for non-ASCII values:

steve@runes:~/Downloads/python-0.9.1$ ./python0.9.1This is *not* ASCII � see the non-ASCII byte.


Any conversion from bytes (including Python 2 strings) to Unicode is
ALWAYS a decoding operation. It can't possibly be anything else. If you
think that it can be, you don't understand the relationship between
strings, Unicode and bytes.

 

[John Nagle]

On Fri, 09 Mar 2012 10:11:58 -0800, John Nagle wrote:
This demonstrates a gross confusion about both Unicode and Python. John,
I honestly don't mean to be rude here, but if you actually believe that
(rather than merely expressing yourself poorly), then it seems to me that
you are desperately misinformed about Unicode and are working on the
basis of some serious misapprehensions about the nature of strings.

In Python 2.6/2.7, there is no ambiguity between str/bytes. The two names
are aliases for each other. The older name, "str", is a misnomer, since
it *actually* refers to bytes (and always has, all the way back to the
earliest days of Python). At best, it could be read as "byte string" or
"8-bit string", but the emphasis should always be on the *bytes*.

There's an inherent ambiguity in that "bytes" and "str" are really
the same type in Python 2.6/2.7. That's a hack for backwards
compatibility, and it goes away in 3.x. The notes for PEP 358
admit this.

It's implicit in allowing

unicode(s)

with no encoding, on type "str", that there is an implicit
assumption that s is ASCII. Arguably, "unicode()" should
have required an encoding in all cases.

Or "str" and "bytes" should have been made separate types in
Python 2.7, in which case unicode() of a str would be a safe
ASCII to Unicode translation, and unicode() of a bytes object
would require an encoding. But that would break too much old code.
So we have an ambiguity and a hack.

"While Python 2 also has a unicode string type, the fundamental
ambiguity of the core string type, coupled with Python 2's default
behavior of supporting automatic coercion from 8-bit strings to unicode
objects when the two are combined, often leads to UnicodeErrors"
- PEP 404

John Nagle