unicode and strings

[Jacob Friis]

I'm trying to learn Python via Marks Feedparser.

<snip src="http://feedparser.org/docs/character-encoding.html">
If the character encoding can not be determined, Universal Feed Parser
sets the bozo bit to 1 and sets bozo_exception to
feedparser.CharacterEncodingUnknown. In this case, parsed values will be
strings, not Unicode strings.
</snip>

I guess this means that all data will be unicode, and to put in a
database I could use my mycode function. Correct?

def mycode(value):
if isinstance(value, unicode):
value = value.encode('utf-8')
return value

What do I do about data that is a string?

Thanks,
Jacob

 

[Diez B. Roggisch]

I'm trying to learn Python via Marks Feedparser.

<snip src="http://feedparser.org/docs/character-encoding.html">
If the character encoding can not be determined, Universal Feed Parser
sets the bozo bit to 1 and sets bozo_exception to
feedparser.CharacterEncodingUnknown. In this case, parsed values will be
strings, not Unicode strings.
</snip>

I guess this means that all data will be unicode, and to put in a
database I could use my mycode function. Correct?

No. It means that you don't get unicode objects, but strings which are
basically sequences of bytes. And there is no way to be sure what encoding
they are in.

def mycode(value):
if isinstance(value, unicode):
value = value.encode('utf-8')
return value

this will either yield a string in utf8-encoding, or a string in an unknown
encoding.