Is it possible to consume UTF8 XML documents using xml.dom.pulldom?

[Simon Willison]

I'm having a horrible time trying to get xml.dom.pulldom to consume a
UTF8 encoded XML file. Here's what I've tried so far:
....
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in
position 21: ordinal not in range(128)

xml.dom.minidom can handle the string just fine:
u'<?xml version="1.0" ?><msg>Simon\u2019s XML nightmare</msg>'

If I pass a unicode string to pulldom instead of a utf8 encoded
bytestring it still breaks:
....
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/
xml/dom/pulldom.py in parseString(string, parser)
346
347 bufsize = len(string)
--> 348 buf = StringIO(string)
349 if not parser:
350 parser = xml.sax.make_parser()
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in
position 32: ordinal not in range(128)

Is it possible to consume utf8 or unicode using xml.dom.pulldom or
should I try something else?

Thanks,

Simon Willison

 

[Simon Willison]

Follow up question: what's the best way of incrementally consuming XML
in Python that's character encoding aware? I have a very large file to
consume but I'd rather not have to fall back to the raw SAX API.

 

[Paul Boddie]

I'm having a horrible time trying to get xml.dom.pulldom to consume a
UTF8 encoded XML file. Here's what I've tried so far:


('START_DOCUMENT', <xml.dom.minidom.Document instance at 0x6f06c0>)>>> parser.next()

('START_ELEMENT', <DOM Element: msg at 0x6f0710>)>>> parser.next()

...
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in
position 21: ordinal not in range(128)

I can't reproduce this on Python 2.3.6 or 2.4.4 on RHEL 4. Instead, I
get the usual...

('CHARACTERS', <DOM Text node "Simon\u2019s XM...">)

And I can get the content of the text node as a proper Unicode object.

[...]

Is it possible to consume utf8 or unicode using xml.dom.pulldom or
should I try something else?

Yes, it is possible, at least in Python 2.3.6 and 2.4.4 configured
with --enable-unicode=ucs4 (which is what Red Hat does and expects).

Paul

P.S. You shouldn't try and pass Unicode to the parser, since XML
parsing in its entirety deals with byte sequences and character
encodings, although I suppose that there's some kind of character-
based (ie. Unicode value-based) parsing method defined somewhere by
some committee or other.

 

[Simon Willison]

I can't reproduce this on Python 2.3.6 or 2.4.4 on RHEL 4. Instead, I
get the usual...

('CHARACTERS', <DOM Text node "Simon\u2019s XM...">)

I'm using Python 2.5.1 on OS X Leopard:

$ python
Python 2.5.1 (r251:54863, Feb 4 2008, 21:48:13)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin

I just tried it out on Python 2.4.2 on an Ubuntu machine and it worked
fine! I guess this must be an OS X Python bug. How absolutely
infuriating.

Thanks,

Simon

 

[Simon Willison]

I just tried it out on Python 2.4.2 on an Ubuntu machine and it worked
fine! I guess this must be an OS X Python bug. How absolutely
infuriating.

Some very useful people in #python on Freenode pointed out that my bug
occurs because I'm trying to display things interactively in the
console. Saving to a variable instead fixes the problem.

Thanks for your help,

Simon

 

[Paul Boddie]

Some very useful people in #python on Freenode pointed out that my bug
occurs because I'm trying to display things interactively in the
console. Saving to a variable instead fixes the problem.

What's strange about that is how the object is represented when
displayed:

('CHARACTERS', <DOM Text node "Simon\u2019s XM...">)

Here, there's no attempt made to encode \u2019 as an ASCII byte
sequence. Does the OS X version of Python do anything special with
string representations?

Paul

 

[Peter Otten]

What's strange about that is how the object is represented when
displayed:

('CHARACTERS', <DOM Text node "Simon\u2019s XM...">)

Here, there's no attempt made to encode \u2019 as an ASCII byte
sequence. Does the OS X version of Python do anything special with
string representations?

I'm on Kubuntu 7.10 and see the same error as Simon. The problem is in the
minidom.CharacterData class which has the following method

def __repr__(self):
data = self.data
if len(data) > 10:
dotdotdot = "..."
else:
dotdotdot = ""
return "<DOM %s node \"%s%s\">" % (
self.__class__.__name__, data[0:10], dotdotdot)

The data attribute is a unicode instance...

Peter

 

[Stefan Behnel]

Follow up question: what's the best way of incrementally consuming XML
in Python that's character encoding aware?

iterparse(), as implemented in (c)ElementTree and lxml. Note that ElementTree
and cElementTree are part of Python 2.5, in the xml.etree package.

I have a very large file to
consume but I'd rather not have to fall back to the raw SAX API.

Large is fairly relative. Both cElementTree and lxml are pretty memory
friendly, even when parsing into an in-memory tree.

Stefan

 

[Paul Boddie]

I'm on Kubuntu 7.10 and see the same error as Simon. The problem is in the
minidom.CharacterData class which has the following method

    def __repr__(self):
        data = self.data
        if len(data) > 10:
            dotdotdot = "..."
        else:
            dotdotdot = ""
        return "<DOM %s node \"%s%s\">" % (
            self.__class__.__name__, data[0:10], dotdotdot)

The data attribute is a unicode instance...

Who wants to be first to submit a patch? ;-)

Paul

 

[Peter Otten]

Who wants to be first to submit a patch? ;-)

And where? The sourceforge page says

"PyXML is no longer maintained."

Peter

 

[Paul Boddie]

And where? The sourceforge page says

"PyXML is no longer maintained."

The minidom code is in the standard library:

http://svn.python.org/view/python/trunk/Lib/xml/dom/

Paul