Problem with minidom and special chars in HTML

Horst Gutmann · Feb 22, 2005

Hi

I currently have quite a big problem with minidom and special chars (for
example &uuml

in HTML.

Let's say I have following input file:
--------------------------------------------------
<?xml version="1.0"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
<body>
ü
</body>
</html>
--------------------------------------------------

And following python script:
--------------------------------------------------
from xml.dom import minidom
if __name__ == '__main__':
doc = minidom.parse('test2.html')
f = open('test3.html','w+')
f.write(doc.toxml())
f.close()
--------------------------------------------------

test3.html only has a blank line where should be the ü It is simply
removed.

Any idea how I could solve this problem?

MfG, Horst

Fredrik Lundh · Feb 22, 2005

Horst said:
I currently have quite a big problem with minidom and special chars (for example &uuml in HTML.

Let's say I have following input file:
--------------------------------------------------
<?xml version="1.0"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
<body>
ü
</body>
</html>
--------------------------------------------------

> test3.html only has a blank line where should be the ü It is simply
removed.

Any idea how I could solve this problem?

umm. doesn't that doctype point to an SGML DTD? even if minidom did fetch
external DTD's (I don't think it does), it would probably choke on that DTD.

running your documents through "tidy -asxml -numeric" before parsing them as
XML might be a good idea...

http://tidy.sourceforge.net/ (command-line binaries, library)
http://utidylib.berlios.de/ (python bindings)

</F>

Horst Gutmann · Feb 22, 2005

Fredrik said:
umm. doesn't that doctype point to an SGML DTD? even if minidom did fetch
external DTD's (I don't think it does), it would probably choke on that DTD.

running your documents through "tidy -asxml -numeric" before parsing them as
XML might be a good idea...

http://tidy.sourceforge.net/ (command-line binaries, library)
http://utidylib.berlios.de/ (python bindings)

</F>

Thanks, but the problem is, that I can't use the numeric representations
of these special chars. I will probably simply play find&replace before
feeding the document into minidom and change the output back afterwards

MfG, Horst

Jarek Zgoda · Feb 22, 2005

Horst Gutmann napisa³(a):

I currently have quite a big problem with minidom and special chars (for
example &uuml in HTML.

Let's say I have following input file:

HTML4 is not an XML application. Even if minidom will fetch this DTD and
be able to parse character entities, it may not be able to parse the
document.

Any idea how I could solve this problem?

Don't use minidom or convert HTML4 to XHTML and change declaration of
doctype.

Horst Gutmann · Feb 23, 2005

Jarek said:
Horst Gutmann napisa³(a):

HTML4 is not an XML application. Even if minidom will fetch this DTD and
be able to parse character entities, it may not be able to parse the
document.

Don't use minidom or convert HTML4 to XHTML and change declaration of
doctype.

This was just a bad example

I get the same problem with XHTML in the
doctype. The funny thing here IMO is, that the special chars are simply
removed. No warning, no nothing :-(

MfG, Horst

Jarek Zgoda · Feb 23, 2005

Horst Gutmann napisa³(a):

This was just a bad example I get the same problem with XHTML in the
doctype. The funny thing here IMO is, that the special chars are simply
removed. No warning, no nothing :-(

As Fredrik pointed out, it's minidom that cann't fetch DTD from remote
location. Download this DTD file to your local machine (it lies at
exactly this URI), try changing PUBLIC identifier to SYSTEM and give
local path to this file.

and-google · Feb 24, 2005

Horst said:
I currently have quite a big problem with minidom and special chars
(for example &uuml in HTML.

Yes. Ignoring the issue of the wrong doctype, minidom is a pure XML
parser and knows nothing of XHTML and its doctype's entities 'uuml' and
the like. Only the built-in entities (& etc.) will work.

Unfortunately the parser minidom uses won't read external entities -
including the external subset of the DTD (which is where all the stuff
about what "ü" means is stored). And because minidom does not
support EntityReference nodes, the information that there was an entity
reference there at all gets thrown away as it is replaced with the
empty string. Which is kind of bad.

Possible workarounds:

1. pass minidom a different parser to use, one which supports external
entities and which will parse all the DTD stuff. I don't know if there
is anything suitable available, though...

2. use a DOM implementation with the option to support external
entities. For example, with pxdom, one can use DOM Level 3 LS methods,
or pxdom.parse(f, {'pxdom-external-entities': True}).

However note that reading and parsing an external entity will introduce
significant slowdown, especially in the case of the rather complex
multi-file XHTML DTD. Other possibilities:

3. hack the content on the way into the parser to replace the DOCTYPE
declaration with one including entity definitions in the internal
subset:

<!DOCTYPE html PUBLIC "..." "..." [
<!ENTITY uuml "ü">
...
]>
<html>...

4. hack the content on the way into the parser to replace entity
references with character references, eg. ü -> ü. This is
'safe' for simple documents without an internal subset; charrefs and
entrefs can be used in the same places with the same meaning, except
for some issues in the internal subset.

5. use a DOM implementation that supports EntityReference nodes, such
as pxdom. Entity references with no replacement text (or all entity
references if the DOM Level 3 LS parameter 'entities' is set) will
exist as EntityReference DOM objects instead of being flattened to
text. They can safely be reserialized as ü without the
implementation having to know what text they represent.

Entities are a big source of complication and confusion, which I wish
had not made it into XML!

display VARCHAR(mysql) and special chars in html	6	Feb 21, 2005
XML Javascript Special Chars encoding	0	Feb 20, 2006
Help with my responsive home page	2	Dec 14, 2022
Unable to make ironpython run in browser with silverlight	4	May 25, 2011
ASP converts Unicode Chars to HTML entities?	2	Sep 5, 2005
Problem applying HTML 4.01 DOM in scripting	27	Mar 18, 2006
need help with simple html problem	9	Nov 26, 2008
Response.Write Problem	2	Jan 22, 2009

Problem with minidom and special chars in HTML

Horst Gutmann

Fredrik Lundh

Horst Gutmann

Jarek Zgoda

Horst Gutmann

Jarek Zgoda

and-google

Ask a Question

Similar Threads

Members online

Forum statistics

Latest Threads