XML Tree Discovery (script, tool, __?)

[eric.pederson]

Hi all,

Finally diving into XML programmatically. Does anyone have a best
practice recommendation for programmatically discovering the structure
of an arbitrary XML document via Python?

It seems like it is a common wheel I'd be re-inventing.

Thanks and cheers


EP

 

[Erik Max Francis]

Finally diving into XML programmatically. Does anyone have a best
practice recommendation for programmatically discovering the structure
of an arbitrary XML document via Python?

It seems like it is a common wheel I'd be re-inventing.

It is. Look up XML DOM.

 

[uche.ogbuji]

"inally diving into XML programmatically. Does anyone have a best
practice recommendation for programmatically discovering the structure
of an arbitrary XML document via Python?"

You can do this with DOM or SAX, or any of the many more friendly XML
processing libraries out there. You might want to be more specific.
What sort of output do you want from this discovery?

 

[eric.pederson]

The output I was contemplating was a DOM "DNA" - that is the DOM
without the instances of the elements or their data, a bare tree, a
prototype tree based on what is in the document (rather than what is
legal to include in the document).

Just enough data that for an arbitrary element I would know:

1) whether the element was in a document
2) where to find it (the chain of parents)


As I mentioned, I'm just starting to think about the subject, so maybe
the best practice is something else like loading the full DOM into
memory. I'm still in the make one to throw away one mode.

EP

 

[Fredrik Lundh]

The output I was contemplating was a DOM "DNA" - that is the DOM
without the instances of the elements or their data, a bare tree, a
prototype tree based on what is in the document (rather than what is
legal to include in the document).

Just enough data that for an arbitrary element I would know:

1) whether the element was in a document
2) where to find it (the chain of parents)

how to you identify the elements ? do they have unique tags, unique identifiers,
or some other "globally unique" identifier (attributes or contents) ?

</F>

 

[eric.pederson]

just namespace + tag

 

[Fredrik Lundh]

just namespace + tag

here's an ElementTree-based example:

# http://effbot.org/tag/elementtree
import elementtree.ElementTree as ET

FILE = "example.xml"

path = ()
path_map = {}

for event, elem in ET.iterparse(FILE, events=("start", "end")):
if event == "start":
path = path + (elem.tag,)
else:
path_map.setdefault(path, []).append(elem)
path = path[:-1]
elem.clear() # won't need the contents any more

for path in path_map:
print "/".join(path), len(path_map[path])

given this document:

<document>
<chapter>
<title>chapter 1</title>
</chapter>
<chapter>
<title>chapter 2</title>
</chapter>
</document>

the above script prints

document 1
document/chapter 2
document/chapter/title 2

the script uses universal names for tags that live in a namespace. for
information on how to decipher such names, see:

http://www.effbot.org/zone/element.htm#xml-namespaces

hope this helps!

</F>

 

[uche.ogbuji]

"""
The output I was contemplating was a DOM "DNA" - that is the DOM
without the instances of the elements or their data, a bare tree, a
prototype tree based on what is in the document (rather than what is
legal to include in the document).

Just enough data that for an arbitrary element I would know:

1) whether the element was in a document
2) where to find it (the chain of parents)
"""

This is easy to do in SAX. For some hints, see page 2 of my article:

http://www.xml.com/pub/a/2004/11/24/py-xml.html

 

[Istvan Albert]

All I can add to this is:

- don't use SAX unless your document is huge
- don't use DOM unless someone is putting a gun to your head

There's a good selection of nice and simple XML processing libraries in
python. You could start with ElementTree.

 

[Kent Johnson]

All I can add to this is:

- don't use SAX unless your document is huge
- don't use DOM unless someone is putting a gun to your head

+1 QOTW

 

[uche.ogbuji]

- don't use SAX unless your document is huge

- don't use DOM unless someone is putting a gun to your head

What I say is: use what works for you. I think SAX would be fine for
this task, but, hey, I personally would use Amara (
http://uche.ogbuji.net/tech/4suite/amara/ ), of course. The following
does the trick:

import sets
import amara
from amara import binderytools

#element_skeleton_rule suppresses char data from the resulting binding
#tree. If you have a large document and only care about element/attr
#structure and not text, this saves a lot of memory
rules = [binderytools.element_skeleton_rule()]
#XML can be a file path, URI, string, or even an open-file-like object
doc = amara.parse(XML, rules=rules)
elems = {}
for e in doc.xml_xpath('//*'):
paths = elems.setdefault((e.namespaceURI, e.localName), sets.Set())
path = u'/'.join([n.nodeName for n in e.xml_xpath(u'ancestor::*')])
paths.add(u'/' + path)

#Pretty-print output
for name in elems:
print name, '\n\t\t\t', '\n\t\t\t'.join(elems[name])

 

[George Sakkis]

Hi all,

Finally diving into XML programmatically. Does anyone have a best
practice recommendation for programmatically discovering the structure
of an arbitrary XML document via Python?

It seems like it is a common wheel I'd be re-inventing.

Thanks and cheers

I was looking for something similar (XML to DTD inference) but I didn't
find anything related in python. Trang
(http://www.thaiopensource.com/relaxng/trang-manual.html#introduction),
on the other hand seems impressive after a few non-trivial tests. It
would be neat to have it ported in python, at least the inference part.

George

 

[uche.ogbuji]

"""
I was looking for something similar (XML to DTD inference) but I didn't
find anything related in python. Trang
(http://www.thaiopensource.com/relaxng/trang-manual.html#introduction),
on the other hand seems impressive after a few non-trivial tests. It
would be neat to have it ported in python, at least the inference part.

"""

If you're OK with RELAX NG rather than DTD as the schema output
(probably a good idea if you're using namespaces), consider
Examplotron, which I've used on many such production tasks.

http://www-128.ibm.com/developerworks/xml/library/x-xmptron/

It's XSLT rather than Python, but the good news is that XSLT is easy to
invoke from Python using tools such as 4Suite.

http://uche.ogbuji.net/tech/akara/nodes/2003-01-01/python-xslt

 

[George Sakkis]

"""
I was looking for something similar (XML to DTD inference) but I didn't
find anything related in python. Trang
(http://www.thaiopensource.com/relaxng/trang-manual.html#introduction),
on the other hand seems impressive after a few non-trivial tests. It
would be neat to have it ported in python, at least the inference part.

"""

If you're OK with RELAX NG rather than DTD as the schema output
(probably a good idea if you're using namespaces), consider
Examplotron, which I've used on many such production tasks.

http://www-128.ibm.com/developerworks/xml/library/x-xmptron/

It's XSLT rather than Python, but the good news is that XSLT is easy to
invoke from Python using tools such as 4Suite.

http://uche.ogbuji.net/tech/akara/nodes/2003-01-01/python-xslt

Neat, though non-trivial XSLT makes my head spin. Just for kicks, I
rewrote in python Michael Kay's DTDGenerator
(http://saxon.sourceforge.net/dtdgen.html), though as the original it
has several limitations on the accuracy of the inferred DTD.

George

 

[uche.ogbuji]

"Neat, though non-trivial XSLT makes my head spin."

Well, you don't have to know XSLT at all to use the Examplotron
transform, although I can understand wanting to understand and hack
what you're using.

"Just for kicks, I
rewrote in python Michael Kay's DTDGenerator
(http://saxon.sourceforge.net/dtdgen.html), though as the original it
has several limitations on the accuracy of the inferred DTD. "

Ah. Cool. Got a link?