XPathAPI, or precompiled XPaths

[David Portabella]

Hello,

I am using XPathAPI class.

The problem that I have is that after using XObject object =
XPathAPI.eval(node, xpathStr)
on 1000 different nodes (same xpathStr), it really becomes very slow.

This behavior is indeed documented in the documentation of XPathAPI:
http://xml.apache.org/xalan-j/apidocs/org/apache/xpath/XPathAPI.html
++++++
The methods in this class are convenience methods into the low-level
XPath API. These functions tend to be a little slow, since a number of
objects must be created for each evaluation. A faster way is to
precompile the XPaths using the low-level API, and then just use the
XPaths over and over. NOTE: In particular, each call to this method
will create a new XPathContext, a new DTMManager... and thus a new
DTM. That's very safe, since it guarantees that you're always
processing against a fully up-to-date view of your document. But it's
also portentially very expensive, since you're rebuilding the DTM
every time. You should consider using an instance of CachedXPathAPI
rather than these static methods.
+++++++

The explanation talks about precompiling the XPaths using the low-
level API, and then just use the XPaths over and over.
How to do this?

I would need something like:

++++++++++++
compiledXPath = XXX.compileXPath(xpathStr);
....

and then run this for 1000 different nodes:
XObject object = compiledXPath.eval(node)
++++++++++++

Do you know how to do this?

Regards,
DAvid

 

[David Portabella]

I have tried the following:

execute this once:
XPathContext xpathSupport = new XPathContext();
PrefixResolverDefault prefixResolver = new
PrefixResolverDefault(document);
XPath xpath = new XPath(xpathStr, null, prefixResolver,
XPath.SELECT, null);

and then, for each node:
int ctxtNode = xpathSupport.getDTMHandleFromNode(contextNode);
XObject object = xpath.execute(xpathSupport, node,
prefixResolver);

but it still very slow (the execution time did not change much).
It goes fast with the first nodes, but after 200 nodes it starts to go
really slow (1 second per node)

What can be the problem?
How can I speed up this?

 

[David Portabella]

Some more info:

I am using xalan 2.7.0: http://xml.apache.org/xalan-j/
As I run XPathAPI.eval(node, xpathStr) over and over again on several
nodes, it gets slower and slower.
This is documented in the XPathAPI documentation, and it suggests to
use the low-level XPath API:
http://xml.apache.org/xalan-j/apidocs/org/apache/xpath/XPathAPI.html

I am now using the low-level XPath API as follows:
XPathContext xpathSupport = new XPathContext();
PrefixResolverDefault prefixResolver = new
PrefixResolverDefault(document);
XPath xpath = new XPath(xpathStr, null, prefixResolver,
XPath.SELECT, null);

and then, for each node:
int ctxtNode = xpathSupport.getDTMHandleFromNode(contextNode);
XObject object = xpath.execute(xpathSupport, node,
prefixResolver);

It gets a bit better, but still, after using over and over again on
several nodes, it gets slower and slower.
I think that the problem is that
XPathContext.getDTMHandleFromNode(child) does not free memory.

Test this simplistic example yourself:
++++++++++++++++++++++++++++++++++++++++++++
import org.w3c.dom.*;
import javax.xml.parsers.*;
import javax.xml.transform.*;
import javax.xml.transform.dom.*;
import org.apache.xpath.*;
import org.apache.xml.utils.*;

public class Test {
public static void main(String[] argv) throws Exception {
int numChilds = 100000+1;

System.out.println("Building a document with " + numChilds + "
childs");
Document doc =
DocumentBuilderFactory.newInstance().newDocumentBuilder().newDocument();
Element root = doc.createElement("root");
doc.appendChild(root);
for (int i = 0; i < numChilds; i ++) {
Element child = doc.createElement("child");
root.appendChild(child);
Element subChild = doc.createElement("sub-child");
child.appendChild(subChild);
Element subSubChild = doc.createElement("sub-sub-child");
subChild.appendChild(subSubChild);
subSubChild.setAttribute("title", "title" + i);
}


XPathContext xpathSupport = new XPathContext();
PrefixResolverDefault prefixResolver = new
PrefixResolverDefault(doc);
XPath titleXpath = new XPath("sub-child/sub-sub-child/@title",
null, prefixResolver, XPath.SELECT, null);
Runtime r = Runtime.getRuntime();

System.out.println("Evaluating XPath for each " + numChilds +
" childs");
NodeList nodeList = root.getChildNodes();
int size = nodeList.getLength();
for (int i = 0; i < size; i++) {
long start = System.currentTimeMillis();
Element child = (Element) nodeList.item(i);
int ctxtNode = xpathSupport.getDTMHandleFromNode(child);
//String title = titleXpath.execute(xpathSupport,
ctxtNode, prefixResolver).toString();
long duration = System.currentTimeMillis() - start;
if (i < 10 || (i % (numChilds/10)) == 0)
System.out.println("child #" + i + "\t took " +
duration + " ms." +
"\tfreeMemory: " + r.freeMemory() +
"\ttotalMemory: "+r.totalMemory());
else if (i == 10)
System.out.println("printing some selected childs only
from now on...");
}
}
}

++++++++++++++++++++++++++++++++++++++++++++
Here you can see an example of the result:

$ java Test
Building a document with 100001 childs
Evaluating XPath for each 100001 childs
child #0 took 77 ms. freeMemory: 10642840 totalMemory:
45129728
child #1 took 1 ms. freeMemory: 10583848 totalMemory:
45129728
child #2 took 0 ms. freeMemory: 10583848 totalMemory:
45129728
child #3 took 0 ms. freeMemory: 10583848 totalMemory:
45129728
child #4 took 0 ms. freeMemory: 10583848 totalMemory:
45129728
child #5 took 0 ms. freeMemory: 10583848 totalMemory:
45129728
child #6 took 0 ms. freeMemory: 10583848 totalMemory:
45129728
child #7 took 1 ms. freeMemory: 10583848 totalMemory:
45129728
child #8 took 0 ms. freeMemory: 10583848 totalMemory:
45129728
child #9 took 0 ms. freeMemory: 10583848 totalMemory:
45129728
printing some selected childs only from now on...
child #10000 took 3 ms. freeMemory: 10980392 totalMemory:
45129728
child #20000 took 5 ms. freeMemory: 9976808 totalMemory:
45129728
child #30000 took 7 ms. freeMemory: 6332656 totalMemory:
45129728
child #40000 took 9 ms. freeMemory: 5112168 totalMemory:
45129728
child #50000 took 12 ms. freeMemory: 1373472 totalMemory:
45129728
child #60000 took 14 ms. freeMemory: 19851264 totalMemory:
66650112
child #70000 took 16 ms. freeMemory: 16515832 totalMemory:
66650112
child #80000 took 19 ms. freeMemory: 15040280 totalMemory:
66650112
child #90000 took 21 ms. freeMemory: 7435744 totalMemory:
66650112
child #100000 took 24 ms. freeMemory: 17416944 totalMemory:
66650112


++++++++++++++++++++++++++++++++++++++++++++
each time I call xpathSupport.getDTMHandleFromNode(child) it does not
free the memory,
and so it gets slower and slower.

How to solve this problem?
Some people has suggested to use the DOM4J package instead of Xalan.
However, we already have quite a lot of software using Xalan and
changing the code would have some cost.
Is it possible to solve this problem without discarding xalan?

Regards,
DAvid