XML - Xpath Memory issues

[suresh.csharp]

Hi,
I am trying to import huge data into XML files it is in more than
700Mbs, creating a index(catalog) XML file for all the imported XML
files for searching xml files content. The idex(catalog) file will be
more than
100mb. When I try to load this index(catalog) file to DOM on IE, it is
taking lot of time and when I do search on this document with Xpath and
load the results it is taking almost like 2-4 minutes and some times it
will give out of memory error. I have to run this entire application
in CD's and run on clients browsers.

I am doing lot of google on this issue.. but not much help. Can any one
tell me.. How much size of XML file load on to DOM, it will stable and
doing Xpath on this document. Any performance tips most welcome..

Thanks
Suresh

 

[Joe Kesselman]

How much size of XML file load on to DOM, it will stable and
doing Xpath on this document. Any performance tips most welcome..

Depends on the DOM implementation. (Remember, the DOM is just an API,
and that same API may have many different kinds of data storage
implemented behind it.)

Simple-minded DOM implementations (one object per node) do tend to burn
a lot of memory, especialy in Java where objects have a lot of overhead.
That shouldn't be a huge problem in a system with decent virtual memory,
but you may run into swapping.

Some DOM implementations may use more compact back-end representations,
simply exposing them through DOM proxy objects.

Or the best answer may not be to use DOM at all. Apache Xalan uses a
non-DOM data model known as DTM. If your task can be implemented as an
essentially linear scan of the document, you may not even need a
complete in-memory model and may prefer to do a SAX-based streaming
solution.

(See also http://www.w3.org/DOM/faq.html#SAXandDOM)