memory consumption in JDOM

Discussion in 'Java' started by vanandh22@gmail.com, May 12, 2006.

  1. Guest

    Hi All
    I am new to JDOM and I am currently coding a program to build xml
    files using JDOM.
    The size of the xml file which i am goin to build will be in the order
    of a few GB's( 1 to 5 GB).
    Will I be able to build such huge documents with JDOM??
    Also if there are any resources on jdom benchmarking please let me
    know the links.

    Thanks in Advance for your help
    Vijay Anandh
     
    , May 12, 2006
    #1
    1. Advertising

  2. Vijay Anandh Guest

    Any updates on this issue....,
    I am eagerly waiting for the answers.
     
    Vijay Anandh, May 13, 2006
    #2
    1. Advertising

  3. Chris Uppal Guest

    wrote:

    > I am new to JDOM and I am currently coding a program to build xml
    > files using JDOM.
    > The size of the xml file which i am goin to build will be in the order
    > of a few GB's( 1 to 5 GB).


    I'm not sure what the typical overhead of a JDOM structure is (how much RAM it
    takes compared to how much disk-space the equivalent XML would take up), and
    it's application-dependent anyway. Assume, for the sake of argument, that it's
    3:1.

    Under that assumption you'll need around 3 to 15 GB of addressable RAM just to
    hold the JDOM structure. So you will have to be running on a 64-bit JVM before
    you can even consider this.

    Then there's the space required for the JVM itself. And possibly the space
    required for the data from which you build your JDOM tree. I would hope that
    you don't try to convert the entire tree into a String before writing it out to
    file (or to the network), but maybe you would have to do so. So, when you are
    considering how much real RAM you need, you have to think in terms of at least
    1 GB more than the above estimate, and maybe even (in the worst case) more than
    double the estimate.

    Only you know whether your application can justify using those kinds of
    resources.

    Personally, I would think /very/ hard about why I cannot just build the XML
    incrementally (writing it out to file in one pass without ever holding all the
    data in memory at once). With luck, some thought would show that it is
    possible after all.

    -- chris
     
    Chris Uppal, May 14, 2006
    #3
  4. On 12 May 2006 05:03:05 -0700, wrote:
    > I am new to JDOM and I am currently coding a program to build xml
    >files using JDOM.
    > The size of the xml file which i am goin to build will be in the order
    >of a few GB's( 1 to 5 GB).
    > Will I be able to build such huge documents with JDOM??
    > Also if there are any resources on jdom benchmarking please let me
    >know the links.


    At a time, I found a few JDOM benchmarks, mostly bechmarking speed.
    JDOM scored pretty well compared to purely SAX tools, it is one of the
    fastest.

    As for the maximum size of an XML file... why not test it yourself?

    I've just run this simple test case:


    import org.jdom.Document;
    import org.jdom.Element;

    public class JDOM {
    // Timeout set to 60 seconds
    private static long timeout = 60 * 1000;

    public static void main(String[] args) {
    Element element = new Element("root");
    Document document = new Document(element);
    long i = 1;
    long startTime = System.currentTimeMillis();

    try {
    for (;; i++) {
    Element currElement = new Element("element" +
    String.valueOf(i));
    element.addContent(currElement);
    element = currElement;
    if (System.currentTimeMillis() - startTime > timeout){
    break;
    }
    }
    } catch (Exception ex) {
    System.out.println("Exception: " + ex.getMessage());
    }

    System.out.println("Number of created elements: " + i);
    }
    }

    It was able to create 28885 elements in 60 seconds. I left it running
    for a while without a timeout, to see if it would throw
    OutOfMemoryException, but it seems that the memory overhead for
    element is pretty low, so it would take forever to drain the heap.

    Domchi

    --
    Ouroboros ltd. - http://www.ouroboros.hr
    Antispam: to reply, remove extra monkey from reply-to address.
     
    Domagoj Klepac, May 15, 2006
    #4
  5. On Sun, 14 May 2006 10:00:43 +0100, "Chris Uppal"
    <-THIS.org> wrote:
    >I'm not sure what the typical overhead of a JDOM structure is (how much RAM it
    >takes compared to how much disk-space the equivalent XML would take up), and
    >it's application-dependent anyway. Assume, for the sake of argument, that it's
    >3:1.


    Yes, I'd say that's about right. On the one hand, Java stores Strings
    as Unicode, so one character takes 2 bytes - one byte more than on
    disk; on the other hand, if you have:

    <longxmlelementname>foo</longxmlelementname>

    You have to store only two Strings in memory:

    longxmlelementname foo

    And of course, there's JDOM element overhead. There's the good article
    on Java object overhead and memory consumption:

    http://www.javaworld.com/javaworld/javatips/jw-javatip130.html

    >Under that assumption you'll need around 3 to 15 GB of addressable RAM just to
    >hold the JDOM structure. So you will have to be running on a 64-bit JVM before
    >you can even consider this.


    It seems that the speed limit is the one you hit before the memory
    limit. If it takes a hours to fill even 500 MB of memory with XML
    data...

    >Personally, I would think /very/ hard about why I cannot just build the XML
    >incrementally (writing it out to file in one pass without ever holding all the
    >data in memory at once). With luck, some thought would show that it is
    >possible after all.


    Yes, some kind of SAX parser which doesn't load XML to memory would
    probably be more appropriate.

    Domchi

    --
    Ouroboros ltd. - http://www.ouroboros.hr
    Antispam: to reply, remove extra monkey from reply-to address.
     
    Domagoj Klepac, May 15, 2006
    #5
  6. Dale King Guest

    wrote:
    > Hi All
    > I am new to JDOM and I am currently coding a program to build xml
    > files using JDOM.


    A better alternative to JDOM that you might want to consider is XOM
    (www.xom.nu) which is much cleaner and easier to use.
    --
    Dale King
     
    Dale King, May 15, 2006
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Wendy S
    Replies:
    1
    Views:
    6,427
    Darren Davison
    Aug 5, 2003
  2. Bernd Oninger
    Replies:
    4
    Views:
    12,575
    GIMME
    Jun 21, 2004
  3. Tinker
    Replies:
    4
    Views:
    5,398
    Harry Bosch
    Oct 9, 2005
  4. Bernd Oninger
    Replies:
    3
    Views:
    2,913
    GIMME
    Jun 21, 2004
  5. Vijay Anandh

    Memory Consumption in JDOM

    Vijay Anandh, May 13, 2006, in forum: XML
    Replies:
    4
    Views:
    674
    Vijay Anandh
    May 13, 2006
Loading...

Share This Page