Re: Load huge xml file

Discussion in 'Java' started by Roedy Green, Apr 14, 2009.

  1. Roedy Green

    Roedy Green Guest

    On Tue, 14 Apr 2009 20:48:47 +0200, Zibi <> wrote,
    quoted or indirectly quoted someone who said :

    ><root>
    > <element1>...</element1>
    > <element1>...</element1>
    > <element2>...</element2>
    > <element2>...</element2>
    ></root>


    That was a file that never should have been XML in the first place
    since there is no nesting.

    I suggest writing a custom little program that reads it line by line
    and writes out the equivalent CSV file using CSV Writer,
    http://mindprod.com/products1.html#CSV (now supports comments) of the
    form:

    element1, data
    element1, data
    element2, data
    element2, data

    Then you can process it easily with CSVReader. You can also look and
    edit it with a text editor.

    You might use regex split or simple indexOf to find the pieces.

    XML was intended for small complex files. They are analysed as a tree
    totally ram resident. 100 MB is still only 0.1 gig. Use JET which auto
    adjusts the heap size depending on available RAM.
    http://mindprod.com/jgloss/jet.html

    With the Sun JVM, there are command line options to increase the
    virtual address space. http://mindprod.com/jgloss/javaexe.html

    Another approach is to split your file in pieces and process the
    pieces separately, then join the data in a binary DataOutputSTream
    file.

    See http://mindprod.com/products1.html#SPLICE (cut a file up based on
    offsets)

    http://mindprod.com/products1.html#SPLITTER (break a file up based on
    embedded markers)

    --
    Roedy Green Canadian Mind Products
    http://mindprod.com

    For most people, a global average temperature rise of 3°C (5°F) does not sound
    alarming. They don’t understand that the warming is uneven, mostly over the
    continents which will rise 6°C (11°F). The maximum annual temperature will
    increase considerably more. Paris will have the temperature of Algiers,
    Moscow the temperature of Paris, San Francisco the temperature of Tijuana and
    New York the temperature of Charleston, and Houston the temperature of some
    obscure dusty town in southern Mexico.
     
    Roedy Green, Apr 14, 2009
    #1
    1. Advertising

  2. Roedy Green

    Arne Vajhøj Guest

    bugbear wrote:
    > Roedy Green wrote:
    >> On Tue, 14 Apr 2009 20:48:47 +0200, Zibi <> wrote,
    >> quoted or indirectly quoted someone who said :
    >>
    >>> <root>
    >>> <element1>...</element1>
    >>> <element1>...</element1>
    >>> <element2>...</element2>
    >>> <element2>...</element2>
    >>> </root>

    >>
    >> That was a file that never should have been XML in the first place
    >> since there is no nesting.

    >
    > He *did* say:
    >
    > "Elements are quite complicated I have XSD for entire XML."


    Not everyone bother to read what they are replying to.

    Arne
     
    Arne Vajhøj, Apr 20, 2009
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Xenia
    Replies:
    4
    Views:
    440
    Xenia
    Nov 25, 2003
  2. Replies:
    3
    Views:
    509
  3. Replies:
    1
    Views:
    354
  4. Patrice
    Replies:
    0
    Views:
    659
    Patrice
    Nov 9, 2009
  5. CSharpner

    Re: Load a huge data into dropdown list

    CSharpner, Nov 11, 2009, in forum: ASP .Net
    Replies:
    2
    Views:
    1,245
    Alexey Smirnov
    Nov 12, 2009
Loading...

Share This Page