Iterparse and ElementTree confusion

Discussion in 'Python' started by paul.sherwood@gmail.com, Aug 17, 2005.

  1. Guest

    Hi

    Im trying to parse a large(150MB) xml file in order to extract specific
    required records.

    import sys
    from elementtree.ElementTree import ElementTree

    root = ElementTree(file=big.xml')

    This works fine for smaller versions of the same xml file but...when i
    attempted the above my PC goes to lala land, theres much HDD grinding
    followed by "windows runnign low on virtual memory" popup after
    10-15mins. Then just more grinding...for an hour before i gave up

    XML file format:
    <root>
    <rubbish1>
    .
    .
    </rubbish1>
    .
    .
    <rubbishX>
    .
    .
    </rubbishX>
    <Products>
    <Product ID="QU17861" UserTypeID="PH_QUOTE" QualifierID="Qualifier
    root" ParentID="LIVE_AREA">
    <Name QualifierID="Qualifier root">23172</Name>
    <Description QualifierID="Qualifier root">Three Spot Rail Light
    Brushed Chrome</Description>
    <ClassificationReference ClassificationID="W2@Kitchen Lighting"
    QualifierID="Qualifier root" Type="" />
    <ProductReference ProductID="QU17749" QualifierID="Qualifier root"
    Type="Accessory / Linked Product">
    <Name QualifierID="Qualifier root">73520</Name>
    .
    .etc
    </Product>
    </Products>
    </root>

    Ok, i thought, surely theres a way to parse this thing in chucnks till
    i get to the element i require then I'll reuse the ElementTree
    goodness.

    I found Iterparse

    def parse_for_products(filename):

    for event, elem in iterparse(filename):
    if elem.tag == "Products":
    root = ElementTree(elem)
    print_all(root)
    else:
    elem.clear()

    My problem is that if i pass the 'elem' found by iterparse then try to
    print all attributes, children and tail text i only get
    elem.tag....elem.keys returns nothing as do all of the other previously
    useful elementtree methods.

    Am i right in thinking that you can pass an element into ElementTree?
    How might i manually iterate through <product>...</product> grabbing
    everything?
     
    , Aug 17, 2005
    #1
    1. Advertising

  2. Guest

    Further to the above....

    I pass my found element 'Product' to ElementTree's handy print all
    function

    def print_all(element):
    root = element
    #Create an iterator
    iter = root.getiterator()
    #Iterate
    for element in iter:
    #First the element tag name
    print "Element:", element.tag
    #Next the attributes (available on the instance itself using
    #the Python dictionary protocol
    if element.keys():
    print "\tAttributes:"
    for name, value in element.items():
    print "\t\tName: '%s', Value: '%s'"%(name, value)
    #Next the child elements and text
    print "\tChildren:"
    #Text that precedes all child elements (may be None)
    if element.text:
    text = element.text
    text = len(text) > 40 and text[:40] + "..." or text
    print "\t\tText:", repr(text)
    if element.getchildren():
    #Can also use: "for child in element.getchildren():"
    for child in element:
    #Child element tag name
    print "\t\tElement", child.tag
    #The "tail" on each child element consists of the text
    #that comes after it in the parent element content, but
    #before its next sibling.
    if child.tail:
    text = child.tail
    text = len(text) > 40 and text[:40] + "..." or text
    print "\t\tText:", repr(text)

    if the element i pass to the above is from

    root = ElementTree(file='somefile.xml')
    iter = root.getiterator()
    #Iterate
    for element in iter:
    if element.tag == 'Product':
    print_all(element)

    I print all of my required element's attributes, children, children's
    children, etc

    However if i pass the element i get from iterparse

    for event, elem in iterparse(filename):
    if elem.tag == "Products":
    print_all(elem)
    else:
    elem.clear()

    i only get the attributes for <product> and a list of its children, no
    further iteration into the children and the children's children

    what am i missing?
     
    , Aug 17, 2005
    #2
    1. Advertising

  3. wrote:

    > def parse_for_products(filename):
    >
    > for event, elem in iterparse(filename):
    > if elem.tag == "Products":
    > root = ElementTree(elem)
    > print_all(root)
    > else:
    > elem.clear()
    >
    > My problem is that if i pass the 'elem' found by iterparse then try to
    > print all attributes, children and tail text i only get
    > elem.tag....elem.keys returns nothing as do all of the other previously
    > useful elementtree methods.
    >
    > Am i right in thinking that you can pass an element into ElementTree?
    > How might i manually iterate through <product>...</product> grabbing
    > everything?


    by default, iterparse only returns "end" events, which means that the
    iterator will visit the Products children before you see the Products
    element itself. with the code above, this means that the children will
    be nuked before you get around to process the parent.

    depending on how much rubbish you have in the file, you can do

    for event, elem in iterparse(filename):
    if elem.tag == "Products":
    process(elem)
    elem.clear()

    or

    for event, elem in iterparse(filename):
    if elem.tag == "Products":
    process(elem)
    elem.clear()
    elif elem.tag in ("Rubbish1", "Rubbish2"):
    elem.clear()

    or

    inside = False
    for event, elem in iterparse(filename, events=("start", "end")):
    if event == "start":
    # we've seen the start tag for this element, but not
    # necessarily the end tag
    if elem.tag == "Products":
    inside = True
    else:
    # we've seen the end tag
    if elem.tag == "Products":
    process(elem)
    elem.clear()
    inside = False
    elif not inside:
    elem.clear()

    for more info, see

    http://effbot.org/zone/element-iterparse.htm

    </F>
     
    Fredrik Lundh, Aug 17, 2005
    #3
  4. Steve M Guest

    "when i attempted [to load 150MB xml file] my PC goes to lala land,
    theres much HDD grinding followed by "windows runnign low on virtual
    memory" popup after 10-15mins. Then just more grinding...for an hour
    before i gave up"

    I have had great success using SAX to parse large XML files. If you use
    care your memory use will peak at a low number no matter how much XML
    you chew through.
     
    Steve M, Aug 17, 2005
    #4
  5. Guest

    Ofcourse!

    Thankyou so much. All is well, using the Iterparse takes a mere 3mins
    to find the last product in the xml and this is probably due to my
    checking 10000 products for a specific.

    <looks at shoes>I feel rather honoured to have 'the' effbot reply to my
    humble post....</looks at shoes>

    Thanks again

    Paul
     
    , Aug 18, 2005
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Stuart McGraw

    elementtree: line numbers and iterparse

    Stuart McGraw, Sep 13, 2006, in forum: Python
    Replies:
    2
    Views:
    460
    Stuart McGraw
    Sep 13, 2006
  2. George Sakkis

    Re: iterparse and unicode

    George Sakkis, Aug 21, 2008, in forum: Python
    Replies:
    6
    Views:
    256
    George Sakkis
    Aug 27, 2008
  3. Kee Nethery
    Replies:
    12
    Views:
    2,094
    Stefan Behnel
    Jun 27, 2009
  4. bfrederi

    Issue with xml iterparse

    bfrederi, Jun 3, 2010, in forum: Python
    Replies:
    4
    Views:
    753
    Stefan Behnel
    Jun 13, 2010
  5. Ben Sizer
    Replies:
    5
    Views:
    959
Loading...

Share This Page