BeautifulSoup and Problem Tables

Discussion in 'Python' started by academicedgar@gmail.com, Sep 21, 2008.

  1. Guest

    Hi

    I would appreciate some help. I am trying to learn Python and want to
    use BeautifulSoup to pull some data from tables. I was really psyched
    earlier tonight when I discovered that I could do this

    from BeautifulSoup import BeautifulSoup
    bst=file(r"c:\bstest.htm").read()
    soup=BeautifulSoup(bst)
    rows=soup.findAll('tr')
    len(rows)
    a=len(rows[0].findAll('td'))
    b=len(rows[1].findAll('td'))
    c=len(rows[2].findAll('td'))
    d=len(rows[3].findAll('td'))
    e=len(rows[4].findAll('td'))
    f=len(rows[5].findAll('td'))
    g=len(rows[6].findAll('td'))
    h=len(rows[8].findAll('td'))
    i=len(rows[9].findAll('td'))
    j=len(rows[10].findAll('td'))
    k=rows[1].findAll('td')[1].contents[0]


    So here I am chortling to myself thinking this is too easy. I know
    that the data columns are in rows[0] and so I can learn some more
    python to figure out how to create tuples so I can lable each data
    item using the row and column headings plucked from the contents.

    However, I discovered that my tables have inconsistent numbers of
    rows. Even though the tables look pretty. It might be that the
    column heading for the third column is "Apples" but the value for
    "Apples" in the fourth row is not in the third position in the row but
    the fourth.

    Now I am reluctant to make any assumptions because the tables were
    created inconsistently. What I mean is that in some tables if there is
    no value for a row/column intersection then there is a blank line, in
    other tables if there is no value for a row/column intersection then
    the length of k (as above) is 0.

    I have been Googling for some insight into this and I have not been
    successful finding anything. I would really appreciate any suggestions
    or some direction about how to better describe the problem.
     
    , Sep 21, 2008
    #1
    1. Advertising

  2. On Sat, 20 Sep 2008 20:51:52 -0700 (PDT), wrote:
    [snip]
    > from BeautifulSoup import BeautifulSoup
    > bst=file(r"c:\bstest.htm").read()
    > soup=BeautifulSoup(bst)
    > rows=soup.findAll('tr')
    > len(rows)
    > a=len(rows[0].findAll('td'))
    > b=len(rows[1].findAll('td'))
    > c=len(rows[2].findAll('td'))
    > d=len(rows[3].findAll('td'))
    > e=len(rows[4].findAll('td'))
    > f=len(rows[5].findAll('td'))
    > g=len(rows[6].findAll('td'))
    > h=len(rows[8].findAll('td'))
    > i=len(rows[9].findAll('td'))
    > j=len(rows[10].findAll('td'))
    > k=rows[1].findAll('td')[1].contents[0]

    [snip]
    > However, I discovered that my tables have inconsistent numbers of
    > rows.

    [snip]
    > I have been Googling for some insight into this and I have not been
    > successful finding anything. I would really appreciate any suggestions
    > or some direction about how to better describe the problem.


    Would it be accurate to describe the problem as wanting to
    extract the contents of the cth column of the rth row of a
    table in spite of various pathologies in the construction of
    the table?

    If so, maybe it would help to post sample HTML (trimmed to a
    minimum) of the pathologies that must be handled. I gotta
    confess, though, that it doesn't take many rowspans or colspans
    to put this problem beyond my reach.

    --
    To email me, substitute nowhere->spamcop, invalid->net.
     
    Peter Pearson, Sep 21, 2008
    #2
    1. Advertising

  3. clurks Guest

    wrote:

    > Hi
    >
    > I would appreciate some help. I am trying to learn Python and want to
    > use BeautifulSoup to pull some data from tables. I was really psyched
    > earlier tonight when I discovered that I could do this
    >
    > from BeautifulSoup import BeautifulSoup
    > bst=file(r"c:\bstest.htm").read()
    > soup=BeautifulSoup(bst)
    > rows=soup.findAll('tr')
    > len(rows)
    > a=len(rows[0].findAll('td'))
    > b=len(rows[1].findAll('td'))
    > c=len(rows[2].findAll('td'))
    > d=len(rows[3].findAll('td'))
    > e=len(rows[4].findAll('td'))
    > f=len(rows[5].findAll('td'))
    > g=len(rows[6].findAll('td'))
    > h=len(rows[8].findAll('td'))
    > i=len(rows[9].findAll('td'))
    > j=len(rows[10].findAll('td'))
    > k=rows[1].findAll('td')[1].contents[0]
    >
    >
    > So here I am chortling to myself thinking this is too easy. I know
    > that the data columns are in rows[0] and so I can learn some more
    > python to figure out how to create tuples so I can lable each data
    > item using the row and column headings plucked from the contents.
    >
    > However, I discovered that my tables have inconsistent numbers of
    > rows. Even though the tables look pretty. It might be that the
    > column heading for the third column is "Apples" but the value for
    > "Apples" in the fourth row is not in the third position in the row but
    > the fourth.
    >
    > Now I am reluctant to make any assumptions because the tables were
    > created inconsistently. What I mean is that in some tables if there is
    > no value for a row/column intersection then there is a blank line, in
    > other tables if there is no value for a row/column intersection then
    > the length of k (as above) is 0.
    >
    > I have been Googling for some insight into this and I have not been
    > successful finding anything. I would really appreciate any suggestions
    > or some direction about how to better describe the problem.


    This may help, and it may not.

    One of the most useful features of BeautifulSoup is the .prettify()
    function. Use it, and you can examine your xml with a browser or
    decent editor. You will be able, hopefully, to see attributes that
    may explain your difficulties. The one that snuck up and bit me was
    table:number-columns-repeated -- I don't know what is biting you,
    but it is weird the way known cells are completely skipped with this
    attribute -- you have to know about it and check for it.

    If there is a way to deal with that attribute using BeautifulSoup I
    was unable to find it given the documentation and examples available
    to me. I took a step back and tried xml.sax -- it is straighforward
    and easy to use -- you build an xml.sax.handler for your xml, and it
    deals with it.

    It worked for me, but, as always, YMMV

    sc
     
    clurks, Sep 22, 2008
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    7
    Views:
    768
    Kent Johnson
    Apr 4, 2006
  2. placid

    BeautifulSoup problem

    placid, Oct 20, 2006, in forum: Python
    Replies:
    2
    Views:
    310
    placid
    Oct 20, 2006
  3. elca
    Replies:
    0
    Views:
    331
  4. Greg
    Replies:
    9
    Views:
    834
    Nobody
    Oct 8, 2011
  5. Replies:
    5
    Views:
    87
    Peter Otten
    Dec 16, 2013
Loading...

Share This Page