HTMLDocument and Xpath

S

swilson

Hi, I want to use xpath to scrape info from a website using pyXML but I
keep getting no results.

For example, in the following, I want to return the text "Element1" I
can't get xpath to return anything at all. What's wrong with this
code?

--------------------
from xml.dom.ext.reader import HtmlLib
from xml.xpath import Evaluate

reader = HtmlLib.Reader()
doc_node = reader.fromString("""
<html>
<head>
<title>Python Programming Language</title>
</head>
<body>
<table><tr><td>element1</td></tr></table>
</body>
</html>
""")

test = Evaluate('td', doc_node.documentElement)
print "test =", test
------------

All I get is an empty list for output.

Thx in advance

Shawn
 
A

Alan Kennedy

[[email protected]]
Hi, I want to use xpath to scrape info from a website using pyXML but I
keep getting no results.

For example, in the following, I want to return the text "Element1" I
can't get xpath to return anything at all. What's wrong with this
code?
Click to expand...

Your xpath expression is wrong.
test = Evaluate('td', doc_node.documentElement)
Click to expand...

Try one of the following alternatives, all of which should work.

test = Evaluate('//td', doc_node.documentElement)
test = Evaluate('/html/body/table/tr/td', doc_node.documentElement)
test = Evaluate('/html/body/table/tr/td[1]', doc_node.documentElement)

HTH,

Alan.
 
S

swilson

Alan said:
[[email protected]]
Hi, I want to use xpath to scrape info from a website using pyXML but I
keep getting no results.

For example, in the following, I want to return the text "Element1" I
can't get xpath to return anything at all. What's wrong with this
code?
Click to expand...

Your xpath expression is wrong.
test = Evaluate('td', doc_node.documentElement)
Click to expand...

Try one of the following alternatives, all of which should work.

test = Evaluate('//td', doc_node.documentElement)
test = Evaluate('/html/body/table/tr/td', doc_node.documentElement)
test = Evaluate('/html/body/table/tr/td[1]', doc_node.documentElement)

HTH,

Alan.
Click to expand...

I tried all of those and in every case, test returns "[]". Does
Evaluate only work with XML documents?

Shawn
 
S

swilson

Got the answer - there's a bug in xpath. I think the HTML parser
converts all the tags (but not the attributes) to uppercase. Xpath
definitely does not like my first string but, these work fine:

test = Evaluate('//TD', doc_node.documentElement)
test = Evaluate('/HTML/BODY/TABLE/TR/TD', doc_node.documentElement)
test = Evaluate('/HTML/BODY/TABLE/TR/TD[1]', doc_node.documentElement)

Shawn
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Sort by number of characters 1
Javascript DOM 1
How to wrap <td> content . 16
Can someone tell me if this a real tracker? Or is it one designed to show you a different message at certain times, ie. acting like one? 0
Script to send email not working 1
When I send email as HTML, why do erroneous whitespaces getintroduced to the HTML source and a few < 2
Loop through record in datareader in aspx for textbox other item 1
Getting extra blank rows from appending HTML..? 2

Members online

Total: 19 (members: 2, guests: 17)
Robots: 291

Forum statistics

Threads
473,755
Messages
2,569,535
Members
45,007
Latest member
obedient dusk

Latest Threads

Top