Scrapy/XPath help

[Always Learning]

Hello all. I'm new to Python, but have been playing around with it for a few weeks now, following tutorials, etc. I've spun off on my own and am trying to do some basic web scraping. I've used Firebug/View XPath in Firefox for some help with the XPaths, however, I still am receiving errors when I try to run this script. If you could help, it would be greatly appreciated!

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from cbb_info.items import CbbInfoItem, Field

class GameInfoSpider(BaseSpider):
name = "game_info"
allowed_domains = ["www.sbrforum.com"]
start_urls = [
'http://www.sbrforum.com/betting-odds/ncaa-basketball/',
]

def parse(self, response):
hxs = HtmlXPathSelector(response)
toplevels = hxs.select("//div[@class='eventLine-value']")
items = []
for toplevels in toplevels:
item = CbbInfoItem()
item ["teams"] = toplevels.select("/span[@class='team-name'/text()").extract()
item ["lines"] = toplevels.select("/div[@rel='19']").extract()
item.append(item)
return items

 

[Grant Rettke]

You might have better luck if you share the python make, version, os,
error message, and some unit tests demonstrating what you expect.

Hello all. I'm new to Python, but have been playing around with it for a few weeks now, following tutorials, etc. I've spun off on my own and am trying to do some basic web scraping. I've used Firebug/View XPath in Firefox for some help with the XPaths, however, I still am receiving errors when I try to run this script. If you could help, it would be greatly appreciated!

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from cbb_info.items import CbbInfoItem, Field

class GameInfoSpider(BaseSpider):
name = "game_info"
allowed_domains = ["www.sbrforum.com"]
start_urls = [
'http://www.sbrforum.com/betting-odds/ncaa-basketball/',
]

def parse(self, response):
hxs = HtmlXPathSelector(response)
toplevels = hxs.select("//div[@class='eventLine-value']")
items = []
for toplevels in toplevels:
item = CbbInfoItem()
item ["teams"] = toplevels.select("/span[@class='team-name'/text()").extract()
item ["lines"] = toplevels.select("/div[@rel='19']").extract()
item.append(item)
return items

 

[Always Learning]

Sorry about that. I'm using Python 2.7.3, 32 bit one Windows 7.

The errors I get are

File "C:\python27\lib\site-packages\scrapy-0.16.3-py2.7.egg\scrapy\selector\lxmlsel.py", line 47, in select
raise ValueError("Invalid XPath: %s" % xpath)
exceptions.ValueError: Invalid XPath: /span[@class='team-name'/text()

Ultimaly, I expect it to gather the team name in text, and then the odds in one of the columns in text as well, so I can then put it into a .csv

 

[Always Learning]

Sorry about that. I'm using Python 2.7.3, 32 bit one Windows 7.

The errors I get are

File "C:\python27\lib\site-packages\scrapy-0.16.3-py2.7.egg\scrapy\selector\lxmlsel.py", line 47, in select
raise ValueError("Invalid XPath: %s" % xpath)
exceptions.ValueError: Invalid XPath: /span[@class='team-name'/text()

Ultimaly, I expect it to gather the team name in text, and then the odds in one of the columns in text as well, so I can then put it into a .csv

 

[Dave Angel]

Sorry about that. I'm using Python 2.7.3, 32 bit one Windows 7.

The errors I get are

File "C:\python27\lib\site-packages\scrapy-0.16.3-py2.7.egg\scrapy\selector\lxmlsel.py", line 47, in select
raise ValueError("Invalid XPath: %s" % xpath)
exceptions.ValueError: Invalid XPath: /span[@class='team-name'/text()

Ultimaly, I expect it to gather the team name in text, and then the odds in one of the columns in text as well, so I can then put it into a .csv

Why are you displaying only the last 3 lines of the error message?
Unless your source code is lxmlsel.py, there are other stack levels
above this one.

(I can't help, but I'm trying to save some time for someone who can)

 

[donarb]

The errors I get are

File "C:\python27\lib\site-packages\scrapy-0.16.3-py2.7.egg\scrapy\selector\lxmlsel.py", line 47, in select
raise ValueError("Invalid XPath: %s" % xpath)
exceptions.ValueError: Invalid XPath: /span[@class='team-name'/text()

You're missing a right bracket in the xpath expression:

/span[@class='team-name']/text()

 

[donarb]

The errors I get are

File "C:\python27\lib\site-packages\scrapy-0.16.3-py2.7.egg\scrapy\selector\lxmlsel.py", line 47, in select
raise ValueError("Invalid XPath: %s" % xpath)
exceptions.ValueError: Invalid XPath: /span[@class='team-name'/text()

You're missing a right bracket in the xpath expression:

/span[@class='team-name']/text()