Scrapy/XPath help

Discussion in 'Python' started by Always Learning, Dec 21, 2012.

  1. Hello all. I'm new to Python, but have been playing around with it for a few weeks now, following tutorials, etc. I've spun off on my own and am trying to do some basic web scraping. I've used Firebug/View XPath in Firefox for some help with the XPaths, however, I still am receiving errors when I try to run this script. If you could help, it would be greatly appreciated!

    from scrapy.spider import BaseSpider
    from scrapy.selector import HtmlXPathSelector
    from cbb_info.items import CbbInfoItem, Field

    class GameInfoSpider(BaseSpider):
    name = "game_info"
    allowed_domains = ["www.sbrforum.com"]
    start_urls = [
    'http://www.sbrforum.com/betting-odds/ncaa-basketball/',
    ]

    def parse(self, response):
    hxs = HtmlXPathSelector(response)
    toplevels = hxs.select("//div[@class='eventLine-value']")
    items = []
    for toplevels in toplevels:
    item = CbbInfoItem()
    item ["teams"] = toplevels.select("/span[@class='team-name'/text()").extract()
    item ["lines"] = toplevels.select("/div[@rel='19']").extract()
    item.append(item)
    return items
    Always Learning, Dec 21, 2012
    #1
    1. Advertising

  2. Always Learning

    Grant Rettke Guest

    You might have better luck if you share the python make, version, os,
    error message, and some unit tests demonstrating what you expect.

    On Fri, Dec 21, 2012 at 3:21 PM, Always Learning <> wrote:
    > Hello all. I'm new to Python, but have been playing around with it for a few weeks now, following tutorials, etc. I've spun off on my own and am trying to do some basic web scraping. I've used Firebug/View XPath in Firefox for some help with the XPaths, however, I still am receiving errors when I try to run this script. If you could help, it would be greatly appreciated!
    >
    > from scrapy.spider import BaseSpider
    > from scrapy.selector import HtmlXPathSelector
    > from cbb_info.items import CbbInfoItem, Field
    >
    > class GameInfoSpider(BaseSpider):
    > name = "game_info"
    > allowed_domains = ["www.sbrforum.com"]
    > start_urls = [
    > 'http://www.sbrforum.com/betting-odds/ncaa-basketball/',
    > ]
    >
    > def parse(self, response):
    > hxs = HtmlXPathSelector(response)
    > toplevels = hxs.select("//div[@class='eventLine-value']")
    > items = []
    > for toplevels in toplevels:
    > item = CbbInfoItem()
    > item ["teams"] = toplevels.select("/span[@class='team-name'/text()").extract()
    > item ["lines"] = toplevels.select("/div[@rel='19']").extract()
    > item.append(item)
    > return items
    > --
    > http://mail.python.org/mailman/listinfo/python-list




    --
    Grant Rettke | ACM, AMA, COG, IEEE
    | http://www.wisdomandwonder.com/
    Wisdom begins in wonder.
    ((λ (x) (x x)) (λ (x) (x x)))
    Grant Rettke, Dec 21, 2012
    #2
    1. Advertising

  3. Sorry about that. I'm using Python 2.7.3, 32 bit one Windows 7.

    The errors I get are

    >>File "C:\python27\lib\site-packages\scrapy-0.16.3-py2.7.egg\scrapy\selector\lxmlsel.py", line 47, in select
    >>raise ValueError("Invalid XPath: %s" % xpath)
    >>exceptions.ValueError: Invalid XPath: /span[@class='team-name'/text()


    Ultimaly, I expect it to gather the team name in text, and then the odds in one of the columns in text as well, so I can then put it into a .csv
    Always Learning, Dec 21, 2012
    #3
  4. Sorry about that. I'm using Python 2.7.3, 32 bit one Windows 7.

    The errors I get are

    >>File "C:\python27\lib\site-packages\scrapy-0.16.3-py2.7.egg\scrapy\selector\lxmlsel.py", line 47, in select
    >>raise ValueError("Invalid XPath: %s" % xpath)
    >>exceptions.ValueError: Invalid XPath: /span[@class='team-name'/text()


    Ultimaly, I expect it to gather the team name in text, and then the odds in one of the columns in text as well, so I can then put it into a .csv
    Always Learning, Dec 21, 2012
    #4
  5. Always Learning

    Dave Angel Guest

    On 12/21/2012 04:58 PM, Always Learning wrote:
    > Sorry about that. I'm using Python 2.7.3, 32 bit one Windows 7.
    >
    > The errors I get are
    >
    >>> File "C:\python27\lib\site-packages\scrapy-0.16.3-py2.7.egg\scrapy\selector\lxmlsel.py", line 47, in select
    >>> raise ValueError("Invalid XPath: %s" % xpath)
    >>> exceptions.ValueError: Invalid XPath: /span[@class='team-name'/text()

    > Ultimaly, I expect it to gather the team name in text, and then the odds in one of the columns in text as well, so I can then put it into a .csv


    Why are you displaying only the last 3 lines of the error message?
    Unless your source code is lxmlsel.py, there are other stack levels
    above this one.

    (I can't help, but I'm trying to save some time for someone who can)

    --

    DaveA
    Dave Angel, Dec 22, 2012
    #5
  6. Always Learning

    donarb Guest

    On Friday, December 21, 2012 1:58:47 PM UTC-8, Always Learning wrote:
    > The errors I get are
    > >>File "C:\python27\lib\site-packages\scrapy-0.16.3-py2.7.egg\scrapy\selector\lxmlsel.py", line 47, in select

    >
    > >>raise ValueError("Invalid XPath: %s" % xpath)

    >
    > >>exceptions.ValueError: Invalid XPath: /span[@class='team-name'/text()

    >



    You're missing a right bracket in the xpath expression:

    /span[@class='team-name']/text()
    donarb, Dec 25, 2012
    #6
  7. Always Learning

    donarb Guest

    On Friday, December 21, 2012 1:58:47 PM UTC-8, Always Learning wrote:
    > The errors I get are
    > >>File "C:\python27\lib\site-packages\scrapy-0.16.3-py2.7.egg\scrapy\selector\lxmlsel.py", line 47, in select

    >
    > >>raise ValueError("Invalid XPath: %s" % xpath)

    >
    > >>exceptions.ValueError: Invalid XPath: /span[@class='team-name'/text()

    >



    You're missing a right bracket in the xpath expression:

    /span[@class='team-name']/text()
    donarb, Dec 25, 2012
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Zeynel
    Replies:
    3
    Views:
    1,462
    Cousin Stanley
    Nov 21, 2009
  2. Sky Larking

    Installing Scrapy on Mac OS X 10.6

    Sky Larking, Mar 3, 2010, in forum: Python
    Replies:
    1
    Views:
    1,356
    Ned Deily
    Mar 3, 2010
  3. Dmitry Arsentiev
    Replies:
    3
    Views:
    292
    Stefan Behnel
    Aug 18, 2012
  4. Nomen Nescio

    a few questions about scrapy

    Nomen Nescio, Sep 18, 2012, in forum: Python
    Replies:
    0
    Views:
    221
    Nomen Nescio
    Sep 18, 2012
  5. YetToCome

    How To Build Scrapy Environment

    YetToCome, Sep 23, 2013, in forum: Python
    Replies:
    5
    Views:
    238
    YetToCome
    Sep 23, 2013
Loading...

Share This Page