Help regarding urllib

[malhar vora]

Hello All,


I am simply fetching data from robots.txt of a url. Below is my code.

siteurl = siteurl.rstrip("/")

 

[malhar vora]

Hello All,





I am simply fetching data from robots.txt of a url. Below is my code.



siteurl = siteurl.rstrip("/")

Sorry for last complete. It was sent by mistake.

Here is my code.

siteurl = siteurl.rstrip("/")
roboturl = siteurl + r'/robots.txt'
robotdata = urllib.urlopen(roboturl).read() # Reading robots.txt of given url
print robotdata

In above code siteurl is fetched simply from local text file.


Whenever I run above code. In place of "/" before robots.txt, it writes "\\" in url as I found in error. The error is given below.

This is main function
Main URL : www.bestrecipes.com.au
$$$$$$$$$$:www.bestrecipes.com.au
###########-->www.bestrecipes.com.au/robots.txt
Traceback (most recent call last):
File "dataintegrator.py", line 104, in <module>
main()
File "dataintegrator.py", line 81, in main
print "Sitemap Url : " + getSiteMapUrl(i)
File "D:\Malhar Data\Projects\Data Parsing\My Code\Final Part\libs\datareader.
py", line 50, in getSiteMapUrl
robotdata = urllib.urlopen(roboturl).read() # Reading robots.txt of given ur
l
File "C:\Python26\lib\urllib.py", line 87, in urlopen
return opener.open(url)
File "C:\Python26\lib\urllib.py", line 203, in open
return getattr(self, name)(url)
File "C:\Python26\lib\urllib.py", line 461, in open_file
return self.open_local_file(url)
File "C:\Python26\lib\urllib.py", line 475, in open_local_file
raise IOError(e.errno, e.strerror, e.filename)
IOError: [Errno 2] The system cannot find the path specified: 'www.bestrecipes.c
om.au\\robots.txt'

I am new to Python and not able to figure out this problem. Please help me.

Thank you,

Malhar Vora

 

[Dave Angel]

Sorry for last complete. It was sent by mistake.

Here is my code.

siteurl = siteurl.rstrip("/")
roboturl = siteurl + r'/robots.txt'
robotdata = urllib.urlopen(roboturl).read() # Reading robots.txt of given url
print robotdata

In above code siteurl is fetched simply from local text file.

Why aren't you showing us what is in that local text file? Or more
specifically what siteurl turns out to be? I suspect it's missing the
http:// prefix

IOError: [Errno 2] The system cannot find the path specified: 'www.bestrecipes.c
om.au\\robots.txt'

Looks to me like it decided this url referred to a file. That's the
default behavior when you don't specify the scheme identifier (eg.
'http")

Also it might well have been necessary to specify what Python version
and OS you're running this on. For example, the single backslash
character is specific to Windows. (The doubling presumably is an
artifact of how the error message is displayed, eg. look at how repr()
displays strings)