get wikipedia source failed (urrlib2)

[shahargs]

Hi,
I'm trying to get wikipedia page source with urllib2:
usock = urllib2.urlopen("http://en.wikipedia.org/wiki/
Albert_Einstein")
data = usock.read();
usock.close();
return data
I got exception because HTTP 403 error. why? with my browser i can't
access it without any problem?

Thanks,
Shahar.

 

[shahargs]

Hi,
I'm trying to get wikipedia page source with urllib2:
usock = urllib2.urlopen("http://en.wikipedia.org/wiki/
Albert_Einstein")
data = usock.read();
usock.close();
return data
I got exception because HTTP 403 error. why? with my browser i can't
access it without any problem?

Thanks,
Shahar.

This source works fine for other site. the problem is in wikipedia. is
someone now any solution for this problem?

 

[Lawrence Oluyede]

This source works fine for other site. the problem is in wikipedia. is
someone now any solution for this problem?

Wikipedia, AFAIK, bans requests without a User Agent.
http://www.voidspace.org.uk/python/articles/urllib2.shtml#headers

 

[Michael J‭. ‬Fromberger]

‭ ‬[email protected] wrote‭:‬

‭> ‬Hi‭,‬
‭> ‬I'm trying to get wikipedia page source with urllib2‭:‬
‭> ‬usock‭ = ‬urllib2‭.‬urlopen‭("‬http‭://‬en.wikipedia.org/wiki‭/‬
‭> ‬Albert_Einstein‭")‬
‭> ‬data‭ = ‬usock.read‭();‬
‭> ‬usock.close‭();‬
‭> ‬return data
‭> ‬I got exception because HTTP 403‭ ‬error‭. ‬why‭? ‬with my browser i can't
‭> ‬access it without any problem‭?‬
‭> ‬
‭> ‬Thanks‭,‬
‭> ‬Shahar‭.‬

It appears that Wikipedia may inspect the contents of the User-Agent‭ ‬
HTTP header‭, ‬and that it does not particularly like the string it‭ ‬
receives from Python's urllib‭. ‬I was able to make it work with urllib‭ ‬
via the following code‭:‬

import urllib

class CustomURLopener‭ (‬urllib.FancyURLopener‭):‬
‭ ‬version‭ = '‬Mozilla/5.0‭'‬

urllib‭.‬_urlopener‭ = ‬CustomURLopener‭()‬

u‭ = ‬urllib.urlopen‭('‬http‭://‬en.wikipedia.org/wiki/Albert_Einstein‭')‬
data‭ = ‬u.read‭()‬

I'm assuming a similar trick could be used with urllib2‭, ‬though I didn't‭ ‬
actually try it‭. ‬Another thing to watch out for‭, ‬is that some sites‭ ‬
will redirect a public URL X to an internal URL Y‭, ‬and will check that‭ ‬
access to Y is only permitted if the Referer field indicates coming from‭ ‬
somewhere internal to the site‭. ‬I have seen both of these techniques‭ ‬
used to foil screen-scraping‭.‬

Cheers‭,‬
‭-‬M

‭-- ‬
Michael J‭. ‬Fromberger‭ | ‬Lecturer‭, ‬Dept‭. ‬of Computer Science
http‭://‬www.dartmouth.edu‭/‬~sting‭/ | ‬Dartmouth College‭, ‬Hanover‭, ‬NH‭, ‬USA