How do I correctly download Wikipedia pages?

[Steven D'Aprano]

I'm trying to scrape a Wikipedia page from Python. Following instructions
here:

http://en.wikipedia.org/wiki/Wikipedia:Database_download
http://en.wikipedia.org/wiki/Special:Export

I use the URL "http://en.wikipedia.org/wiki/Special:Export/Train" instead
of just "http://en.wikipedia.org/wiki/Train". But instead of getting the
page I expect, and can see in my browser, I get an error page:

....
Our servers are currently experiencing a technical problem. This is
probably temporary and should be fixed soon
....


(Output is obviously truncated for your sanity and mine.)


Is there a trick to downloading from Wikipedia with urllib?

 

[ShoqulKutlu]

Hi,

Try not to be caught if you send multiple requests

Have a look at here: http://wolfprojects.altervista.org/changeua.php

Regards
Kutlu

 

[Steven D'Aprano]

Hi,

Try not to be caught if you send multiple requests

Have a look at here: http://wolfprojects.altervista.org/changeua.php

Thanks, that seems to work perfectly.

 

[Cousin Stanley]

I'm trying to scrape a Wikipedia page from Python.
....

On occasion I use a program under Debian Linux
called wikipedia2text that is very handy
for downloading wikipedia pages as plain text files ....

Description: displays Wikipedia articles on the command line

This script fetches Wikipedia articles (currently supports
around 30 Wikipedia languages) and displays them as plain text
in a pager or just sends the text to standard out. Alternatively
it opens the Wikipedia article in a (possibly GUI) web browser
or just shows the URL of the appropriate Wikipedia article.

Example directed through the lynx browser ....

wp2t -b lynx gorilla > gorilla.txt