Obtaining Webpage Source with Python

[Ryan Kaskel]

How can I obtain the source of a remote webpage (e.g.
http://www.python.org/index.html) using Python?

Something like:

pyPage = open('http://www.python.org/index.html',r).read()

Obviously that won't work but how can I do something to that effect?
Thanks,
Ryan Kaskel

--I posed this before but it seems it is not showing up...

 

[Paul Rubin]

Something like:

pyPage = open('http://www.python.org/index.html',r).read()

Obviously that won't work but how can I do something to that effect?

import urllib
pyPage = urllib.urlopen('http://www.python.org/index.html',r).read()

 

[Paul Rubin]

import urllib
pyPage = urllib.urlopen('http://www.python.org/index.html',r).read()

oops:

import urllib
pyPage = urllib.urlopen('http://www.python.org/index.html').read()

i.e. omit the 'r' argument to urlib.urlopen.

 

[=?iso-8859-15?Q?Pierre-Fr=E9d=E9ric_Caillaud?=]

pyPage = open('http://www.python.org/index.html',r).read()

using open() for local files and ORLs is called url-fopen and works in
PHP, which is a major security hole, because it even allows one to
include() code files from the web without knowing it, that kind of thing...

python has two functions so you know what you're doing.

If your webpage needs cookies or something, you'll need urllib2

If you wanna parse it afterwards use Htmllib or BeautifulSoup

 

[Phil Frost]

Take a look at the urllib module:

http://python.org/doc/2.3.3/lib/module-urllib.html