web crawling.

[S Borg]

Hello,

I have been writing very simple Python programs that parse HTML and
such, mainly just to get
a better feel for the language. Here is my question: If I parsed an
HTML page into all of the image
files listed on that page, how could I request all of those images and
download them into some specified folder? I am sure this is quite easy,
but I am stuck.

Thank you very much.
Burgeoning Pythonista

 

[Alex Martelli]

Hello,

I have been writing very simple Python programs that parse HTML and
such, mainly just to get
a better feel for the language. Here is my question: If I parsed an
HTML page into all of the image
files listed on that page, how could I request all of those images and
download them into some specified folder? I am sure this is quite easy,
but I am stuck.

There's a good crawler in the Demo directory of the Python source
distribution, so download and unpack said sources and look there.


Alex

 

[gene tani]

Hello,

I have been writing very simple Python programs that parse HTML and
such, mainly just to get
a better feel for the language. Here is my question: If I parsed an
HTML page into all of the image
files listed on that page, how could I request all of those images and
download them into some specified folder? I am sure this is quite easy,
but I am stuck.

Thank you very much.
Burgeoning Pythonista

http://sig.levillage.org/?p=588

 

[Fuzzyman]

Use BeautifulSoup to get all the image tags out of the html.

You'll need to join the urls of the images to the url of the page
(urlparse.urljoin off the top of my head). If you look at BeautifulSoup
you will see how to get the 'src' reference of each image tag.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

 

[John M. Gabriele]

There's a good crawler in the Demo directory of the Python source
distribution, so download and unpack said sources and look there.


Alex

Hm. Looks like that's:

Python-2.4.2/Tools/webchecker

See 'pydoc ./webchecker.py' for more info.

---J