spidering script

[David Waizer]

Hello..

I'm looking for a script (perl, python, sh...)or program (such as wget)
that will help me get a list of ALL the links on a website.

For example ./magicscript.pl www.yahoo.com and outputs it to a file, it
would be kind of like a spidering software..

Any suggestions would be appreciated.

David

 

[Jonathan Curran]

Hello..

I'm looking for a script (perl, python, sh...)or program (such as wget)
that will help me get a list of ALL the links on a website.

For example ./magicscript.pl www.yahoo.com and outputs it to a file, it
would be kind of like a spidering software..

Any suggestions would be appreciated.

David

David, this is a touchy topic but whatever Look into sgmllib, and you can
filter on the "A" tag. The book 'Dive Into Python' covers it quite nicely:
http://www.diveintopython.org/html_processing/index.html

Jonathan

 

[dubs]

Check out the quick start section in the documentation at Beautiful
Soup http://www.crummy.com/software/BeautifulSoup/


Wes

 

[Bernard]

4 easy steps to get the links:

1. Download BeautifulSoup and import it in your script file.
2. Use urllib2 to download the html of the url.
3. mash the html using BeautifulSoup
4.

for tag in BeautifulSoupisedHTML.findAll('a'):
        print tag

David Waizer a écrit :

 

[Nikita the Spider]

Hello..

I'm looking for a script (perl, python, sh...)or program (such as wget)
that will help me get a list of ALL the links on a website.

For example ./magicscript.pl www.yahoo.com and outputs it to a file, it
would be kind of like a spidering software..

David,
In addition to others' suggestions about Beautiful Soup, you might also
want to look at the HTMLData module:

http://oregonstate.edu/~barnesc/htmldata/

 

[William Park]

Hello..

I'm looking for a script (perl, python, sh...)or program (such as wget)
that will help me get a list of ALL the links on a website.

lynx -dump (look at the bottom)

--
William Park <[email protected]>, Toronto, Canada
ThinFlash: Linux thin-client on USB key (flash) drive
http://home.eol.ca/~parkw/thinflash.html
BashDiff: Super Bash shell
http://freshmeat.net/projects/bashdiff/