Ping Aaron - [Rel: Spiders]

[Steven Burn]

Aaron,
Apologies for the ping but, your the only one I can think of offhand that
will know what I'm looking for.

Basically, to cut a long story short, I run an sURL service (surl.co.uk)
that allows one to convert long URL's to short etc, and am looking to have a
search facility on it that will allow one to locate an sURL for a specific
subject.

Now, the problem I'm running into is, I'd like to have some sort of spider
that will go to each site in the database, and grab the title, keywords, and
description from the meta tags, As I've never written a spider in my life,
and have been unsuccessful in locating info on how to go about doing so, I
was hoping you could point me in the right direction?

--

Regards

Steven Burn
Ur I.T. Mate Group
www.it-mate.co.uk

Keeping it FREE!

 

[Aaron [SQL Server MVP]]

One way is to use ServerXMLHTTP (see http://www.aspfaq.com/2173). Once you
have the response text back, you can parse it for <title> and </title>, etc.

 

[Steven Burn]

Cheers Aaron..... I had a feeling that was going to be the response (I've
got a page on my site that uses it (checks whether a site is online or not),
but it's terribly slow for some reason)

--

Regards

Steven Burn
Ur I.T. Mate Group
www.it-mate.co.uk

Keeping it FREE!