trying to collect links on my site, how to crawl it?

[Danny]

I am trying to crawl my site to get a list of links. I am using the regular
expressions to get the href tags from the pages and reading the links using
xmlhttp module.

is there an efficient way to loop through the links? If you start with the
home page, how do you do it? How do yo ukeep track of the pages you have
been to and which to do next.

I am not sure how to go about tihs.

I tried some loop but it seemed to take too long to crawl my site.

Thanks again

 

[Aaron Bertrand [MVP]]

For all .asp and .html files etc., you could build the list of files to hit
using FileSystemObject... store the list in a database or text file, and
scroll through it...