trying to collect links on my site, how to crawl it?

Discussion in 'ASP General' started by Danny, May 27, 2004.

  1. Danny

    Danny Guest

    I am trying to crawl my site to get a list of links. I am using the regular
    expressions to get the href tags from the pages and reading the links using
    xmlhttp module.

    is there an efficient way to loop through the links? If you start with the
    home page, how do you do it? How do yo ukeep track of the pages you have
    been to and which to do next.

    I am not sure how to go about tihs.

    I tried some loop but it seemed to take too long to crawl my site.

    Thanks again
    Danny, May 27, 2004
    #1
    1. Advertising

  2. For all .asp and .html files etc., you could build the list of files to hit
    using FileSystemObject... store the list in a database or text file, and
    scroll through it...

    --
    Aaron Bertrand
    SQL Server MVP
    http://www.aspfaq.com/


    "Danny" <> wrote in message
    news:Wlctc.55736$...
    >I am trying to crawl my site to get a list of links. I am using the
    >regular
    > expressions to get the href tags from the pages and reading the links
    > using
    > xmlhttp module.
    >
    > is there an efficient way to loop through the links? If you start with
    > the
    > home page, how do you do it? How do yo ukeep track of the pages you have
    > been to and which to do next.
    >
    > I am not sure how to go about tihs.
    >
    > I tried some loop but it seemed to take too long to crawl my site.
    >
    > Thanks again
    >
    >
    Aaron Bertrand [MVP], May 27, 2004
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John

    Can Google crawl .aspx pages?

    John, Jan 31, 2007, in forum: ASP .Net
    Replies:
    2
    Views:
    433
    bruce barker
    Jan 31, 2007
  2. JIM.H.
    Replies:
    9
    Views:
    402
    Alexey Smirnov
    Feb 10, 2009
  3. Justin
    Replies:
    1
    Views:
    307
    Stefan Behnel
    Aug 21, 2009
  4. Steve Mauldin
    Replies:
    0
    Views:
    125
    Steve Mauldin
    Feb 6, 2004
  5. Dave

    Crawl a site

    Dave, Jun 23, 2004, in forum: ASP General
    Replies:
    1
    Views:
    110
    Eric Gibson
    Jun 27, 2004
Loading...

Share This Page