Preventing Loops

Discussion in 'ASP .Net' started by Brad Baker, Jan 8, 2007.

  1. Brad Baker

    Brad Baker Guest

    We have a custom 404 error page setup so that when a 404 is generated the
    user gets sent to redirect.asp. Redirect.asp sends a user back to
    default.asp (or any other page specified)

    One problem we have been experiencing is with poorly written webbot/spiders
    getting hung up as follows:

    bot indexes default.asp, gets a bad link, generates a 404
    IIS sends bot to redirect.asp
    redirect.asp sends bot back to default asp
    bot reindexes links on default.asp until it gets to the bad link, generates
    a 404, etc etc.
    Loop forever

    I'm looking for recommendations on how we can enhance redirect.asp to detect
    loop conditions such as this. It goes without saying that we are trying to
    come up with an approach that is semi-easy to implement and doesn't require
    a lot of overhead.

    Any thoughts would be greatly appreciated.

    Thanks,
    Brad
     
    Brad Baker, Jan 8, 2007
    #1
    1. Advertising

  2. On Mon, 8 Jan 2007 11:08:44 -0500, Brad Baker wrote:

    > We have a custom 404 error page setup so that when a 404 is generated the
    > user gets sent to redirect.asp. Redirect.asp sends a user back to
    > default.asp (or any other page specified)
    >
    > One problem we have been experiencing is with poorly written webbot/spiders
    > getting hung up as follows:
    >
    > bot indexes default.asp, gets a bad link, generates a 404
    > IIS sends bot to redirect.asp
    > redirect.asp sends bot back to default asp
    > bot reindexes links on default.asp until it gets to the bad link, generates
    > a 404, etc etc.
    > Loop forever
    >
    > I'm looking for recommendations on how we can enhance redirect.asp to detect
    > loop conditions such as this. It goes without saying that we are trying to
    > come up with an approach that is semi-easy to implement and doesn't require
    > a lot of overhead.
    >
    > Any thoughts would be greatly appreciated.
    >
    > Thanks,
    > Brad


    Hey Brad,

    In your 404 asp page you could check the user agent. Most crawlers/spiders
    usually have a descriptive user agent name. You can use that for your
    conditional logic

    --
    Bits.Bytes
    http://bytes.thinkersroom.com
     
    Rad [Visual C# MVP], Jan 8, 2007
    #2
    1. Advertising

  3. Hi Brad,

    I agree with Rad. For your scenario, you need to define an exit condition
    to end such recursive redirection. And for web crawler or other robot, one
    thing to distinguish them from normal web clients is the "UserAgent" http
    header. Different web client(browsers or crawlers will set different value
    for this field). Here is web article describing detect browser in ASP.NET

    http://aspnet.4guysfromrolla.com/articles/120402-1.aspx

    in classic ASP, you can directly check the UserAgent field from
    Request.Headers collection.

    Hope this also helps.

    Sincerely,

    Steven Cheng

    Microsoft MSDN Online Support Lead


    This posting is provided "AS IS" with no warranties, and confers no rights.
     
    Steven Cheng[MSFT], Jan 9, 2007
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Alvin Bruney

    Re: Preventing user from clicking Back?

    Alvin Bruney, Jun 26, 2003, in forum: ASP .Net
    Replies:
    0
    Views:
    459
    Alvin Bruney
    Jun 26, 2003
  2. John
    Replies:
    0
    Views:
    331
  3. Volkan Karaboða

    preventing go back and next page

    Volkan Karaboða, Aug 5, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    360
  4. Replies:
    2
    Views:
    486
    Jerry III
    Aug 11, 2003
  5. Me
    Replies:
    2
    Views:
    246
Loading...

Share This Page