Search engines crawling our .NET site

Discussion in 'ASP .Net' started by Mark, Mar 4, 2005.

  1. Mark

    Mark Guest

    Our site gets searched by robots all the time. This is great. However,
    many of our pages that we want to be cataloged are data driven, so we end up
    with pages like:

    www.ourdomain.com/products.aspx?productid=356

    Let's assume that we stop selling productid 356. This means that the url
    above is invalid. If a general user has bookmarked this page or pastes in a
    url into a browser that isn't quite right, we want them to get a 'pretty'
    error message. However, using this approach means that a search engine like
    google will interpret this page as still being "OK" and will continue to
    catalog it.

    Off the top of my head, I see two solutions:

    1. Redirect to a bogus page that doesn't exist, so a 404 message displays.
    Search engines would remove the link in their catalog(hopefully????), but
    users would not have a "pleasing" experience.
    2. Throw an unhandled exception. A ugly ASP.NET error message page is
    displayed. This ticks off the user, but hopefully the search engine
    realizes that the page does not actually exist? Probably not.

    Suggestions on how to handle this?

    Thanks in advance.

    Mark
    Mark, Mar 4, 2005
    #1
    1. Advertising

  2. Option 3:
    Don't worry about it and send the consumer to a page that says "we no longer
    carry that product, but suggest this instead" and keep old products in a
    "dead product" table with cross sell possibilities.


    ---

    Gregory A. Beamer
    MVP; MCP: +I, SE, SD, DBA

    ***************************
    Think Outside the Box!
    ***************************

    "Mark" wrote:

    > Our site gets searched by robots all the time. This is great. However,
    > many of our pages that we want to be cataloged are data driven, so we end up
    > with pages like:
    >
    > www.ourdomain.com/products.aspx?productid=356
    >
    > Let's assume that we stop selling productid 356. This means that the url
    > above is invalid. If a general user has bookmarked this page or pastes in a
    > url into a browser that isn't quite right, we want them to get a 'pretty'
    > error message. However, using this approach means that a search engine like
    > google will interpret this page as still being "OK" and will continue to
    > catalog it.
    >
    > Off the top of my head, I see two solutions:
    >
    > 1. Redirect to a bogus page that doesn't exist, so a 404 message displays.
    > Search engines would remove the link in their catalog(hopefully????), but
    > users would not have a "pleasing" experience.
    > 2. Throw an unhandled exception. A ugly ASP.NET error message page is
    > displayed. This ticks off the user, but hopefully the search engine
    > realizes that the page does not actually exist? Probably not.
    >
    > Suggestions on how to handle this?
    >
    > Thanks in advance.
    >
    > Mark
    >
    >
    >
    =?Utf-8?B?Q293Ym95IChHcmVnb3J5IEEuIEJlYW1lcikgLSBN, Mar 4, 2005
    #2
    1. Advertising

  3. Mark

    bruce barker Guest

    most search robots support the robot headers, just include on your friendly
    page:

    <META name="ROBOTS" content="NOINDEX">

    -- bruce (sqlwork.com)


    "Mark" <> wrote in message
    news:...
    | Our site gets searched by robots all the time. This is great. However,
    | many of our pages that we want to be cataloged are data driven, so we end
    up
    | with pages like:
    |
    | www.ourdomain.com/products.aspx?productid=356
    |
    | Let's assume that we stop selling productid 356. This means that the url
    | above is invalid. If a general user has bookmarked this page or pastes in
    a
    | url into a browser that isn't quite right, we want them to get a 'pretty'
    | error message. However, using this approach means that a search engine
    like
    | google will interpret this page as still being "OK" and will continue to
    | catalog it.
    |
    | Off the top of my head, I see two solutions:
    |
    | 1. Redirect to a bogus page that doesn't exist, so a 404 message displays.
    | Search engines would remove the link in their catalog(hopefully????), but
    | users would not have a "pleasing" experience.
    | 2. Throw an unhandled exception. A ugly ASP.NET error message page is
    | displayed. This ticks off the user, but hopefully the search engine
    | realizes that the page does not actually exist? Probably not.
    |
    | Suggestions on how to handle this?
    |
    | Thanks in advance.
    |
    | Mark
    |
    |
    bruce barker, Mar 4, 2005
    #3
  4. Mark

    fd123456 Guest

    Hi Mark,

    You could also place a "robots.txt" text file in the root directory of
    your app, containing this :

    User-agent:*
    Disallow:products.aspx

    Although I find Gregory's solution very elegant, user-friendly and
    commercialy sound. I'd go that way. Out of the box...

    HTH,

    Michel

    "Mark" <> wrote in message news:<>...
    > Our site gets searched by robots all the time. This is great. However,
    > many of our pages that we want to be cataloged are data driven, so we end up
    > with pages like:
    >
    > www.ourdomain.com/products.aspx?productid=356
    >
    > Let's assume that we stop selling productid 356. This means that the url
    > above is invalid. If a general user has bookmarked this page or pastes in a
    > url into a browser that isn't quite right, we want them to get a 'pretty'
    > error message. However, using this approach means that a search engine like
    > google will interpret this page as still being "OK" and will continue to
    > catalog it.
    >
    > Off the top of my head, I see two solutions:
    >
    > 1. Redirect to a bogus page that doesn't exist, so a 404 message displays.
    > Search engines would remove the link in their catalog(hopefully????), but
    > users would not have a "pleasing" experience.
    > 2. Throw an unhandled exception. A ugly ASP.NET error message page is
    > displayed. This ticks off the user, but hopefully the search engine
    > realizes that the page does not actually exist? Probably not.
    >
    > Suggestions on how to handle this?
    >
    > Thanks in advance.
    >
    > Mark
    fd123456, Mar 7, 2005
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Cowboy \(Gregory A. Beamer\) [MVP]

    Need to get my site listed with search engines, where do I start?

    Cowboy \(Gregory A. Beamer\) [MVP], Jun 8, 2004, in forum: ASP .Net
    Replies:
    3
    Views:
    294
    Charlie@CBFC
    Jun 8, 2004
  2. Miguel Dias Moura

    Web Site & Search Engines

    Miguel Dias Moura, Jul 4, 2004, in forum: ASP .Net
    Replies:
    3
    Views:
    299
    Miguel Dias Moura
    Jul 6, 2004
  3. phl
    Replies:
    0
    Views:
    340
  4. Chris K.
    Replies:
    3
    Views:
    726
    Chris K.
    Jun 8, 2008
  5. SROSeaner

    Meta-Search the Search Engines

    SROSeaner, Sep 15, 2004, in forum: ASP General
    Replies:
    1
    Views:
    135
Loading...

Share This Page