404, SEO, robot.txt?

[David Lozzi]

Howdy,

A lot of my sites are data driven pages pulling from SQL using url
rewriting. Works great. some sites that i have changed are getting hits on
the old pages that no longer exist from search engine bots. I have my site
setup so that if someone tries to get to a page that isn't in the database i
redirect them to a friendly page, oops.aspx. Unfortunately some search
engines are cataloging this page as the content of the original pages, so it
thinks engineering.aspx is oops.aspx. What is the best way to handle this? I
could add oops.aspx to robot.txt and technically the search engine won't
catalog it. Can I someone send the bot a 404 code so that it doesn't look
for this page anymore? My other option is to manually enter in all of the
pages being hit into my urlrewriting, but i'm hoping not to.

Thanks!

 

[bruce barker]

using the user agent string you can try to identify the request a bot
request, then respond with a 404 instead of 301.

-- bruce (sqlwork.com)

 

[David Lozzi]

how does one respond with a 404?