note: following these bot rules is strictly up to the bot. the major engines
are polite and do, but not all.
-- bruce (sqlwork.com)
| You can check the Request.UserAgent property. Each major search bot
| uses a different user agent, so you'd need to check against a list. The
| biggest ones are googlebot, msnbot, and slurp (Yahoo).
|
| Here's a partial list:
|
http://webmaster-resources.web-mastery.net/spider-identification.html
|
| If you just want to prevent search engines from spidering all or
| portions of your site, you can do this with a robots.txt file at the
| root of your site:
|
http://www.searchengineworld.com/robots/robots_tutorial.htm
|
| Keep in mind that it is possible for users can change their user agent
| (for instance via the Firefox User Agent Switcher extension), so if you
| have pages of your site that you want to allow only subscribers or
| search engines, you'd need to check against an IP whitelist to keep
| people out.
|
| - Jon
|
http://weblogs.asp.net/jgalloway
|