How can I tell if a visitor is a search engine bot?

J

jongalloway

You can check the Request.UserAgent property. Each major search bot
uses a different user agent, so you'd need to check against a list. The
biggest ones are googlebot, msnbot, and slurp (Yahoo).

Here's a partial list:
http://webmaster-resources.web-mastery.net/spider-identification.html

If you just want to prevent search engines from spidering all or
portions of your site, you can do this with a robots.txt file at the
root of your site:
http://www.searchengineworld.com/robots/robots_tutorial.htm

Keep in mind that it is possible for users can change their user agent
(for instance via the Firefox User Agent Switcher extension), so if you
have pages of your site that you want to allow only subscribers or
search engines, you'd need to check against an IP whitelist to keep
people out.

- Jon
http://weblogs.asp.net/jgalloway
 
M

Mark

Hi, I am writing some code to log vistors to my website.

I notice in some statistics pacakages that they are able to differentiate
between normal visitors and search bots. Does anyone know how that is done?

I thought that they might be looking at the IP address of the remote visitor
and then if the IP range is associated with Google or whatever than that
"person" is a bot...

Any ideas welcome
thanks
Mark
 
B

bruce barker

note: following these bot rules is strictly up to the bot. the major engines
are polite and do, but not all.

-- bruce (sqlwork.com)

| You can check the Request.UserAgent property. Each major search bot
| uses a different user agent, so you'd need to check against a list. The
| biggest ones are googlebot, msnbot, and slurp (Yahoo).
|
| Here's a partial list:
| http://webmaster-resources.web-mastery.net/spider-identification.html
|
| If you just want to prevent search engines from spidering all or
| portions of your site, you can do this with a robots.txt file at the
| root of your site:
| http://www.searchengineworld.com/robots/robots_tutorial.htm
|
| Keep in mind that it is possible for users can change their user agent
| (for instance via the Firefox User Agent Switcher extension), so if you
| have pages of your site that you want to allow only subscribers or
| search engines, you'd need to check against an IP whitelist to keep
| people out.
|
| - Jon
| http://weblogs.asp.net/jgalloway
|
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top