Using Request.Browser.Crawler - is it reliable?

[Bill]

Has anyone used/tested Request.Browser.Crawler ? Is it reliable, or are there false
positives/negatives?

Thanks!

 

[Karl Seguin]

It's based on the browserCaps, which you can override in the web.config.
Otherwise it uses the built-in one found in the machine.config file
(C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\CONFIG)

If you do a search for "browserCaps" and look for "crawler" in that section,
you'll see what sets it off.

Obviously it can easily be fooled, anyone can simply change the header
information. I think for the most part however, you won't get false
positives (Crawler == true when it isn't), but you will get false negatives
(crawler == false when it is).

Karl

 

[Bill]

Cool - thanks!

Follow up - how can I use, or access, "Request.Browser.Crawler" in a classical ASP page?

I need to detect if a browser is a crawler so that I do not update the page count. Of
course, because of the nature of crawler beasties, I can not do a redirect. And the
application is too extensive for me to do a rewrite at this time.

Thanks,

Bill.

 

[Karl Seguin]

The only way I know of is to either (a) parse the header itself, or (b) use
a 3rd party component like BrowserHawk
(https://www.cyscape.com/products/bhawk/)

Karl