Using Request.Browser.Crawler - is it reliable?

Discussion in 'ASP .Net' started by Bill, Dec 13, 2005.

  1. Bill

    Bill Guest

    Has anyone used/tested Request.Browser.Crawler ? Is it reliable, or are there false
    positives/negatives?

    Thanks!
     
    Bill, Dec 13, 2005
    #1
    1. Advertising

  2. Bill

    Karl Seguin Guest

    It's based on the browserCaps, which you can override in the web.config.
    Otherwise it uses the built-in one found in the machine.config file
    (C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\CONFIG)

    If you do a search for "browserCaps" and look for "crawler" in that section,
    you'll see what sets it off.

    Obviously it can easily be fooled, anyone can simply change the header
    information. I think for the most part however, you won't get false
    positives (Crawler == true when it isn't), but you will get false negatives
    (crawler == false when it is).

    Karl

    --
    MY ASP.Net tutorials
    http://www.openmymind.net/


    "Bill" <> wrote in message
    news:...
    > Has anyone used/tested Request.Browser.Crawler ? Is it reliable, or are
    > there false
    > positives/negatives?
    >
    > Thanks!
    >
    >
    >
     
    Karl Seguin, Dec 13, 2005
    #2
    1. Advertising

  3. Bill

    Bill Guest

    Cool - thanks!

    Follow up - how can I use, or access, "Request.Browser.Crawler" in a classical ASP page?

    I need to detect if a browser is a crawler so that I do not update the page count. Of
    course, because of the nature of crawler beasties, I can not do a redirect. And the
    application is too extensive for me to do a rewrite at this time.

    Thanks,

    Bill.



    "Karl Seguin" wrote...
    > It's based on the browserCaps, which you can override in the web.config.
    > Otherwise it uses the built-in one found in the machine.config file
    > (C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\CONFIG)
    >
    > If you do a search for "browserCaps" and look for "crawler" in that section,
    > you'll see what sets it off.
    >
    > Obviously it can easily be fooled, anyone can simply change the header
    > information. I think for the most part however, you won't get false
    > positives (Crawler == true when it isn't), but you will get false negatives
    > (crawler == false when it is).
    >
    > Karl
    >
    > --
    > MY ASP.Net tutorials
    > http://www.openmymind.net/
    >
    >
    > "Bill" wrote...
    > > Has anyone used/tested Request.Browser.Crawler ? Is it reliable, or are
    > > there false
    > > positives/negatives?
    > >
    > > Thanks!
    > >
    > >
    > >

    >
    >
     
    Bill, Dec 13, 2005
    #3
  4. Bill

    Karl Seguin Guest

    The only way I know of is to either (a) parse the header itself, or (b) use
    a 3rd party component like BrowserHawk
    (https://www.cyscape.com/products/bhawk/)

    Karl

    --

    MY ASP.Net tutorials
    http://www.openmymind.net/



    "Bill" <> wrote in message
    news:...
    > Cool - thanks!
    >
    > Follow up - how can I use, or access, "Request.Browser.Crawler" in a
    > classical ASP page?
    >
    > I need to detect if a browser is a crawler so that I do not update the
    > page count. Of
    > course, because of the nature of crawler beasties, I can not do a
    > redirect. And the
    > application is too extensive for me to do a rewrite at this time.
    >
    > Thanks,
    >
    > Bill.
    >
    >
    >
    > "Karl Seguin" wrote...
    >> It's based on the browserCaps, which you can override in the web.config.
    >> Otherwise it uses the built-in one found in the machine.config file
    >> (C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\CONFIG)
    >>
    >> If you do a search for "browserCaps" and look for "crawler" in that
    >> section,
    >> you'll see what sets it off.
    >>
    >> Obviously it can easily be fooled, anyone can simply change the header
    >> information. I think for the most part however, you won't get false
    >> positives (Crawler == true when it isn't), but you will get false
    >> negatives
    >> (crawler == false when it is).
    >>
    >> Karl
    >>
    >> --
    >> MY ASP.Net tutorials
    >> http://www.openmymind.net/
    >>
    >>
    >> "Bill" wrote...
    >> > Has anyone used/tested Request.Browser.Crawler ? Is it reliable, or are
    >> > there false
    >> > positives/negatives?
    >> >
    >> > Thanks!
    >> >
    >> >
    >> >

    >>
    >>

    >
    >
     
    Karl Seguin, Dec 13, 2005
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Nicolas
    Replies:
    0
    Views:
    358
    Nicolas
    Jan 28, 2005
  2. Paul Morrison

    Web Crawler

    Paul Morrison, Oct 17, 2005, in forum: Java
    Replies:
    3
    Views:
    4,923
    lamantpirate
    Jun 30, 2012
  3. HTML crawler/parser

    , Sep 14, 2005, in forum: HTML
    Replies:
    1
    Views:
    458
    David Dorward
    Sep 14, 2005
  4. gmax2006
    Replies:
    1
    Views:
    450
    Andy Dingley
    Jul 18, 2006
  5. Charles Zhang

    Browser.Crawler

    Charles Zhang, Oct 1, 2006, in forum: ASP .Net
    Replies:
    3
    Views:
    703
    Justin Huang
    Oct 4, 2006
Loading...

Share This Page