Forms authentication and search engines

Discussion in 'ASP .Net Security' started by Guy Incognito, Oct 5, 2005.

  1. Hello,

    Can forms authentication in asp.net be set up so that search engines
    like Google can get through?

    My client wants their site to force visitors to register before they can
    read the content, but still wants search engines like Google to index
    the content.

    Is this possible?

    Thanks,
    Jason

    *** Sent via Developersdex http://www.developersdex.com ***
     
    Guy Incognito, Oct 5, 2005
    #1
    1. Advertising

  2. Hello Guy,

    to do what?? index your sensitive data - or why is the area login protected??
    sorry couldn't resist

    no - this is not possible (or you give google a login :)

    ---------------------------------------
    Dominick Baier - DevelopMentor
    http://www.leastprivilege.com

    > Hello,
    >
    > Can forms authentication in asp.net be set up so that search engines
    > like Google can get through?
    >
    > My client wants their site to force visitors to register before they
    > can read the content, but still wants search engines like Google to
    > index the content.
    >
    > Is this possible?
    >
    > Thanks,
    > Jason
    > *** Sent via Developersdex http://www.developersdex.com ***
    >
     
    Dominick Baier [DevelopMentor], Oct 5, 2005
    #2
    1. Advertising

  3. Hi Dominick,

    >> to do what?? index your sensitive data - or why is the area login

    protected??

    There's nothing sensitive about the content. The strategy behind the
    login feature is to count users, and encourage them to sign up for a
    mailing list.

    The idea is similar to some online newspapers, requiring registration
    but not charging a fee. But don't ask me to explain the management
    strategy, I just have to implement it.

    If it can't be done through forms authentication, can anybody suggest
    another way?

    And Dominick, thanks for your advice earlier this week.

    - Jason




    *** Sent via Developersdex http://www.developersdex.com ***
     
    Guy Incognito, Oct 5, 2005
    #3
  4. Hello Guy,

    well - fact is - as long as you are enforcing a login - how can google index
    the content whithout logging in?

    you could make the login optional (technically) but don't expose the direct
    links in your application (i recommend this only because there is no sensitive
    content as you say)

    or create an abstract of the content for each page (whithout the need to
    login) with a "read more" link that requires auth...

    ---------------------------------------
    Dominick Baier - DevelopMentor
    http://www.leastprivilege.com

    > Hi Dominick,
    >
    >>> to do what?? index your sensitive data - or why is the area login
    >>>

    > protected??
    >
    > There's nothing sensitive about the content. The strategy behind the
    > login feature is to count users, and encourage them to sign up for a
    > mailing list.
    >
    > The idea is similar to some online newspapers, requiring registration
    > but not charging a fee. But don't ask me to explain the management
    > strategy, I just have to implement it.
    >
    > If it can't be done through forms authentication, can anybody suggest
    > another way?
    >
    > And Dominick, thanks for your advice earlier this week.
    >
    > - Jason
    >
    > *** Sent via Developersdex http://www.developersdex.com ***
    >
     
    Dominick Baier [DevelopMentor], Oct 5, 2005
    #4
  5. You do understand, that once google is able to index the page, a user can
    simply go to Google's cache, view your page there, and never have to
    register with your website.

    Dominick's suggestion is the best.

    If, however, you still NEED to do this (and I understand client
    requirements), then perhaps you can perform a 'hardcode bypass' by checking
    the 'browser' of the visitor. If it matches Google Bot's header, you can
    send it directly to the confidential pages.

    -Altaf
    [MVP - VB]
    --------------------------------------------------------------------------------
    All that glitters has a high refractive index.
    www.mendhak.com


    "Dominick Baier [DevelopMentor]" <>
    wrote in message news:...
    > Hello Guy,
    >
    > well - fact is - as long as you are enforcing a login - how can google
    > index the content whithout logging in?
    >
    > you could make the login optional (technically) but don't expose the
    > direct links in your application (i recommend this only because there is
    > no sensitive content as you say)
    >
    > or create an abstract of the content for each page (whithout the need to
    > login) with a "read more" link that requires auth...
    >
    > ---------------------------------------
    > Dominick Baier - DevelopMentor
    > http://www.leastprivilege.com
    >
    >> Hi Dominick,
    >>
    >>>> to do what?? index your sensitive data - or why is the area login
    >>>>

    >> protected??
    >>
    >> There's nothing sensitive about the content. The strategy behind the
    >> login feature is to count users, and encourage them to sign up for a
    >> mailing list.
    >>
    >> The idea is similar to some online newspapers, requiring registration
    >> but not charging a fee. But don't ask me to explain the management
    >> strategy, I just have to implement it.
    >>
    >> If it can't be done through forms authentication, can anybody suggest
    >> another way?
    >>
    >> And Dominick, thanks for your advice earlier this week.
    >>
    >> - Jason
    >>
    >> *** Sent via Developersdex http://www.developersdex.com ***
    >>

    >
    >
     
    S.M. Altaf [MVP], Oct 6, 2005
    #5
  6. The latter suggestion would probably work pretty well in practice. You
    would simply write an HttpModule or global.asax handler that ran on
    BeginRequest, checked for the bot header and called SkipAuthorization if it
    was detected.

    An enterprising user could then bypass your forms
    authentication/authorization by including the bot's header in their
    requests, but since IE doesn't make it very easy for you to spoof these
    things, in practice this will repel most users. Since they obviously don't
    really care about security, having a percentage of users bypass
    authentication should not bother the customer. You could even add some
    instrumentation to show what the percentage of users bypassing authorization
    actually is and log those requests.

    The users would be able to bypass authorization at the site by using
    Google's cache. No way around that.

    Joe K.

    "S.M. Altaf [MVP]" <> wrote in message
    news:%...
    > You do understand, that once google is able to index the page, a user can
    > simply go to Google's cache, view your page there, and never have to
    > register with your website.
    >
    > Dominick's suggestion is the best.
    >
    > If, however, you still NEED to do this (and I understand client
    > requirements), then perhaps you can perform a 'hardcode bypass' by
    > checking the 'browser' of the visitor. If it matches Google Bot's header,
    > you can send it directly to the confidential pages.
    >
    > -Altaf
    > [MVP - VB]
    > --------------------------------------------------------------------------------
    > All that glitters has a high refractive index.
    > www.mendhak.com
    >
    >
    > "Dominick Baier [DevelopMentor]" <>
    > wrote in message news:...
    >> Hello Guy,
    >>
    >> well - fact is - as long as you are enforcing a login - how can google
    >> index the content whithout logging in?
    >>
    >> you could make the login optional (technically) but don't expose the
    >> direct links in your application (i recommend this only because there is
    >> no sensitive content as you say)
    >>
    >> or create an abstract of the content for each page (whithout the need to
    >> login) with a "read more" link that requires auth...
    >>
    >> ---------------------------------------
    >> Dominick Baier - DevelopMentor
    >> http://www.leastprivilege.com
    >>
    >>> Hi Dominick,
    >>>
    >>>>> to do what?? index your sensitive data - or why is the area login
    >>>>>
    >>> protected??
    >>>
    >>> There's nothing sensitive about the content. The strategy behind the
    >>> login feature is to count users, and encourage them to sign up for a
    >>> mailing list.
    >>>
    >>> The idea is similar to some online newspapers, requiring registration
    >>> but not charging a fee. But don't ask me to explain the management
    >>> strategy, I just have to implement it.
    >>>
    >>> If it can't be done through forms authentication, can anybody suggest
    >>> another way?
    >>>
    >>> And Dominick, thanks for your advice earlier this week.
    >>>
    >>> - Jason
    >>>
    >>> *** Sent via Developersdex http://www.developersdex.com ***
    >>>

    >>
    >>

    >
    >
     
    Joe Kaplan \(MVP - ADSI\), Oct 6, 2005
    #6
  7. Hello Joe,

    yeah - that would probably work (ouch - did i say that :)

    as long as Google does not change the agent header...


    ---------------------------------------
    Dominick Baier - DevelopMentor
    http://www.leastprivilege.com

    > The latter suggestion would probably work pretty well in practice.
    > You would simply write an HttpModule or global.asax handler that ran
    > on BeginRequest, checked for the bot header and called
    > SkipAuthorization if it was detected.
    >
    > An enterprising user could then bypass your forms
    > authentication/authorization by including the bot's header in their
    > requests, but since IE doesn't make it very easy for you to spoof
    > these things, in practice this will repel most users. Since they
    > obviously don't really care about security, having a percentage of
    > users bypass authentication should not bother the customer. You could
    > even add some instrumentation to show what the percentage of users
    > bypassing authorization actually is and log those requests.
    >
    > The users would be able to bypass authorization at the site by using
    > Google's cache. No way around that.
    >
    > Joe K.
    >
    > "S.M. Altaf [MVP]" <> wrote in message
    > news:%...
    >
    >> You do understand, that once google is able to index the page, a user
    >> can simply go to Google's cache, view your page there, and never have
    >> to register with your website.
    >>
    >> Dominick's suggestion is the best.
    >>
    >> If, however, you still NEED to do this (and I understand client
    >> requirements), then perhaps you can perform a 'hardcode bypass' by
    >> checking the 'browser' of the visitor. If it matches Google Bot's
    >> header, you can send it directly to the confidential pages.
    >>
    >> -Altaf
    >> [MVP - VB]
    >> ---------------------------------------------------------------------
    >> -----------
    >> All that glitters has a high refractive index.
    >> www.mendhak.com
    >> "Dominick Baier [DevelopMentor]"
    >> <> wrote in message
    >> news:...
    >>
    >>> Hello Guy,
    >>>
    >>> well - fact is - as long as you are enforcing a login - how can
    >>> google index the content whithout logging in?
    >>>
    >>> you could make the login optional (technically) but don't expose the
    >>> direct links in your application (i recommend this only because
    >>> there is no sensitive content as you say)
    >>>
    >>> or create an abstract of the content for each page (whithout the
    >>> need to login) with a "read more" link that requires auth...
    >>>
    >>> ---------------------------------------
    >>> Dominick Baier - DevelopMentor
    >>> http://www.leastprivilege.com
    >>>> Hi Dominick,
    >>>>
    >>>>>> to do what?? index your sensitive data - or why is the area login
    >>>>>>
    >>>> protected??
    >>>>
    >>>> There's nothing sensitive about the content. The strategy behind
    >>>> the login feature is to count users, and encourage them to sign up
    >>>> for a mailing list.
    >>>>
    >>>> The idea is similar to some online newspapers, requiring
    >>>> registration but not charging a fee. But don't ask me to explain
    >>>> the management strategy, I just have to implement it.
    >>>>
    >>>> If it can't be done through forms authentication, can anybody
    >>>> suggest another way?
    >>>>
    >>>> And Dominick, thanks for your advice earlier this week.
    >>>>
    >>>> - Jason
    >>>>
    >>>> *** Sent via Developersdex http://www.developersdex.com ***
    >>>>
     
    Dominick Baier [DevelopMentor], Oct 6, 2005
    #7
  8. That could go in a config file, but yeah, it is a little brittle. I
    especially like the bit on reporting metrics on how many people are
    bypassing security. :)

    Joe K.

    "Dominick Baier [DevelopMentor]" <>
    wrote in message news:...
    > Hello Joe,
    >
    > yeah - that would probably work (ouch - did i say that :)
    > as long as Google does not change the agent header...
    >
    >
    > ---------------------------------------
    > Dominick Baier - DevelopMentor
    > http://www.leastprivilege.com
    >
    >> The latter suggestion would probably work pretty well in practice.
    >> You would simply write an HttpModule or global.asax handler that ran
    >> on BeginRequest, checked for the bot header and called
    >> SkipAuthorization if it was detected.
    >>
    >> An enterprising user could then bypass your forms
    >> authentication/authorization by including the bot's header in their
    >> requests, but since IE doesn't make it very easy for you to spoof
    >> these things, in practice this will repel most users. Since they
    >> obviously don't really care about security, having a percentage of
    >> users bypass authentication should not bother the customer. You could
    >> even add some instrumentation to show what the percentage of users
    >> bypassing authorization actually is and log those requests.
    >>
    >> The users would be able to bypass authorization at the site by using
    >> Google's cache. No way around that.
    >>
    >> Joe K.
    >>
    >> "S.M. Altaf [MVP]" <> wrote in message
    >> news:%...
    >>
    >>> You do understand, that once google is able to index the page, a user
    >>> can simply go to Google's cache, view your page there, and never have
    >>> to register with your website.
    >>>
    >>> Dominick's suggestion is the best.
    >>>
    >>> If, however, you still NEED to do this (and I understand client
    >>> requirements), then perhaps you can perform a 'hardcode bypass' by
    >>> checking the 'browser' of the visitor. If it matches Google Bot's
    >>> header, you can send it directly to the confidential pages.
    >>>
    >>> -Altaf
    >>> [MVP - VB]
    >>> ---------------------------------------------------------------------
    >>> -----------
    >>> All that glitters has a high refractive index.
    >>> www.mendhak.com
    >>> "Dominick Baier [DevelopMentor]"
    >>> <> wrote in message
    >>> news:...
    >>>
    >>>> Hello Guy,
    >>>>
    >>>> well - fact is - as long as you are enforcing a login - how can
    >>>> google index the content whithout logging in?
    >>>>
    >>>> you could make the login optional (technically) but don't expose the
    >>>> direct links in your application (i recommend this only because
    >>>> there is no sensitive content as you say)
    >>>>
    >>>> or create an abstract of the content for each page (whithout the
    >>>> need to login) with a "read more" link that requires auth...
    >>>>
    >>>> ---------------------------------------
    >>>> Dominick Baier - DevelopMentor
    >>>> http://www.leastprivilege.com
    >>>>> Hi Dominick,
    >>>>>
    >>>>>>> to do what?? index your sensitive data - or why is the area login
    >>>>>>>
    >>>>> protected??
    >>>>>
    >>>>> There's nothing sensitive about the content. The strategy behind
    >>>>> the login feature is to count users, and encourage them to sign up
    >>>>> for a mailing list.
    >>>>>
    >>>>> The idea is similar to some online newspapers, requiring
    >>>>> registration but not charging a fee. But don't ask me to explain
    >>>>> the management strategy, I just have to implement it.
    >>>>>
    >>>>> If it can't be done through forms authentication, can anybody
    >>>>> suggest another way?
    >>>>>
    >>>>> And Dominick, thanks for your advice earlier this week.
    >>>>>
    >>>>> - Jason
    >>>>>
    >>>>> *** Sent via Developersdex http://www.developersdex.com ***
    >>>>>

    >
    >
     
    Joe Kaplan \(MVP - ADSI\), Oct 6, 2005
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Eric
    Replies:
    2
    Views:
    1,568
    Tommy
    Feb 13, 2004
  2. phl
    Replies:
    0
    Views:
    363
  3. Chris K.
    Replies:
    3
    Views:
    795
    Chris K.
    Jun 8, 2008
  4. Eric
    Replies:
    2
    Views:
    649
  5. SROSeaner

    Meta-Search the Search Engines

    SROSeaner, Sep 15, 2004, in forum: ASP General
    Replies:
    1
    Views:
    164
Loading...

Share This Page