Forms authentication and search engines

[Guy Incognito]

Hello,

Can forms authentication in asp.net be set up so that search engines
like Google can get through?

My client wants their site to force visitors to register before they can
read the content, but still wants search engines like Google to index
the content.

Is this possible?

Thanks,
Jason

 

[Dominick Baier [DevelopMentor]]

Hello Guy,

to do what?? index your sensitive data - or why is the area login protected??
sorry couldn't resist

no - this is not possible (or you give google a login

 

[Guy Incognito]

Hi Dominick,
protected??

There's nothing sensitive about the content. The strategy behind the
login feature is to count users, and encourage them to sign up for a
mailing list.

The idea is similar to some online newspapers, requiring registration
but not charging a fee. But don't ask me to explain the management
strategy, I just have to implement it.

If it can't be done through forms authentication, can anybody suggest
another way?

And Dominick, thanks for your advice earlier this week.

- Jason

 

[Dominick Baier [DevelopMentor]]

Hello Guy,

well - fact is - as long as you are enforcing a login - how can google index
the content whithout logging in?

you could make the login optional (technically) but don't expose the direct
links in your application (i recommend this only because there is no sensitive
content as you say)

or create an abstract of the content for each page (whithout the need to
login) with a "read more" link that requires auth...

 

[S.M. Altaf [MVP]]

You do understand, that once google is able to index the page, a user can
simply go to Google's cache, view your page there, and never have to
register with your website.

Dominick's suggestion is the best.

If, however, you still NEED to do this (and I understand client
requirements), then perhaps you can perform a 'hardcode bypass' by checking
the 'browser' of the visitor. If it matches Google Bot's header, you can
send it directly to the confidential pages.

-Altaf
[MVP - VB]

 

[Joe Kaplan \(MVP - ADSI\)]

The latter suggestion would probably work pretty well in practice. You
would simply write an HttpModule or global.asax handler that ran on
BeginRequest, checked for the bot header and called SkipAuthorization if it
was detected.

An enterprising user could then bypass your forms
authentication/authorization by including the bot's header in their
requests, but since IE doesn't make it very easy for you to spoof these
things, in practice this will repel most users. Since they obviously don't
really care about security, having a percentage of users bypass
authentication should not bother the customer. You could even add some
instrumentation to show what the percentage of users bypassing authorization
actually is and log those requests.

The users would be able to bypass authorization at the site by using
Google's cache. No way around that.

Joe K.

You do understand, that once google is able to index the page, a user can
simply go to Google's cache, view your page there, and never have to
register with your website.

Dominick's suggestion is the best.

If, however, you still NEED to do this (and I understand client
requirements), then perhaps you can perform a 'hardcode bypass' by
checking the 'browser' of the visitor. If it matches Google Bot's header,
you can send it directly to the confidential pages.

-Altaf
[MVP - VB]
--------------------------------------------------------------------------------
All that glitters has a high refractive index.
www.mendhak.com

Hello Guy,

well - fact is - as long as you are enforcing a login - how can google
index the content whithout logging in?

you could make the login optional (technically) but don't expose the
direct links in your application (i recommend this only because there is
no sensitive content as you say)

or create an abstract of the content for each page (whithout the need to
login) with a "read more" link that requires auth...

 

[Dominick Baier [DevelopMentor]]

Hello Joe,

yeah - that would probably work (ouch - did i say that

as long as Google does not change the agent header...


---------------------------------------
Dominick Baier - DevelopMentor
http://www.leastprivilege.com

The latter suggestion would probably work pretty well in practice.
You would simply write an HttpModule or global.asax handler that ran
on BeginRequest, checked for the bot header and called
SkipAuthorization if it was detected.

An enterprising user could then bypass your forms
authentication/authorization by including the bot's header in their
requests, but since IE doesn't make it very easy for you to spoof
these things, in practice this will repel most users. Since they
obviously don't really care about security, having a percentage of
users bypass authentication should not bother the customer. You could
even add some instrumentation to show what the percentage of users
bypassing authorization actually is and log those requests.

The users would be able to bypass authorization at the site by using
Google's cache. No way around that.

Joe K.

You do understand, that once google is able to index the page, a user
can simply go to Google's cache, view your page there, and never have
to register with your website.

Dominick's suggestion is the best.

If, however, you still NEED to do this (and I understand client
requirements), then perhaps you can perform a 'hardcode bypass' by
checking the 'browser' of the visitor. If it matches Google Bot's
header, you can send it directly to the confidential pages.

-Altaf
[MVP - VB]
---------------------------------------------------------------------
-----------
All that glitters has a high refractive index.
www.mendhak.com

Hello Guy,

well - fact is - as long as you are enforcing a login - how can
google index the content whithout logging in?

you could make the login optional (technically) but don't expose the
direct links in your application (i recommend this only because
there is no sensitive content as you say)

or create an abstract of the content for each page (whithout the
need to login) with a "read more" link that requires auth...

---------------------------------------
Dominick Baier - DevelopMentor
http://www.leastprivilege.com
Hi Dominick,

to do what?? index your sensitive data - or why is the area login

protected??

There's nothing sensitive about the content. The strategy behind
the login feature is to count users, and encourage them to sign up
for a mailing list.

The idea is similar to some online newspapers, requiring
registration but not charging a fee. But don't ask me to explain
the management strategy, I just have to implement it.

If it can't be done through forms authentication, can anybody
suggest another way?

And Dominick, thanks for your advice earlier this week.

- Jason

 

[Joe Kaplan \(MVP - ADSI\)]

That could go in a config file, but yeah, it is a little brittle. I
especially like the bit on reporting metrics on how many people are
bypassing security.

Joe K.

Hello Joe,

yeah - that would probably work (ouch - did i say that
as long as Google does not change the agent header...


---------------------------------------
Dominick Baier - DevelopMentor
http://www.leastprivilege.com

The latter suggestion would probably work pretty well in practice.
You would simply write an HttpModule or global.asax handler that ran
on BeginRequest, checked for the bot header and called
SkipAuthorization if it was detected.

An enterprising user could then bypass your forms
authentication/authorization by including the bot's header in their
requests, but since IE doesn't make it very easy for you to spoof
these things, in practice this will repel most users. Since they
obviously don't really care about security, having a percentage of
users bypass authentication should not bother the customer. You could
even add some instrumentation to show what the percentage of users
bypassing authorization actually is and log those requests.

The users would be able to bypass authorization at the site by using
Google's cache. No way around that.

Joe K.

You do understand, that once google is able to index the page, a user
can simply go to Google's cache, view your page there, and never have
to register with your website.

Dominick's suggestion is the best.

If, however, you still NEED to do this (and I understand client
requirements), then perhaps you can perform a 'hardcode bypass' by
checking the 'browser' of the visitor. If it matches Google Bot's
header, you can send it directly to the confidential pages.

-Altaf
[MVP - VB]
---------------------------------------------------------------------
-----------
All that glitters has a high refractive index.
www.mendhak.com
"Dominick Baier [DevelopMentor]"

Hello Guy,

well - fact is - as long as you are enforcing a login - how can
google index the content whithout logging in?

you could make the login optional (technically) but don't expose the
direct links in your application (i recommend this only because
there is no sensitive content as you say)

or create an abstract of the content for each page (whithout the
need to login) with a "read more" link that requires auth...

---------------------------------------
Dominick Baier - DevelopMentor
http://www.leastprivilege.com
Hi Dominick,

to do what?? index your sensitive data - or why is the area login

protected??

There's nothing sensitive about the content. The strategy behind
the login feature is to count users, and encourage them to sign up
for a mailing list.

The idea is similar to some online newspapers, requiring
registration but not charging a fee. But don't ask me to explain
the management strategy, I just have to implement it.

If it can't be done through forms authentication, can anybody
suggest another way?

And Dominick, thanks for your advice earlier this week.

- Jason