How to make dynamic aspx pages indexable by spiders

[moondaddy]

I'm writing an ecommerce app in asp.net/vb.net and need to make the pages
searchable and crawlable by spiders, particularly Google's. As far as I
know, if my pages's contents are mostly populated by user controls on a
single page and I call these different controls by passing one or more
parameters like this: myweb.com/default.aspx?MenuID=44, then the spiders
aren't going to be able do to anything with this. asp.net offers lots of
great technology, but if the success of the site depends on people's ability
to find it, then all of this work is to no avail. I'm surprised to find
very little (none actually) threads or documentation about this. the only
information I can find on optimizing a site for search engine ratings are by
non-ms portals and they all talk about static html pages. its like they
never heard of asp.net.

Can anyone help on this under talked about topic?

historically I've built business applications and never had to worry about
his. I spent tons of time building and optimizing the site for usability
and performance, and now that I'm trying to get it indexed by the search
engine, I'm finding that my high performance database driven site wont be
seen by the spiders, or I'm just not finding the right documentation.

 

[Cowboy \(Gregory A. Beamer\)]

Fully dynamic? It is not going to happen as the spider does not know if it
can trust what you are going to deliver.

Predicable, but with a dynamic engine? Consider an HTTP Handler that makes
up a URL like so:

http://mysite.com/august/2004/1/somepage.html

For

http://mysite.com/article.aspx?id=16287674465

the VBPJ site (www.vbpj.com) had an article on hierarchical URLs that was
very nice in showing how to set up the handler for this type of URL. The
spider will see an HTML page, not a dynamic URL, even though you are
creating a page on the fly.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

**********************************************************************
Think Outside the Box!
**********************************************************************

 

[moondaddy]

did you mean this one?
http://www.ftponline.com/vsm/2002_02/online/online_print/jgoodyear/default.aspx
If so, I didnt see the connect. If not, do you have a url to the article
you were talking about?

Thanks.

--
(e-mail address removed)

Fully dynamic? It is not going to happen as the spider does not know if it
can trust what you are going to deliver.

Predicable, but with a dynamic engine? Consider an HTTP Handler that makes
up a URL like so:

http://mysite.com/august/2004/1/somepage.html

For

http://mysite.com/article.aspx?id=16287674465

the VBPJ site (www.vbpj.com) had an article on hierarchical URLs that was
very nice in showing how to set up the handler for this type of URL. The
spider will see an HTML page, not a dynamic URL, even though you are
creating a page on the fly.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

**********************************************************************
Think Outside the Box!
**********************************************************************

I'm writing an ecommerce app in asp.net/vb.net and need to make the pages
searchable and crawlable by spiders, particularly Google's. As far as I
know, if my pages's contents are mostly populated by user controls on a
single page and I call these different controls by passing one or more
parameters like this: myweb.com/default.aspx?MenuID=44, then the spiders
aren't going to be able do to anything with this. asp.net offers lots of
great technology, but if the success of the site depends on people's ability
to find it, then all of this work is to no avail. I'm surprised to find
very little (none actually) threads or documentation about this. the only
information I can find on optimizing a site for search engine ratings

are

 

[moondaddy]

Ok maybe I missed the point if this was in fact the article you were talking
about. The light when on when I read another article about this problem:
http://www.seochat.com/c/a/Search-Engine-Optimization/Dynamic-URLs-In-The-Eyes-Of-A-Search-Engine/
where they were talking about php pages, but the same concept (although it
didn't get into the mechanics, just an overview).

so in short is this how it would work: Rather than passing in a parameter,
I'd pass in a url like http://www..mysite/procucts/category/animals and on
the backend I'll use the http handler to intersept this, parse it, and run
select case on the last word for example (in this case animals), then call
the real page using a redirect passing in the real parameter like
products.aspx?89 (where 89 could be the category id animals). Is this the
basic idea?

--
(e-mail address removed)

did you mean this one?
http://www.ftponline.com/vsm/2002_02/online/online_print/jgoodyear/default.aspx
If so, I didnt see the connect. If not, do you have a url to the article
you were talking about?

Thanks.

 

[clintonG]

"Rewrite URLs on the Fly" by Jeff Prosise (ASP.NET C#)

The article will show up here [1] but I do not know when as it was
just published by asp.netPRO, March 2004 and can only be accessed
via [2] by magazine subscribers.

[1] http://www.wintellect.com/resources/articles/articles.aspx?authorID=1
[2] www.aspnetPRO.com/download

--
<%= Clinton Gallagher
A/E/C Consulting, Web Design, e-Commerce Software Development
Wauwatosa, Milwaukee County, Wisconsin USA
NET (e-mail address removed)
URL http://www.metromilwaukee.com/clintongallagher/

 

[darrel]

As far as I

know, if my pages's contents are mostly populated by user controls on a
single page and I call these different controls by passing one or more
parameters like this: myweb.com/default.aspx?MenuID=44, then the spiders
aren't going to be able do to anything with this.

This is incorrect, at least in terms of Google. Google can handle query
strings just fine, with a few caveats:

1) It will index your site slower than if it does not see querystrings (this
is actually because it is polite...if it sees a query string, it assumes a
database, and doesn't want to overload it with queries).

2) It seems to not like more than 3 values in the query string, with two
being the most common limit.

The other issue with query strings, of course, is that they are not very
human readable. If this is a big issue for you, look into a URL rewriting
tool like ISAPI rewrite.

-Darrel

 

[Steven Cheng[MSFT]]

Thanks for all your informative suggestions.

Hi Moondaddy,

Here are some further resources on ASP.NET Url Rewriting and how to perform
search engine friendly url:

#URL Rewriting in ASP.NET
http://msdn.microsoft.com/library/en-us/dnaspp/html/urlrewriting.asp?frame=t
rue

#Rewriting with ASP.NET
http://www.codeproject.com/aspnet/URLRewriter.asp

#Search engine friendly URLs using ASP.NET (C#.NET)
http://www.codetoad.com/asp.net_ma_searchenginefriendly.asp

Hope also helps. Thanks.


Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

Get Preview at ASP.NET whidbey
http://msdn.microsoft.com/asp.net/whidbey/default.aspx

 

[moondaddy]

These are great links! Just what the good doctor ordered.

THANKS

 

[George Ter-Saakov]

Not ordered, Prescribed.

Just my 2 cents

George.

 

[Cowboy \(Gregory A. Beamer\)]

I was actually thinking this link:
http://www.ftponline.com/vsm/2002_02/magazine/columns/qa/

This one also gives some hints on using it to mask URLs:
http://www.15seconds.com/issue/030522.htm

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

**********************************************************************
Think Outside the Box!
**********************************************************************

did you mean this one?
http://www.ftponline.com/vsm/2002_02/online/online_print/jgoodyear/default.aspx
If so, I didnt see the connect. If not, do you have a url to the article
you were talking about?

Thanks.

 

[Cowboy \(Gregory A. Beamer\)]

There are some that will completely ignore dynamic strings, however. Google
will do fine, over time. Some will index if you add pages in robots.txt, but
will otherwise ignore any dynamic strings. The ASP.NET redirection will
solve the issue, as the URL appears to be a static URL.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

**********************************************************************
Think Outside the Box!
**********************************************************************