Web user controls & Google SEO spiders?

[Guest]

Hi all,

I posted this question before but didn't get any good answers so I give it
one more try.

I am wondering how the search engine spiders like Google's or Yahoo's
handles your asp.net web page if it is built with Web User Controls? Most
of our web site is built with web user controls which means that the HTML is
not generated before a user actually visits the pages. The result of using
the web controls is that we have big problems to get it up on some search
engines (especially Google) - even if we have been up on the Internet for
more than 4 months.

Most of our navigation links are based on pure HTML so its not a problem
with Javascript or Postbacks. But the problem is that this navigation HTML
resides in different web user controls (example: my main menu is a web user
control) which means that it is dynamicly created when the user visits the
pages.

I know the discussion about spiders & frames and that frames can cause a
problem for spiders since the HTML is dynamicly generated. It seems like it
is the same with web user controls but I really hope thats not the case.

Anyway this must be a pretty common problem for asp.net developers using web
controls for building commercial web sites that you want to be on the search
engines, so can anyone answer this one?

Thanks
Thomas

 

[bruce barker]

search engines don't care how the html is produced, they just have to parse
it, index it, find the links and crawl the links. generally search engies
will only crawl so deep. also most search engines will not crawl your site,
until there are enough other sites that link to your site. some search
engine you can buy crawling, but i don't believe google sells this service.


-- bruce (sqlwork.com)

 

[Steven Cheng[MSFT]]

Thanks for bruce's suggestions.

Hi Thomas,

As Bruce has mentioned the web search engine just crawl pages through url,
so the most important thing is whether your web pages' url can be found by
it. If so, the search engine will send request to your pages and get your
page 's response content then, it can parse the response html locally and
get the related links in it and parse into deep. Also, for asp.net/ asp
such dynamic generate pages(rather than static html pages), the url
sometimes is not very search engine friendly , for example, we often use
querystring parameter to generate different output on the same aspx or asp
page, for such page , it is hard for the search engine to locate it . And
there're also some articles discussing provide search engine friendly url
in asp.net web application( using URLRewriting), here is some:

#URL Rewriting - Search Engine Friendly URLs
http://www.seoconsultants.com/articles/1000/urls.asp

#Ensuring Your Site is Search Engine Friendly
http://authors.aspalliance.com/brettb/Search_Engine_Optimization.asp

#Search engine friendly URLs using ASP.NET (C#.NET)
http://www.codetoad.com/asp.net_ma_searchenginefriendly.asp

#Provide Custom Hierarchical URLs
http://www.ftponline.com/vsm/2002_02/magazine/columns/qa/

In addition ,here is a MSDN tech article on building a custome web spider
in .net , maybe also helpful for getting how the web search engine work:

#Crawl Web Sites and Catalog Info to Any Data Store with ADO.NET and Visual
Basic .NET
http://msdn.microsoft.com/msdnmag/issues/02/10/spiderinnet/

Thanks.

Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

 

[Guest]

search engines don't care how the html is produced,

That is not true. HTML frames and spiders is a well known problem in SEO.
http://searchenginewatch.com/webmasters/article.php/2167901

Thomas

 

[Guest]

Hi Steven,

All my pages contains static URL's so thats not a problem here. What I'm
concerned about is that my pages (or actually the menus that contains my
links) is dynamicly generated via web user controls when the USERS visits the
pages.

So - will web user controls be rendered in the same way for search engine
spiders and for normal users than there is no problem. But if the web user
controls doesn't render its html for spiders (like frames don't render its
html to spiders/or spiders can't follow the links) then I have a problem.

...If so, the search engine will send request to your pages and get your
page 's response content then, it can parse the response html locally and
get the related links in it and parse into deep.

Just to clarify your answer - so you mean that web user controls render its
HTML to spiders as they do for normal users, and that spiders have no problem
what so ever to get the HTML that a web user control render?

Thank you!
Thomas

 

[Guest]

Hi Steven,

All my pages contains static URL's so thats not a problem here. What I'm
concerned about is that my pages (or actually the menus that contains my
links) is dynamicly generated via web user controls when the USERS visits the
pages.

So - will web user controls be rendered in the same way for search engine
spiders and for normal users than there is no problem. But if the web user
controls doesn't render its html for spiders (like frames don't render its
html to spiders/or spiders can't follow the links) then I have a problem.

...If so, the search engine will send request to your pages and get your
page 's response content then, it can parse the response html locally and
get the related links in it and parse into deep.

So just to clarify your answer - so you mean that web user controls render
its HTML to spiders as they do for normal users, and that spiders have no
problem what so ever to get the HTML that a web user control render?

Thank you!
Thomas

 

[Steven Cheng[MSFT]]

Hi Thomas,

Thanks for your followup.
From your further description, I've got your actual concerns. You're worry
about the frames, yes , frames (such as <frame> for <iframe> ) will break
the page's html sources since a page embeded in frame is separate from its
parent page.

However, the ASP.NET UserControls are not frame based, when we use
Usercontrols in our asp.net page, the UserControl's output html will be
embeded into the parent page( become part of its container page ). That's
why we said that the web spider won't care about the Usercontrol since it
has no sense of whehter there is Usercontrol on a web page , it will only
get the final output hmlt of the whole page and which also contains the
embeded UserControl's output html.

Frames are html elements which has no concerns with asp.net. Pages that
dosn't use Usercontrol can also contains frames. For example, if a normal
static html pages which use <frame> or <iframe> , that will also cause the
search engine unable to retrieve the embeded page's html output.

If you still have anything unclear, please feel free to post here. Thanks.


Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

 

[Steven Cheng[MSFT]]

Hi Thomas,

Have you had a chance to check the things in my last reply or have you got
any further ideas on this issue? If there 're anything else we can help,
please feel free to post here. Thanks.

Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)