googlebot and CheckVirtualFileExists Exceptions!

[Guest]

Hi everyone!

I get some errors lately regarding:
HTTP_USER_AGENT Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)

and:

System.Web.UI.Util.CheckVirtualFileExists(VirtualPath virtualPath)

After googling that,
I found this page:
http://www.inventua.com/forums.content?forumid=1&postid=2948&view=topic

which notes the following:

The user-agent for this one is the Googlebot, which is known to deliberately
generate bad Urls sometimes as part of it's algorithm.

It will keep happening when the GoogleBot tries to index your page. It's
"remembering" the existance of the page from the last time it indexed your
site.

I'm not real familiar with the GoogleBot, but it sounds like that's Google's
bot that crawls around the internet indexing pages. I read an article on the
DotNetNuke site that talked about GoogleBot generated errors taxing the
webserver, and eventually causing an IIS restart due to memory issues. The
general exception errors build up in memory until they are batched to the DB,
and if you get enough errors quick enough, IIS thinks there's a memory leak
and restarts. So, is there anythign I can do about this?

So, does anyone know of a way we can stop googlebot from generating these
errors?

Thanks in advance!

 

[Juan T. Llibre]

re:
!> So, does anyone know of a way we can stop googlebot from generating these errors?

Redirect 404's to a page of your own choosing.

Hi everyone!

I get some errors lately regarding:
HTTP_USER_AGENT Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)

and:

System.Web.UI.Util.CheckVirtualFileExists(VirtualPath virtualPath)

After googling that,
I found this page:
http://www.inventua.com/forums.content?forumid=1&postid=2948&view=topic

which notes the following:

The user-agent for this one is the Googlebot, which is known to deliberately
generate bad Urls sometimes as part of it's algorithm.

It will keep happening when the GoogleBot tries to index your page. It's
"remembering" the existance of the page from the last time it indexed your
site.

I'm not real familiar with the GoogleBot, but it sounds like that's Google's
bot that crawls around the internet indexing pages. I read an article on the
DotNetNuke site that talked about GoogleBot generated errors taxing the
webserver, and eventually causing an IIS restart due to memory issues. The
general exception errors build up in memory until they are batched to the DB,
and if you get enough errors quick enough, IIS thinks there's a memory leak
and restarts. So, is there anythign I can do about this?

So, does anyone know of a way we can stop googlebot from generating these
errors?

Thanks in advance!

 

[Guest]

Redirect 404's to a page of your own choosing.

I don't want that,
I need to somehow stop googlebot from trying to hit these 'invalid' urls,
is there any workaround for this?

 

[Juan T. Llibre]

re:
!>> Redirect 404's to a page of your own choosing.
!> I don't want that

That's the best solution.

It's not expensive, in terms of CPU cycles, and it eliminates
the memory problems stemming from unresolved urls.

re:
!> I need to somehow stop googlebot from trying to hit these 'invalid' urls

You can't do that. Don't bang your head against the wall.
You cannot stop queries to invalid urls...no matter who sends them.

re:
!> is there any workaround for this?

Not that I know of, but don't let me stop you.

Try to find a different solution.

 

[Guest]

re:
!>> Redirect 404's to a page of your own choosing.
!> I don't want that

That's the best solution.

It's not expensive, in terms of CPU cycles, and it eliminates
the memory problems stemming from unresolved urls.

re:
!> I need to somehow stop googlebot from trying to hit these 'invalid' urls

You can't do that. Don't bang your head against the wall.
You cannot stop queries to invalid urls...no matter who sends them.

re:
!> is there any workaround for this?

Not that I know of, but don't let me stop you.

Try to find a different solution.

ok, thanks for the answer,
some other thing,
is there any way to find out why does googlebot hit those invalid urls?

 

[Juan T. Llibre]

re:
!> is there any way to find out why does googlebot hit those invalid urls?

You might want to sieve through this Google query :
http://www.google.com/search?q=googlebot+"invalid+urls"&hl=en&start=10&sa=N

This article has some tips :

http://www.seo4fun.com/notes/supplementals.html