C
Ciaran
I have a piece of code that I'd rather google's spider did not follow. Is
this
possible please?
this
possible please?
I have a piece of code that I'd rather google's spider did not follow.
Is this possible please?
Evertjan. said:Ciaran wrote on 28 jul 2007 in comp.lang.javascript:
Only link URLs are "FOLLOWED", code is not.
Or do you want it not to be "INDEXED"?
<http://www.google.com/support/webmasters>
Not really on topic for javascript, meseems.
Sorry, you're right. I have a bit of javascript code that I dont want to
google to index. Is there a Javascript method to hide this code from the
crawler?
Evertjan. said:Ciaran wrote on 28 jul 2007 in comp.lang.javascript:
No, not with clientside javascript.
You could test for the word "googlebot" with serverside jscriptin
Request.ServerVariables("HTTP_USER_AGENT"),
but I doubt Google will ever visit your site, if they find out.
Anyhow, why would you want to do that?
Is your scripting so unique and valuable,
that anyone would be interested?
If so, use serverside techniques, which codes are hidden by default.
Evertjan. said:Ciaran wrote:
You could test for the word "googlebot" with serverside jscriptin
Request.ServerVariables("HTTP_USER_AGENT"),
but I doubt Google will ever visit your site, if they find out.
You shouldn't do that.
The Robots Exclusion Protocol contains widely
accepted rules how to keep crawlers away from your website, or from
some parts of it. Just create a file named "robots.txt" in your
website's root directory.
This "robots.txt"-file keeps all robots out of .js files:
User-agent: *
Disallow: /*.js$
This "robots.txt"-file tells (only) Google to stay away from
directory /javascript/:
User-agent: Googlebot
Disallow: /javascript/
Information:
http://en.wikipedia.org/wiki/Robots_Exclusion_Standard
http://www.google.com/support/webmasters/bin/answer.py?answer=33570
Evertjan. said:Bart Van der Donck wrote on 29 jul 2007 in comp.lang.javascript:
Do what? Ever visit his site when you are Google?
My interpretation of "a piece of code" is inHTML code,
not a complete .js file.
Bart Van der Donck said:I think it's better to use robots.txt rather than reading out the UA,
and then decide to (whether or not) output javascript code based on
that info.
The general goal is not clear for me. Crawlers don't do much with
javascript; at best they may detect some hyperlinks to index them. So
what's the use of hiding js code for crawlers ? Preventing the code to
be displayed in search results ? I'ld say a well-programmed crawler
wouldn't do that. Preventing source code to influence algorithms like
PageRank ?
When I read the orignal poster's sentence "I have a piece of code that
I'd rather google's spider did not follow", I surmised he meant a a
separate .js file, because that's the only thing a bot could follow.
Want to reply to this thread or ask your own question?
You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.