I have a hitcounter that logs how many visitors I get. Over the last
month this counter has gone through the roof. It know apears that it is
Robots. My website does not have any meta tags like keywords
description. So they should not be going there. I think someone has
nominated me to them, but I would not know. The database records
clearly indicate a date. I can re-adjust the counter because I have
database records. but I don't want robots increasing my counter.
Desmond,
I think you misunderstand how search engine bots work. It is an
unwritten rule on the Net that any site that is public is open to anyone
who wants to visit, be that a human with a Web browser or a search
engine bot or any other kind of user agent. Search spiders don't wait
for an invitation to spider a Web site. You don't have to have meta tags
and you don't have to submit your site to the search engines. Any public
mention of your site (such as in this newsgroup!) or in some cases even
a non-public mention (such as a URL sent via GMail, which might be
picked up by Google) can make search engines aware of your site. THey're
aggressively competing against one another to provide the best results
and part of "best" is "most complete" which means that if search engine
A knows about more Web sites than search engine B, then A has an
advantage -- hence their enthusiasm for discovering new sites.
They also realize that they will get banned from sites if they spider
them too aggressively and piss people off, so they're (usually) polite
and will try not to overwhelm a site with too many requests at once.
That statement is almost sure to spur a comment from a Webmaster who
feels that her site has been abused by Googlebot/Yahoo Slurp/MSNBot and
I'm sure that happens once in a while, but by and large they try to be
nice because generating hostility works heavily against them.
Also note (as I believe someone else mentioned) that the user agent that
is sent along with a request is based on an honor system. It is trivial
for an evil bot to masquerade as some other bot via the user agent
string.
Please don't top-post.
http://en.wikipedia.org/wiki/Top_posting