Stopping robots searching particular page

[Nick Theodorakis]

[...]

I doubt that decomposing rats will be a sufficiently universal
deterrence. In fact, I'm not sure you can settle on any image that will,
say, tick off, what? 80% of viewers? 90%? ...

Goatse should do the trick.

Nick

 

[Nikita the Spider]

3. The page _was_ linked to from another page.

4. An indexing robot generates URLs automatically, more or less at random,
and tries them. It might for example try servers known to exist and append
to the server name some strings that are known to be common for web pages,
like /help.htm, /news.html....

5. Someone visits your page[1] and has the Google Toolbar (or others
similar things) installed and reporting back to Google about the sites
they are visiting, thus allowing Google to add the site to their index.

6. Someone sends the URL in an email via a mail service (like GMail)
that's also related to a search engine.

 

[Nikita the Spider]

There are two popular "standards" (neither of which is a standard in
the formal sense). One uses <meta ...> elements in your HTML, and the
other uses separate robots.txt files. Both are described here:

<http://www.robotstxt.org/>

This technique has worked for me given the same parameters of success
that dorayme described.

Both approaches depend on cooperative robots. For uncooperative robots,
all you can do is shout "klaatu barada nikto" and hope for the best.

AFAICT all of the major search engines are well-behaved in this regard.

 

[dorayme]

I gotta go get a drink. I read it, I (sorta) got it, and now my head
hurts so much ...

Do it or don't do it. It is a solution. If you or your client don't
like it, fine. Your choice. But, it's simple, it exists, and, let's
face it, if it's a commercial app? "Use of this site/facility requires
..." And it is NOT onerous.

Ok, I'm wandering downstairs now ...

You come back right up here young man and listen to me some
more... I have not even started... <g>