F
Frankie
I'd appreciate your informed perspectives and opinions on implementing
robots.txt in web sites on the Internet.
I kind of assumed that it is a general and good practice done on a regular
basis. But while cruising around to different Web sites this morning, I
discovered that many well-known and respected sites don't have one. Or, at
least I could not retrieve it by typing in the URL to see it in my browser.
Please note that I understand the philosophy and intent of robots.txt and
the "honor system" according to which spiders are supposed to make use of
robots.txt.
What I'm wondering is why so many prominent sites don't have one (or perhaps
it's just not accessible to my browser?).
I'd also be interested in knowing any good reasons to *not* put a robots.txt
in one's site. Is it perhaps bad or dangerous to "announce" the existance of
certain files and folders in a Web site?
Thanks!
robots.txt in web sites on the Internet.
I kind of assumed that it is a general and good practice done on a regular
basis. But while cruising around to different Web sites this morning, I
discovered that many well-known and respected sites don't have one. Or, at
least I could not retrieve it by typing in the URL to see it in my browser.
Please note that I understand the philosophy and intent of robots.txt and
the "honor system" according to which spiders are supposed to make use of
robots.txt.
What I'm wondering is why so many prominent sites don't have one (or perhaps
it's just not accessible to my browser?).
I'd also be interested in knowing any good reasons to *not* put a robots.txt
in one's site. Is it perhaps bad or dangerous to "announce" the existance of
certain files and folders in a Web site?
Thanks!