OT: Opinions on Robots.txt

F

Frankie

I'd appreciate your informed perspectives and opinions on implementing
robots.txt in web sites on the Internet.

I kind of assumed that it is a general and good practice done on a regular
basis. But while cruising around to different Web sites this morning, I
discovered that many well-known and respected sites don't have one. Or, at
least I could not retrieve it by typing in the URL to see it in my browser.

Please note that I understand the philosophy and intent of robots.txt and
the "honor system" according to which spiders are supposed to make use of
robots.txt.

What I'm wondering is why so many prominent sites don't have one (or perhaps
it's just not accessible to my browser?).

I'd also be interested in knowing any good reasons to *not* put a robots.txt
in one's site. Is it perhaps bad or dangerous to "announce" the existance of
certain files and folders in a Web site?

Thanks!
 
S

S. Justin Gengo

Frankie,

I like to include a Robots.txt file even if I'm not limiting a spider at
all. The reason? Because if I don't my log is going to have a bunch of 404
errors when spiders come looking for it...

--
Sincerely,

S. Justin Gengo, MCP
Web Developer / Programmer

www.aboutfortunate.com

"Out of chaos comes order."
Nietzsche
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,733
Messages
2,569,440
Members
44,832
Latest member
GlennSmall

Latest Threads

Top