googlebot indexes links it finds in newsgroups - just for your info

J

Joe Butler

You probably know this already, but 8 hours after publishing a testing area
link in this newsgroup, googlebot goes straight for it - it checked
robots.txt first, which I hadn't set up to exclude this folder.
 
C

CW Northrop

You probably know this already, but 8 hours after publishing a testing
area link in this newsgroup, googlebot goes straight for it - it
checked robots.txt first, which I hadn't set up to exclude this
folder.

Yes it seems to spring like a panther on new domain names too, the
strange and unpredictable beast.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,769
Messages
2,569,582
Members
45,057
Latest member
KetoBeezACVGummies

Latest Threads

Top