googlebot indexes links it finds in newsgroups - just for your info

Discussion in 'HTML' started by Joe Butler, Sep 24, 2007.

  1. Joe Butler

    Joe Butler Guest

    You probably know this already, but 8 hours after publishing a testing area
    link in this newsgroup, googlebot goes straight for it - it checked
    robots.txt first, which I hadn't set up to exclude this folder.
    Joe Butler, Sep 24, 2007
    1. Advertisements

  2. Joe Butler

    CW Northrop Guest

    Yes it seems to spring like a panther on new domain names too, the
    strange and unpredictable beast.
    CW Northrop, Sep 24, 2007
    1. Advertisements

  3. Joe Butler

    mrcakey Guest


    mrcakey, Sep 24, 2007
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.