googlebot indexes links it finds in newsgroups - just for your info

Discussion in 'HTML' started by Joe Butler, Sep 24, 2007.

  1. Joe Butler

    Joe Butler Guest

    You probably know this already, but 8 hours after publishing a testing area
    link in this newsgroup, googlebot goes straight for it - it checked
    robots.txt first, which I hadn't set up to exclude this folder.
     
    Joe Butler, Sep 24, 2007
    #1
    1. Advertisements

  2. Joe Butler

    CW Northrop Guest

    Yes it seems to spring like a panther on new domain names too, the
    strange and unpredictable beast.
     
    CW Northrop, Sep 24, 2007
    #2
    1. Advertisements

  3. Joe Butler

    mrcakey Guest

    LOL

    +mrcakey
     
    mrcakey, Sep 24, 2007
    #3
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.