meta robots and robots txt

Discussion in 'HTML' started by Tim w, May 22, 2014.

  1. Tim w

    Tim w Guest

    I think I have this right but would be grateful if somebody could confirm.

    I upload prototype sites for my clients. I used to put them in a sub
    directory of my site ( so ) and put
    in the head of the html (every page).

    I have made a sub-domain called 'demo' specifically for uploading
    proto-sites and work in progress. I don't want them to mess up the
    google returns for my site and my clients won't want google indexing a
    lot of Lorem Ipsum and other crap so in the root of
    I put a text file named robots.txt which reads
    User-agent: *
    Disallow: /
    And that's all. I can forget the meta robots thing. And my main site
    will still be indexed.

    Is that right? Sort of important that it should be right and I didn't go
    to college for this stuff.

    Tim W
    Tim w, May 22, 2014
    1. Advertisements

  2. Tim w

    se Guest

    A robots.txt should always be placed in the root of a website.
    This is because, a search engine is looking for a robots.txt the
    very first arriving to a website. Only one robots.txt is allowed.
    Whereas you can have an .htaccess file in any folder.
    google for more info on robots.txt.
    se, May 22, 2014
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.