robots.txt

Discussion in 'HTML' started by windandwaves, Jan 16, 2006.

  1. windandwaves

    windandwaves Guest

    Hi Folk

    I am trying to reduce the number of pages that are indexed on my site.

    If I were to put

    User-agent: *
    Disallow: /*/*.php
    Disallow: /*/*.html

    in the robots.txt file then would that mean that any files that are not in
    the root www directory will be ignored by robots?

    Thanks in advance (AKA TIA)

    - Nicolaas
     
    windandwaves, Jan 16, 2006
    #1
    1. Advertisements

  2. windandwaves

    Toby Inkster Guest

    No. Most robots do not support globbing, except for the special case
    of "User-agent: *"
     
    Toby Inkster, Jan 17, 2006
    #2
    1. Advertisements

  3. windandwaves

    windandwaves Guest

    Toby Inkster wrote:
    ....
    Would it work for Google? They do support globbing, but I am not sure if
    the syntax is correct.

    Cheers

    Nicolaas
     
    windandwaves, Jan 17, 2006
    #3
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.