robots.txt

Discussion in 'HTML' started by windandwaves, Jan 16, 2006.

  1. windandwaves

    windandwaves Guest

    Hi Folk

    I am trying to reduce the number of pages that are indexed on my site.

    If I were to put

    User-agent: *
    Disallow: /*/*.php
    Disallow: /*/*.html

    in the robots.txt file then would that mean that any files that are not in
    the root www directory will be ignored by robots?

    Thanks in advance (AKA TIA)

    - Nicolaas
     
    windandwaves, Jan 16, 2006
    #1
    1. Advertisements

  2. windandwaves

    Toby Inkster Guest

    windandwaves wrote:

    > If I were to put
    >
    > User-agent: *
    > Disallow: /*/*.php
    > Disallow: /*/*.html
    >
    > in the robots.txt file then would that mean that any files that are not in
    > the root www directory will be ignored by robots?


    No. Most robots do not support globbing, except for the special case
    of "User-agent: *"

    --
    Toby A Inkster BSc (Hons) ARCS
    Contact Me ~ http://tobyinkster.co.uk/contact
     
    Toby Inkster, Jan 17, 2006
    #2
    1. Advertisements

  3. windandwaves

    windandwaves Guest

    Toby Inkster wrote:
    ....

    > No. Most robots do not support globbing, except for the special case
    > of "User-agent: *"


    Would it work for Google? They do support globbing, but I am not sure if
    the syntax is correct.

    Cheers

    Nicolaas
     
    windandwaves, Jan 17, 2006
    #3
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Frankie

    OT: Opinions on Robots.txt

    Frankie, Oct 9, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    1,224
    S. Justin Gengo
    Oct 10, 2005
  2. Daniel Vesma
    Replies:
    15
    Views:
    1,724
    Jacqui or (maybe) Pete
    Jul 2, 2003
  3. Joe Blow

    Missing robots.txt file

    Joe Blow, Aug 29, 2004, in forum: HTML
    Replies:
    5
    Views:
    1,948
    data64
    Aug 30, 2004
  4. Luigi Donatello Asero

    Robots.txt

    Luigi Donatello Asero, Oct 26, 2005, in forum: HTML
    Replies:
    17
    Views:
    1,074
    Edwin van der Vaart
    Oct 29, 2005
  5. CRON
    Replies:
    10
    Views:
    706
    Andy Dingley
    Jul 25, 2006
  6. Replies:
    53
    Views:
    2,545
    John Bokma
    Aug 26, 2006
  7. Paul Furman

    robots.txt

    Paul Furman, Jan 4, 2007, in forum: HTML
    Replies:
    3
    Views:
    476
    Paul Furman
    Jan 5, 2007
  8. Tim w

    meta robots and robots txt

    Tim w, May 22, 2014, in forum: HTML
    Replies:
    1
    Views:
    419
Loading...