robots.txt

Discussion in 'HTML' started by windandwaves, Jan 16, 2006.

  1. windandwaves

    windandwaves Guest

    Hi Folk

    I am trying to reduce the number of pages that are indexed on my site.

    If I were to put

    User-agent: *
    Disallow: /*/*.php
    Disallow: /*/*.html

    in the robots.txt file then would that mean that any files that are not in
    the root www directory will be ignored by robots?

    Thanks in advance (AKA TIA)

    - Nicolaas
     
    windandwaves, Jan 16, 2006
    #1
    1. Advertising

  2. windandwaves

    Toby Inkster Guest

    windandwaves wrote:

    > If I were to put
    >
    > User-agent: *
    > Disallow: /*/*.php
    > Disallow: /*/*.html
    >
    > in the robots.txt file then would that mean that any files that are not in
    > the root www directory will be ignored by robots?


    No. Most robots do not support globbing, except for the special case
    of "User-agent: *"

    --
    Toby A Inkster BSc (Hons) ARCS
    Contact Me ~ http://tobyinkster.co.uk/contact
     
    Toby Inkster, Jan 17, 2006
    #2
    1. Advertising

  3. windandwaves

    windandwaves Guest

    Toby Inkster wrote:
    ....

    > No. Most robots do not support globbing, except for the special case
    > of "User-agent: *"


    Would it work for Google? They do support globbing, but I am not sure if
    the syntax is correct.

    Cheers

    Nicolaas
     
    windandwaves, Jan 17, 2006
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Frankie

    OT: Opinions on Robots.txt

    Frankie, Oct 9, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    1,048
    S. Justin Gengo
    Oct 10, 2005
  2. Daniel Vesma
    Replies:
    15
    Views:
    1,550
    Jacqui or (maybe) Pete
    Jul 2, 2003
  3. Neil White

    Re: robots.txt

    Neil White, Aug 8, 2003, in forum: HTML
    Replies:
    0
    Views:
    414
    Neil White
    Aug 8, 2003
  4. lostinspace

    Re: robots.txt

    lostinspace, Aug 8, 2003, in forum: HTML
    Replies:
    0
    Views:
    394
    lostinspace
    Aug 8, 2003
  5. Tim w

    meta robots and robots txt

    Tim w, May 22, 2014, in forum: HTML
    Replies:
    1
    Views:
    159
Loading...

Share This Page