robots.txt

[windandwaves]

Hi Folk

I am trying to reduce the number of pages that are indexed on my site.

If I were to put

User-agent: *
Disallow: /*/*.php
Disallow: /*/*.html

in the robots.txt file then would that mean that any files that are not in
the root www directory will be ignored by robots?

Thanks in advance (AKA TIA)

- Nicolaas

 

[Toby Inkster]

If I were to put

User-agent: *
Disallow: /*/*.php
Disallow: /*/*.html

in the robots.txt file then would that mean that any files that are not in
the root www directory will be ignored by robots?

No. Most robots do not support globbing, except for the special case
of "User-agent: *"

 

[windandwaves]

Toby Inkster wrote:
....

No. Most robots do not support globbing, except for the special case
of "User-agent: *"

Would it work for Google? They do support globbing, but I am not sure if
the syntax is correct.

Cheers

Nicolaas