T
Tomasz Chmielewski
I'm not sure what is the right group for asking questions about
robots.txt file, so I'm asking it here.
I would like to exclude robots from accessing such links:
/index.php?title=One_page&action=edit
/index.php?title=Other_page&action=edit
What should be a robots.txt line to exclude such pages (for bots which
understand regexps, like Googlebot, Yahoo Slurp etc.)?
1) Disallow: /index.php*action=edit
2) Disallow: /index\.php.*action=edit
According to http://www.google.com/help/faq_codesearch.html#regexp (and
http://en.wikipedia.org/wiki/Regular_expression#Syntax), it should be
the 2) one.
However, almost every "robots.txt regexp" search result seem to point to
the 1) one.
What is the correct answer?
robots.txt file, so I'm asking it here.
I would like to exclude robots from accessing such links:
/index.php?title=One_page&action=edit
/index.php?title=Other_page&action=edit
What should be a robots.txt line to exclude such pages (for bots which
understand regexps, like Googlebot, Yahoo Slurp etc.)?
1) Disallow: /index.php*action=edit
2) Disallow: /index\.php.*action=edit
According to http://www.google.com/help/faq_codesearch.html#regexp (and
http://en.wikipedia.org/wiki/Regular_expression#Syntax), it should be
the 2) one.
However, almost every "robots.txt regexp" search result seem to point to
the 1) one.
What is the correct answer?