Hi,\n\nThere is something I really don't understand ; and I would like your\nadvises...\n\n 1. Some websites, (for instance news.google.fr) contains a\nsyndication feed (like [URL]http://news.google.fr/nwshp?topic=po&output=atom[/URL]).\n\n 2. Theses websites have a robots.txt file preventing some robots\n(declared by user-agents) from indexation.\nFor example : [URL]http://news.google.fr/robots.txt[/URL] contains (extract) :\nUser-agent: *\nDisallow: /nwshp\n\n 3. I've developped an syndication aggregator, and I woul'd like to\nrespect these robots.txt files. but as I can see and understand, my\nuser-agent isn't authorized to acces /nwshp?topic=po&output=atom\nbecause of this robots.txt...\n\nSo, is it normal ? robots.txt files are only for indexation robots ?\nto sum up, my syndication aggregator should respect these files or\nnot ?\n\nThanks.