/robots.txt at end of URL?

Discussion in 'ASP .Net' started by Les Caudle, Jul 20, 2007.

  1. Les Caudle

    Les Caudle Guest

    I'm noticing that web requrests are coming in with /robots.txt appended at the
    end:

    http://www.domain.com/ProductDetails.aspx?productID=527/robots.txt

    I can correct these, one by one for each page, but I'd like to find a way to
    have ASP.NET 2.,0 strip this invalid /robots.txt off the end of any URL for me.

    Is this possible?
    --
    Thanks in advance, Les Caudle
     
    Les Caudle, Jul 20, 2007
    #1
    1. Advertising

  2. Let me asking why do you want to strip it?
    robors.txt is asked by automatic robots that collect information. Like
    google or yaho.

    But in your case apparently it's some lame robot that can not parse out url
    correctly simply ads /robot.txt to the end of url and asks for it from the
    server.
    So why would you worry what kind of garbage it will get in return from your
    page.
    All important robots you should be worring about like Google or Yahoo parse
    out url correctly.

    So i am saying just ignore it. It's not human.

    George.



    "Les Caudle" <> wrote in message
    news:eek:...
    > I'm noticing that web requrests are coming in with /robots.txt appended at
    > the
    > end:
    >
    > http://www.domain.com/ProductDetails.aspx?productID=527/robots.txt
    >
    > I can correct these, one by one for each page, but I'd like to find a way
    > to
    > have ASP.NET 2.,0 strip this invalid /robots.txt off the end of any URL
    > for me.
    >
    > Is this possible?
    > --
    > Thanks in advance, Les Caudle
     
    George Ter-Saakov, Jul 20, 2007
    #2
    1. Advertising

  3. Les Caudle

    Les Caudle Guest

    Well, it creates an event in my event log that would distract me from real
    events caused by users who had problems.

    Would be nice to be able to globally deal with it.

    Regards, Les Caudle

    On Fri, 20 Jul 2007 10:28:48 -0400, "George Ter-Saakov" <>
    wrote:

    >Let me asking why do you want to strip it?
    >robors.txt is asked by automatic robots that collect information. Like
    >google or yaho.
    >
    >But in your case apparently it's some lame robot that can not parse out url
    >correctly simply ads /robot.txt to the end of url and asks for it from the
    >server.
    >So why would you worry what kind of garbage it will get in return from your
    >page.
    >All important robots you should be worring about like Google or Yahoo parse
    >out url correctly.
    >
    >So i am saying just ignore it. It's not human.
    >
    >George.
    >
    >
    >
    >"Les Caudle" <> wrote in message
    >news:eek:...
    >> I'm noticing that web requrests are coming in with /robots.txt appended at
    >> the
    >> end:
    >>
    >> http://www.domain.com/ProductDetails.aspx?productID=527/robots.txt
    >>
    >> I can correct these, one by one for each page, but I'd like to find a way
    >> to
    >> have ASP.NET 2.,0 strip this invalid /robots.txt off the end of any URL
    >> for me.
    >>
    >> Is this possible?
    >> --
    >> Thanks in advance, Les Caudle

    >
     
    Les Caudle, Jul 20, 2007
    #3
  4. Well, the problem is that this time it's robot.txt. Next time (with another
    bad robot) it will be something else.
    You can not fix it for every bad robot that out there.
    Override Application_OnError and send an email to yourself every time
    something bad happened. Then you can use Email's rules to filter out the
    most annoying ones

    George.


    "Les Caudle" <> wrote in message
    news:...
    > Well, it creates an event in my event log that would distract me from real
    > events caused by users who had problems.
    >
    > Would be nice to be able to globally deal with it.
    >
    > Regards, Les Caudle
    >
    > On Fri, 20 Jul 2007 10:28:48 -0400, "George Ter-Saakov"
    > <>
    > wrote:
    >
    >>Let me asking why do you want to strip it?
    >>robors.txt is asked by automatic robots that collect information. Like
    >>google or yaho.
    >>
    >>But in your case apparently it's some lame robot that can not parse out
    >>url
    >>correctly simply ads /robot.txt to the end of url and asks for it from the
    >>server.
    >>So why would you worry what kind of garbage it will get in return from
    >>your
    >>page.
    >>All important robots you should be worring about like Google or Yahoo
    >>parse
    >>out url correctly.
    >>
    >>So i am saying just ignore it. It's not human.
    >>
    >>George.
    >>
    >>
    >>
    >>"Les Caudle" <> wrote in message
    >>news:eek:...
    >>> I'm noticing that web requrests are coming in with /robots.txt appended
    >>> at
    >>> the
    >>> end:
    >>>
    >>> http://www.domain.com/ProductDetails.aspx?productID=527/robots.txt
    >>>
    >>> I can correct these, one by one for each page, but I'd like to find a
    >>> way
    >>> to
    >>> have ASP.NET 2.,0 strip this invalid /robots.txt off the end of any URL
    >>> for me.
    >>>
    >>> Is this possible?
    >>> --
    >>> Thanks in advance, Les Caudle

    >>

    >
     
    George Ter-Saakov, Jul 20, 2007
    #4
  5. Hi Les,

    I agree with George, as long as your web site can be accessed publicly, you
    cannot guarantee every URL request is valid or in expected manner.


    Regards,
    Walter Wang (, remove 'online.')
    Microsoft Online Community Support

    ==================================================
    When responding to posts, please "Reply to Group" via your newsreader so
    that others may learn and benefit from your issue.
    ==================================================

    This posting is provided "AS IS" with no warranties, and confers no rights.
     
    Walter Wang [MSFT], Jul 22, 2007
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Frankie

    OT: Opinions on Robots.txt

    Frankie, Oct 9, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    1,064
    S. Justin Gengo
    Oct 10, 2005
  2. Daniel Vesma
    Replies:
    15
    Views:
    1,553
    Jacqui or (maybe) Pete
    Jul 2, 2003
  3. Neil White

    Re: robots.txt

    Neil White, Aug 8, 2003, in forum: HTML
    Replies:
    0
    Views:
    416
    Neil White
    Aug 8, 2003
  4. lostinspace

    Re: robots.txt

    lostinspace, Aug 8, 2003, in forum: HTML
    Replies:
    0
    Views:
    396
    lostinspace
    Aug 8, 2003
  5. Tim w

    meta robots and robots txt

    Tim w, May 22, 2014, in forum: HTML
    Replies:
    1
    Views:
    166
Loading...

Share This Page