Re: robots.txt

Discussion in 'HTML' started by lostinspace, Aug 8, 2003.

  1. lostinspace

    lostinspace Guest

    ----- Original Message -----
    From: David Graham <>
    Newsgroups: alt.html
    Sent: Friday, August 08, 2003 10:42 AM
    Subject: robots.txt


    > Hi
    > I have the following 2 lines in my robots.txt to prevent search engines
    > indexing the files and folders off the sefriendly folder:
    >
    > User-agent: *
    > Disallow: /sefriendly/
    >
    > To prevent access to another folder off the root folder called www2, do I
    > just add another disallow line like below
    >
    >
    > Disallow: /www2/
    >
    >
    > Thanks
    > David
    >



    David,
    Prevent and disallow or even allow or not adequate descriptions of
    what robots.txt does.
    Although those terms are frequently used.

    Robots.txt are "suggestions" for compliant bots.

    Htaccess is a control which implements prevention, allows and disallows.

    hope this helps.
    lostinspace, Aug 8, 2003
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Frankie

    OT: Opinions on Robots.txt

    Frankie, Oct 9, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    997
    S. Justin Gengo
    Oct 10, 2005
  2. Daniel Vesma
    Replies:
    15
    Views:
    1,503
    Jacqui or (maybe) Pete
    Jul 2, 2003
  3. Neil White

    Re: robots.txt

    Neil White, Aug 8, 2003, in forum: HTML
    Replies:
    0
    Views:
    390
    Neil White
    Aug 8, 2003
  4. Joe Blow

    Missing robots.txt file

    Joe Blow, Aug 29, 2004, in forum: HTML
    Replies:
    5
    Views:
    1,713
    data64
    Aug 30, 2004
  5. Tim w

    meta robots and robots txt

    Tim w, May 22, 2014, in forum: HTML
    Replies:
    1
    Views:
    98
Loading...

Share This Page