Code to keep search engines at bay ?

Discussion in 'HTML' started by wiley@volcanomail.com, Oct 7, 2006.

  1. Guest

    Hello,

    Was wondering if there is any code to keep search engine bots from
    picking up my ebay auction pages and archiveing them, earning them free
    publicity and competing against my off-ebay website ?

    Any help appreciated, Wm
    , Oct 7, 2006
    #1
    1. Advertising

  2. <> skrev i meddelandet
    news:...
    > Hello,
    >
    > Was wondering if there is any code to keep search engine bots from
    > picking up my ebay auction pages and archiveing them, earning them free
    > publicity and competing against my off-ebay website ?
    >
    > Any help appreciated, Wm


    You could use https and then protect those pages by using a password.
    As search engines do not know the password they could not come in...

    --
    Luigi Donatello Asero
    https://www.scaiecat-spa-gigi.com/it/svezia.html
    谢谢你, ÑпаÑибо, tack sÃ¥ mycket!
    Luigi Donatello Asero, Oct 7, 2006
    #2
    1. Advertising

  3. Luigi Donatello Asero wrote:
    > <> skrev i meddelandet
    > news:...
    >> Hello,
    >>
    >> Was wondering if there is any code to keep search engine bots from
    >> picking up my ebay auction pages and archiveing them, earning them free
    >> publicity and competing against my off-ebay website ?
    >>
    >> Any help appreciated, Wm

    >
    > You could use https and then protect those pages by using a password.
    > As search engines do not know the password they could not come in...


    No, then his eBay auctions would not be viewable from eBay... https is
    *not* what you think it is...

    Put eBay auction files in a folder and use a robot.txt file to indicate
    that that folder is not to be indexed. Google "robot.txt" for syntax...

    --
    Take care,

    Jonathan
    -------------------
    LITTLE WORKS STUDIO
    http://www.LittleWorksStudio.com
    Jonathan N. Little, Oct 7, 2006
    #3
  4. John Hosking Guest

    Jonathan N. Little wrote:

    > Put eBay auction files in a folder and use a robot.txt file to indicate
    > that that folder is not to be indexed. Google "robot.txt" for syntax...


    That would be robots.txt, with an "s". When you've googled, be sure to
    read the part about robots.txt not working with misbehaving spiders.

    --
    John
    John Hosking, Oct 7, 2006
    #4
  5. John Hosking wrote:

    >> Put eBay auction files in a folder and use a robot.txt file to indicate
    >> that that folder is not to be indexed. Google "robot.txt" for syntax...

    >
    > That would be robots.txt, with an "s". When you've googled, be sure to
    > read the part about robots.txt not working with misbehaving spiders.


    .... and where the file has to go. I doubt the OP can write to that section
    of ebay's site.

    --
    David Dorward <http://blog.dorward.me.uk/> <http://dorward.me.uk/>
    Home is where the ~/.bashrc is
    David Dorward, Oct 7, 2006
    #5
  6. richard Guest

    "Luigi Donatello Asero" <> wrote in message
    news:hJDVg.19304$...
    >
    > <> skrev i meddelandet
    > news:...
    >> Hello,
    >>
    >> Was wondering if there is any code to keep search engine bots from
    >> picking up my ebay auction pages and archiveing them, earning them free
    >> publicity and competing against my off-ebay website ?
    >>
    >> Any help appreciated, Wm

    >
    > You could use https and then protect those pages by using a password.
    > As search engines do not know the password they could not come in...
    >


    The 's' means "secure". IE will show a little icon like a padlock when
    you're on a secure site.
    Password protection is no good for public auctions.
    richard, Oct 7, 2006
    #6
  7. richard Guest

    <> wrote in message
    news:...
    > Hello,
    >
    > Was wondering if there is any code to keep search engine bots from
    > picking up my ebay auction pages and archiveing them, earning them free
    > publicity and competing against my off-ebay website ?
    >
    > Any help appreciated, Wm
    >


    "Your" pages?
    If it's an actual auction listed on the ebay site, then you are at their
    mercy.
    Chances are if you try to install any coding at all, and ebay found out,
    you'd be kicked off.
    Then how you would do it if you don't have access to the root of the domain
    where that stuff would go?
    richard, Oct 7, 2006
    #7
  8. David Dorward wrote:
    > John Hosking wrote:
    >
    >>> Put eBay auction files in a folder and use a robot.txt file to indicate
    >>> that that folder is not to be indexed. Google "robot.txt" for syntax...

    >> That would be robots.txt, with an "s". When you've googled, be sure to
    >> read the part about robots.txt not working with misbehaving spiders.

    >
    > ... and where the file has to go. I doubt the OP can write to that section
    > of ebay's site.
    >

    I think the OP does what I do, host my own images for eBay rather than
    use eBay's serverice to host them so the imagea and/or parts of the
    auction page are on his server not eBay's. So he could prevent the
    indexing of -his- eBay folder.

    And yes, a typo, should have been 'robots.txt'

    And to misbehaving spiders, if they don't play by the rules then there
    is not much you can do...except maybe redirect in an .htaccess entry for
    the little bugger. But you would have to trap the user agent of each
    misbehaving spider.

    --
    Take care,

    Jonathan
    -------------------
    LITTLE WORKS STUDIO
    http://www.LittleWorksStudio.com
    Jonathan N. Little, Oct 7, 2006
    #8
  9. richard wrote:
    >
    > "Luigi Donatello Asero" <> wrote in message
    > news:hJDVg.19304$...
    >>
    >> <> skrev i meddelandet
    >> news:...
    >>> Hello,
    >>>
    >>> Was wondering if there is any code to keep search engine bots from
    >>> picking up my ebay auction pages and archiveing them, earning them free
    >>> publicity and competing against my off-ebay website ?
    >>>
    >>> Any help appreciated, Wm

    >>
    >> You could use https and then protect those pages by using a password.
    >> As search engines do not know the password they could not come in...
    >>

    > The 's' means "secure".


    "Secure" means many things. In this case, it means the stream is
    encrypted so that anyone intercepting it on its way between the client
    and the server is unable to read it. This has nothing to do with robots
    or access.
    Harlan Messinger, Oct 7, 2006
    #9
  10. wrote:
    > Hello,
    >
    > Was wondering if there is any code to keep search engine bots from
    > picking up my ebay auction pages and archiveing them, earning them free
    > publicity and competing against my off-ebay website ?


    I don't understand what it is you're trying to prevent. A search engine
    isn't getting publicity from your pages turning up in search results. As
    for the competition, it sounds like you don't want your eBay pages
    competing against your non-eBay pages. What, you want *fewer* potential
    buyers to find the items you're trying to sell?
    Harlan Messinger, Oct 7, 2006
    #10
  11. In article <2fc8a$4527add2$40cba78b$>,
    "Jonathan N. Little" <> wrote:

    > And to misbehaving spiders, if they don't play by the rules then there
    > is not much you can do...except maybe redirect in an .htaccess entry for
    > the little bugger. But you would have to trap the user agent of each
    > misbehaving spider.


    I'm sure you're aware that misbehaving spiders can also send a false
    user-agent string. There's nothing (except the threat of legal action)
    to stop them from masquerading as Googlebot, for instance. That said, a
    lot of ill-behaved spiders that I see in my server logs look like they
    send distinctive user-agent strings.

    --
    Philip
    http://NikitaTheSpider.com/
    Whole-site HTML validation, link checking and more
    Nikita the Spider, Oct 8, 2006
    #11
  12. Nikita the Spider wrote:
    > In article <2fc8a$4527add2$40cba78b$>,
    > "Jonathan N. Little" <> wrote:
    >
    >> And to misbehaving spiders, if they don't play by the rules then there
    >> is not much you can do...except maybe redirect in an .htaccess entry for
    >> the little bugger. But you would have to trap the user agent of each
    >> misbehaving spider.

    >
    > I'm sure you're aware that misbehaving spiders can also send a false
    > user-agent string. There's nothing (except the threat of legal action)
    > to stop them from masquerading as Googlebot, for instance. That said, a
    > lot of ill-behaved spiders that I see in my server logs look like they
    > send distinctive user-agent strings.
    >


    As I said "if they don't play by the rules then there is not much you
    can do"

    --
    Take care,

    Jonathan
    -------------------
    LITTLE WORKS STUDIO
    http://www.LittleWorksStudio.com
    Jonathan N. Little, Oct 9, 2006
    #12
  13. Solbu Guest

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    sent the following transmission through subspace:

    > Hello,
    >
    > Was wondering if there is any code to keep search engine bots from
    > picking up my ebay auction pages and archiveing them


    http://www.robotstxt.org/wc/exclusion.html
    http://www.robotstxt.org/wc/norobots.html

    http://www.robotstxt.org/wc/robots.html

    - --
    Solbu - http://www.solbu.net
    Remove 'ugyldig.' for email
    PGP key ID: 0xFA687324
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.2.2 (GNU/Linux)

    iD8DBQFFKcTmT1rWTfpocyQRAmu6AKDgzgI6uSakT72nyLqSNIeSntb5AwCg/eNe
    igCbeKp3JK5mYpTznlZAXtM=
    =Qn2d
    -----END PGP SIGNATURE-----
    Solbu, Oct 9, 2006
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. phl
    Replies:
    0
    Views:
    334
  2. Chris K.
    Replies:
    3
    Views:
    723
    Chris K.
    Jun 8, 2008
  3. SROSeaner

    Meta-Search the Search Engines

    SROSeaner, Sep 15, 2004, in forum: ASP General
    Replies:
    1
    Views:
    134
  4. Grant Steinfeld
    Replies:
    0
    Views:
    115
    Grant Steinfeld
    Oct 28, 2003
  5. Grant Steinfeld
    Replies:
    0
    Views:
    109
    Grant Steinfeld
    Oct 29, 2003
Loading...

Share This Page