Google spidering & traffic


C

Cinamon Thunder

Google and other engines spider my website every single day and as
result generates QUITE A BIT of traffic.

I've tried:

<META NAME="revisit-after" CONTENT="7 days">

--but the engines still visit daily.

I dont want to use:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

--as it will prevent spidering all together unless I remove the tag
every week or so for a few days?

I'm not sure about robot.txt as I want the entire site indexed (just not
every single day) and no pages excluded.

Any advice would be appreciated!

Thx
C
 
Ad

Advertisements

B

Brian Cryer

Cinamon Thunder said:
Google and other engines spider my website every single day and as result
generates QUITE A BIT of traffic.

I've tried:

<META NAME="revisit-after" CONTENT="7 days">

Personally, I don't bother with this one. Its a request for robots to
revisit after 7 days, but its only a request. It doesn't mean only revisit
after 7 days. Also, its a hint for the current page. So you are saying for
*this* page *please* visit again after 7 days.
--but the engines still visit daily.

I dont want to use:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

--as it will prevent spidering all together unless I remove the tag every
week or so for a few days?

I'm not sure about robot.txt as I want the entire site indexed (just not
every single day) and no pages excluded.

How many pages to you have? In my experience search engines don't tend to
pull down the same pages regularly (unless they've changed), but if you have
a large site then they can spend days (or more) working through all the
pages on your site.

Regarding google, whilst I don't know a way of saying only spider one day in
seven, you can change the crawl rate. Go to
https://www.google.com/webmasters/sitemaps/ (which redirects you), sign up
for a Google Webmaster tools account. There is a minor hoop to jump through
so they they can validate that you are responsible for the domain you claim
to be responsible for, but thereafter you can change the crawl rate down
from "Normal" to "Slower".

Be aware that this will mean it takes longer for google to pick up changes
and new pages on your sit.e

Can't advise about other robots. Normally about all you can do is block
them, which you probably don't want.

Hope this helps.
 
J

Jukka K. Korpela

Scripsit Cinamon Thunder:
Google and other engines spider my website every single day and as
result generates QUITE A BIT of traffic.

Do you mean the _spider visits_? Even if a spider visits your all pages
every day, that's still just one HEAD request per page. If that's
considerably more than normal traffic to your site, I'd be more worried
about the low usage than the spiders.

(I'm assuming that your server sends adequate Last-Modified headers so that
spiders can just send a HEAD request and see that nothing has changed,
instead of retrieving the actual content.)
<META NAME="revisit-after" CONTENT="7 days">

Important search engines probably ignore that routinely, since there has
been too much abuse.
Any advice would be appreciated!

Please convince me that daily visits by spiders are a _problem_. Many
webmasters would pay real money to achieve such a situation.
 
K

kobus.dp

Here are spider bandwidth stats for the 19th

Googlebot (Google) 262.93 MB 19 Feb 2007 - 23:54
Inktomi Slurp 74.83 MB 19 Feb 2007 - 23:58
MSNBot 270.99 MB 19 Feb 2007 -
23:34
Unknown robot (identified by 'robot') 597.13 KB 19 Feb 2007 - 05:23

Thats 1.2+ Gig for the day?

Yet when I look at my overall summary for the 19th:

Day Number of visits Pages Hits Bandwidth
19 Feb 2007 287 24266 45669 181.29 MB

I smell a rat...

thx
C
 
Ad

Advertisements

S

Steve Pugh

Here are spider bandwidth stats for the 19th

Googlebot (Google) 262.93 MB 19 Feb 2007 - 23:54
Inktomi Slurp 74.83 MB 19 Feb 2007 - 23:58
MSNBot 270.99 MB 19 Feb 2007 -
23:34
Unknown robot (identified by 'robot') 597.13 KB 19 Feb 2007 - 05:23

Thats 1.2+ Gig for the day?

There's a mistake there. Either you typed 597.13 Kb when you meant
597.13 Mb or you read it the other way round,
Yet when I look at my overall summary for the 19th:

Day Number of visits Pages Hits Bandwidth
19 Feb 2007 287 24266 45669 181.29 MB

I smell a rat...

Depends on which flavour of smoke and mirrors your stats use.

Now as you full quoted and top posted you probably didn't read Jukka's
reply carefully, so I'll just draw you attention to this part:

Steve
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top