how to track unqiue vistors

[Christine Genzer]

Hi,

I wonder - how do tools like Google Analytics differ real unique users
from all those millions of bots, crawlers, and so on?
Is it based on IP ranges, or "Javascript=active", or what????

Thanks in advance,

Christine

 

[Evertjan.]

Christine Genzer wrote on 11 dec 2007 in comp.lang.javascript:

I wonder - how do tools like Google Analytics differ real unique users
from all those millions of bots, crawlers, and so on?
Is it based on IP ranges, or "Javascript=active", or what????

Most bots have
bot,crawler,slurp,netcraft,Jeeves, etc
in their HTTP_USER_AGENT header string.

You can never catch them all, as some, like live.com,
also try to mimic normal browsing behavour,
to see if one feeds them different pages as the "normal" folks,
or so I believe, while coming from the same IP range as the same crawler
visit moments before.

 

[Christine Genzer]

Google analytics and similar use javascript to fetch a url from the
analytics server. Crawlers don't execute javascript so they never touch the
analytics server.

Well, this was my idea too, but is that really true? Everyone says that
crawlers and bots could not execute javascript - but why?! If a browser
can do it, why shouldn't a bot be able to do the same? The harvesters
and spam bots get more and more intelligent I guess - a lot of email
adresses are "hidden" by Javascript - so wouldn't a good spam bot
also execute Javascript? Javascript is source code, instructions just
like html and so on - why could this not be executed?!

Thanks in advance, Christine

 

[Thomas 'PointedEars' Lahn]

Well, this was my idea too, but is that really true? Everyone says that
crawlers and bots could not execute javascript - but why?! If a browser
can do it, why shouldn't a bot be able to do the same?

For simple code, it would need not only a script parser but also a script
engine, i.e. an ECMAScript implementation. For more complex code, it would
also need to implement a AOM/DOM and an engine for that, too. That would by
far outweigh the expected gain of finding URLs in script code. Reasonable
authors also write Web sites that degrade gracefully, so there is not really
a need to crawl scripts.

Please don't multi-post to several JavaScript newsgroups, and have your
keyboard repaired. Thanks in advance.


PointedEars