JS Web Robot

[Paul Dennis]

Hi,

I'm trying to write a web robot using JavaScript.
It's objective would be to surf around and look
for patterns in the way web pages link to each
other or in the text they contain. Data would be
returned in a web box which could later be copied
into another application.

That's not to tough a challenge. I can make a
JS application surf around my hard drive or
web site with ease. I simply click an html into
a second window and wait for the document
readyState to be complete, then grab the
document.links array and point the window
at a new location. Off it goes.

But when it tries to surf from my drive to
my web site, or from my web site to another
web site, it gets an error. It crashes the first
time it tries to check the readyState of a
document from a different server.

I think that maybe JS has been designed to foil
attempts to build web robots with it. If so, is there
any way around it? Or maybe I'm just missing a
critical JS detail or two. So, does anyone know
what's going on here? Can anyone help me out?

-Paul Dennis.

 

[Lasse Reichstein Nielsen]

But when it tries to surf from my drive to
my web site, or from my web site to another
web site, it gets an error. It crashes the first
time it tries to check the readyState of a
document from a different server.

I think that maybe JS has been designed to foil
attempts to build web robots with it.

The browser security model has. If you try to access the content of a
page from a different domain, you are stopped - the hard way.

If so, is there any way around it?

Not in any browser, but if it is just your own browser you might be
able to give it extended permissions. If the browser is IE, you can
look into HTML Applications (google for "HTML application HTA").

/L