Javascript and Microsoft Windows

[Dag Sunde]

I have almost no understanding of JavaScript.

http://patft.uspto.gov/netacgi/nph-...50&s1=7046848.PN.&OS=PN/7046848&RS=PN/7046848

This technology enables a truly universal GUI scripting language to
be created. I am attempting to estimate how difficult it would be to
approximate the functional benefits of my technology using
alternative means.

Peter...

As far as I understand it, you have written some kind of scripting
language coupled with a "screen scraping method" that can analyze
the visible "controls" of any other application ,from MS-DOS to native
windows applications, and expose those to your scripting language.

And now you're asking if this is possible to achieve in javascript
(And Java, in another NG)...?

JavaScript is not a stand-alone programming language. It always
runs in a "Host".

When it runs in the Windows Scripting Host, it can access
and manipulate the objects and properties exposed to it by the WSH.
There is no objects exposed by the WSH that enables javascript
to analyze the visual state other applications external to the WSH.

When running in a webbrowser, it can access the objects and properties
exposed to it by the webbrowser.
There is a *huge* object model exposed to JS by the webbrowser that
let JS analyze an manipulate the visual state of the document loaded in
the webbrowser hosting it. But not any applications external to the
browser.

So rest assured! JS will/can *never* be used to develop something
that can compete with what your patent claims to do.

 

[Richard Cornford]

Dag Sunde wrote:

When it runs in the Windows Scripting Host, it can access
and manipulate the objects and properties exposed to it by
the WSH. There is no objects exposed by the WSH that
enables javascript to analyze the visual state other
applications external to the WSH.

WSH allows Windows programs to be executed by scripts, so RUNDLL.exe,
and so any method from any DLL (at least in theory), with the
(significant) practical limitation that JScript does not have the data
types to provide appropriate arguments to many DLL calls. As a result
there is a chance that WSH does expose enough to allow JScirpt tot
analyze the visual state of other applications (particularly if
provided with a suitable custom DLL).

Richard.

 

[The Magpie]

The United States represents about half of the world software market,
so that patent has good coverage, in half the world.

Interestingly, I cannot find a survey of global software market value.
However, a number of websites do record global software sales numbers
but not value. From that, it appears that Europe is approximately 25% of
the software sales, China and the Far East about 45%, Japan around 12%
and the USA around 18%. These do fit in well with population
demographics so I tend to think they may be accurate, although one of
the websites did point out that it is thought up to 80% of the China and
Far East software may be pirated.

The rest of the world must still legally respect the copyright.

All signatories of the Berne Convention must respect a Berne Convention
copyright, of course, but that is nothing whatever to do with a patent.
As I did point out, a patent is to protect the *invention* but a
copyright is to protect the *expression* (or implementation) of a novel
idea in a specific form.

My original question can not be rephrased. What I really need to know is
exactly how difficult it is for another program to determine the
exact location and current state of any graphical user interface
controls. Someone told me that this is pretty easy using Ajax.

What you are asking makes several assumptions about the computer system
on which the software is running. Without knowing those assumptions, it
is impossible to answer.

If you do not make *any* assumptions then the answer is simple - you
can't do it (for instance, my own BBC Micro / adapted PDP-8 system -
self-built, self-designed and self-installed with my own software will
simply not report to other programmes).

 

[Peter Olcott]

Interestingly, I cannot find a survey of global software market value.
However, a number of websites do record global software sales numbers
but not value. From that, it appears that Europe is approximately 25% of
the software sales, China and the Far East about 45%, Japan around 12%
and the USA around 18%. These do fit in well with population
demographics so I tend to think they may be accurate, although one of
the websites did point out that it is thought up to 80% of the China and
Far East software may be pirated.

I derived my estimates from worldwide computer sales figures. The United States
have a much higher per capita computer usage than any other nation.

All signatories of the Berne Convention must respect a Berne Convention
copyright, of course, but that is nothing whatever to do with a patent.
As I did point out, a patent is to protect the *invention* but a
copyright is to protect the *expression* (or implementation) of a novel
idea in a specific form.

It looks like the best way to protect my rights world wide is penetration
pricing, pricing too low for competition to take hold.

What you are asking makes several assumptions about the computer system
on which the software is running. Without knowing those assumptions, it
is impossible to answer.

When anyone asks the question "Is X possible?" this logically entails the
universal set of every possible assumption. How about this, under any possible
assumption to what extent is it possible to make a universal scripting language
that can control every program that will run under modern operating system
versions?

 

[Martin Honnen]

Dag Sunde wrote:

JavaScript is not a stand-alone programming language. It always
runs in a "Host".

When it runs in the Windows Scripting Host, it can access
and manipulate the objects and properties exposed to it by the WSH.
There is no objects exposed by the WSH that enables javascript
to analyze the visual state other applications external to the WSH.

Many Windows applications like Internet Explorer or Word or Excel can be
automated by script run with WSH. The Windows Shell itself can be
automated e.g. the following script grabs all open Internet Explorer
browser windows and outputs details about the document title and the
number of forms (document.forms.length):

var shell = new ActiveXObject('Shell.Application');
var windows = shell.Windows();
for (var i = 0, l = windows.Count; i < l; i++) {
try {
var document = windows(i).document;
WScript.Echo("document.title: \"" + document.title +
"\"; forms.length: " + document.forms.length);
}
catch (e) {}
}

Thus J(ava)Script (or VBScript) used with WSH can get details about the
visual state of Windows application that expose automation interfaces.

 

[Matt Kruse]

How about this, under
any possible assumption to what extent is it possible to make a
universal scripting language that can control every program that will
run under modern operating system versions?

Completely and totally impossible. IMO.

You would either need the API to every application available (impossible) or
you would need to write software that could act as a user and intelligently
locate controls and content within any application. This would require that
your application be able to visually recognize a GUI like a human does,
since not all GUI's will look the same or even use standard OS components.
Further, even if an app uses standard OS components, those can be styled
differently depending on users's skins or OS settings.

Your patent application is long, and I didn't read a single sentence of it.
But if what you describe above is your goal, then
a) It is not achievable (IMO)
b) It certainly has nothing to do with javascript

You may wish to pursue this in some microsoft newsgroups or general
application development groups.

 

[Peter Olcott]

Dag Sunde wrote:



Many Windows applications like Internet Explorer or Word or Excel can be
automated by script run with WSH. The Windows Shell itself can be automated
e.g. the following script grabs all open Internet Explorer browser windows and
outputs details about the document title and the number of forms
(document.forms.length):

var shell = new ActiveXObject('Shell.Application');
var windows = shell.Windows();
for (var i = 0, l = windows.Count; i < l; i++) {
try {
var document = windows(i).document;
WScript.Echo("document.title: \"" + document.title +
"\"; forms.length: " + document.forms.length);
}
catch (e) {}
}

Thus J(ava)Script (or VBScript) used with WSH can get details about the visual
state of Windows application that expose automation interfaces.

The last four words forming the most significant limitations.

 

[Tom Cole]

Ask Peter Olcott, he'll give you an entirely different answer, as he
claims he's made one.

 

[Peter Olcott]

Completely and totally impossible. IMO.

You would either need the API to every application available (impossible) or
you would need to write software that could act as a user and intelligently
locate controls and content within any application. This would require that
your application be able to visually recognize a GUI like a human does, since
not all GUI's will look the same or even use standard OS components. Further,
even if an app uses standard OS components, those can be styled differently
depending on users's skins or OS settings.

Yet if a scripting language could see anything one the screen by matching the
pixel patterns, then the script programmer could easily adjust for all of these
different looks on a per script basis. As for the different user settings of the
GUI, the script host could adjust for these, and the script programmer would
simply refer to them by their conventional names, menu text, icon text, et
cetera.

Your patent application is long, and I didn't read a single sentence of it.
But if what you describe above is your goal, then
a) It is not achievable (IMO)
b) It certainly has nothing to do with javascript

You may wish to pursue this in some microsoft newsgroups or general
application development groups.

I am doing that concurrently.

 

[Dag Sunde]

Dag Sunde wrote:



Many Windows applications like Internet Explorer or Word or Excel can
be automated by script run with WSH. The Windows Shell itself can be
automated e.g. the following script grabs all open Internet Explorer
browser windows and outputs details about the document title and the
number of forms (document.forms.length):

var shell = new ActiveXObject('Shell.Application');
var windows = shell.Windows();
for (var i = 0, l = windows.Count; i < l; i++) {
try {
var document = windows(i).document;
WScript.Echo("document.title: \"" + document.title +
"\"; forms.length: " + document.forms.length);
}
catch (e) {}
}

Thus J(ava)Script (or VBScript) used with WSH can get details about
the visual state of Windows application that expose automation
interfaces.

You can get the information the application choose to expose thru its
automation interface, nothing more.

The point of my previous post was to illustrate that while JS can,
in some cases give you limited information about some applications
it can *not* be used to take total control over all types of apps
possibly running on a pc.

(And that's what the OP claims his system can do).

The question here is: "Can Javascript be used to analyze the current
state of every visual control the user can interact with, in every
application, regardless og automation interface, and interact with it?

The answer is no!

 

[Peter Olcott]

You can get the information the application choose to expose thru its
automation interface, nothing more.

The point of my previous post was to illustrate that while JS can,
in some cases give you limited information about some applications
it can *not* be used to take total control over all types of apps
possibly running on a pc.

(And that's what the OP claims his system can do).

The question here is: "Can Javascript be used to analyze the current
state of every visual control the user can interact with, in every
application, regardless og automation interface, and interact with it?

The answer is no!

I came here for the purpose of the question turned around. Can any possible
program take complete control of every possible JavaScript application? That was
my reason for being here.

 

[Stephen Kellett]

You said earlier on you have patented this idea. What is the patent
number. Then perhaps we can work out what you want to do from examining
the patent?

Since you have it patented, it is in your interests to disclose the
patent number and any relevant information.

Stephen

 

[Randy Webb]

Peter Olcott said the following on 8/16/2006 4:19 PM:

Yet if a scripting language could see anything one the screen by matching the
pixel patterns, then the script programmer could easily adjust for all of these
different looks on a per script basis.

Matching pixel patterns won't work. You can't tell the difference
between a screen shot of an input button and an input button based on
pixel patterns.

 

[Stephen Kellett]

http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL
&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=1&f=G&l=50&s1=7046848.PN.&OS=PN
/7046848&RS=PN/7046848

This technology enables a truly universal GUI scripting language to be
created.

But because you have patented it, no one will use it. Take a good hard
look at all the languages in use today. They were created and provided
to use, no charge. Some implementations of some languages charge, but
the implementation creators are not charged by the creators of the
language.

Consider the Fractal Image Format - a lossy, highly compressed image
format created in the early 1990s. Royalties required for use. Where is
it now? Consigned to the "won't pay for that" dustbin because suitable
other alternatives existed for free (JPEG, PNG...).

If you think people are going to pay you to implement your universal
scripting language I am afraid you are in for a serious reality check.

Patent something that people need. Like a cheap, pollution free means of
energy creation.

Stephen

 

[Stephen Kellett]

Yet if a scripting language could see anything one the screen by matching the
pixel patterns, then the script programmer could easily adjust for all
of these
different looks on a per script basis. As for the different user
settings of the
GUI, the script host could adjust for these, and the script programmer would
simply refer to them by their conventional names, menu text, icon text, et
cetera.

Peter,

Screen Readers and other assistive technology for vision impaired and
blind computer users have been doing this for years. Also, many
automated user-interaction test software suites (such as WinRunner), on
various computer operating systems have also been trying to do this
resolution-independent method.

None of them, I repeat, none of them have succeeded. They've been at it
longer than you and I guess know more about the problems associated with
these particular topics than you do. They all have solutions that work
some of the time, or most of the time, at a level acceptable by their
customer base. None of them are 100% accurate AND are resolution and GUI
independent.

You are wasting your time with this patent.

Stephen

 

[Stephen Kellett]

It depends on the browser and version you use. Some place a real Win32
window with button class on the page, others just place an image that
"looks" like the win32 button. Others just place a grey rectable with
text as a button.

Agreed. During 1996-1998 I wrote a web browser / multimedia authoring
suite. I chose to use a Windows button. Firefox renders its own buttons
- they are not Windows buttons. No idea what IE, Opera or Safari do.

Stephen

 

[Stephen Kellett]

I came here for the purpose of the question turned around. Can any possible
program take complete control of every possible JavaScript application?
That was
my reason for being here.

Define control. I can write you an application that can monitor a
JavaScript application running inside Firefox or Flock on Windows, but
not on Linux or MacOS and not in Safari, Internet Explorer, Opera, etc.

I know I can do the above, because I was part of a team that has done
the above. See softwareverify.com.

Even then, Define "Control". I can pause it, force garbage collections,
even gather various statistics and some of the time see variable values.

Make it do something else? No.
Find out what is on the screen? No.

Stephen

 

[Peter Olcott]

You said earlier on you have patented this idea. What is the patent number.
Then perhaps we can work out what you want to do from examining the patent?

Here is a link to the patent itself.
http://patft.uspto.gov/netacgi/nph-...50&s1=7046848.PN.&OS=PN/7046848&RS=PN/7046848

 

[Peter Olcott]

Peter Olcott said the following on 8/16/2006 4:19 PM:

Matching pixel patterns won't work. You can't tell the difference between a
screen shot of an input button and an input button based on pixel patterns.

But, when is this ever a problem is actual GUI scripting?

 

[Peter Olcott]

But because you have patented it, no one will use it. Take a good hard look at
all the languages in use today. They were created and provided to use, no
charge. Some implementations of some languages charge, but the implementation
creators are not charged by the creators of the language.

Consider the Fractal Image Format - a lossy, highly compressed image format
created in the early 1990s. Royalties required for use. Where is it now?
Consigned to the "won't pay for that" dustbin because suitable other
alternatives existed for free (JPEG, PNG...).

If you think people are going to pay you to implement your universal scripting
language I am afraid you are in for a serious reality check.

(1) There are three different products that can be implemented with this
technology.
A universal scripting language and a generic software component are two.
(2) I am going for market penetration pricing so the price of the software will
be really affordable.
(3) I am implementing the original Borland marketing model, top quality, and
very low prices.