Java and Microsoft Windows

[Peter Olcott]

I hope you won't be offended, but may I ask how old you are?

Here is the actual patent, my age is not relevant to the subject at hand.
http://patft.uspto.gov/netacgi/nph-...50&s1=7046848.PN.&OS=PN/7046848&RS=PN/7046848
Can you answer the questions now?

 

[Stefan Schmiedl]

When the slave application is running as a segregated part of this
multi-megabyte OS it can interface perfectly.

That is the difference between *every platform*
and *current versions of Windows*.

FYI: Many DOS-based applications interfacing directly to the hardware
do *not* run on modern Windows systems any more, as the access has
been severely restricted (already with NT4, IIRC).

Take care,
s.

 

[Peter Olcott]

That is the difference between *every platform*
and *current versions of Windows*.

FYI: Many DOS-based applications interfacing directly to the hardware
do *not* run on modern Windows systems any more, as the access has
been severely restricted (already with NT4, IIRC).

Take care,
s.

By every platform, I mean MS Windows including all MSDOS applications that still
run under MS Windows, also any software that will run under the current versions
of Apple OS, or Linux. Basically any software that will run under any modern GUI
OS.

 

[Peter Olcott]

Not on its own, of course. You'd have to write some code specific to the game
you want to play. If you've written a class that woke up one day and decided
to get its own avatar in Second Life, I want a copy.


Not sure what you mean... Are you talking about Windows 3.1, or Apple DOS
3.1? If you're talking about your program sharing a 286 with actual DOS, my
hat's off to you. I wouldn't want to squeeze the J2SE into low RAM.


That's a pretty bold claim, but if we're talking about automating GUI
functions, I would guess Robot is the most portable engine you're likely to
find.


java.awt is just the package where it lives, which makes sense because it
needs tight integration with the host environment. Sorry, I guess I wasn't
clear enough... Why not read the documentation for yourself?

To know that a system lacks a capability requires a much deeper understanding
than finding out if a system has a capability. This deeper understanding takes a
lot of time. I am already working 85 hours a week, so I find that simply asking
people that already know these things is a much more cost-effective use of my
time.

 

[Peter Olcott]

Er... Like java.awt.Robot?

http://java.sun.com/j2se/1.5.0/docs/api/java/awt/Robot.html

I didn't see anything that enabled it to form anything like the basis for a
universal scripting language in this documentation. Although there is capability
for operating the user interface input hardware, there did not seem to be any
capability to "see" graphical objects on the screen. Capturing the screen is not
at all like intelligently recognizing the current state and location of
graphical user interface control elements. Without this ability, no universal
graphical user interface scripting language would be possible. Perhaps there are
many more deeper levels to this documentation than this link seems to indicate.

 

[Michael Rauscher]

What about ordinary business applications and applets?

Applications: Swing.

Applets: here you have to separate between (I'd say) decorative applets
that are used to beautify homepages and applets that are part of serious
applications. The former are most often implemented using AWT, the
latter use Swing. IMO.

Universal GUI Scripting Language that can "see" anything on the screen by
processing the screen pixels in real time using a deterministic finite
automaton.

Ah, and you want to know if it's possible to do the same using Java? I'd
say 'yes' with respect to the language Java. E.g. it shouldn't be a big
problem using JNI (the interface to the native system) to get window
handles and so on.

Bye
Michael

 

[Peter Olcott]

Applications: Swing.

Applets: here you have to separate between (I'd say) decorative applets that
are used to beautify homepages and applets that are part of serious
applications. The former are most often implemented using AWT, the latter use
Swing. IMO.


Ah, and you want to know if it's possible to do the same using Java? I'd say
'yes' with respect to the language Java. E.g. it shouldn't be a big problem
using JNI (the interface to the native system) to get window handles and so
on.

The technology that I invented can obtain the current location and state of
graphical user interface controls for literally anything. I want to know the
degree to which another set of methods might be able to accomplish this same
result, in other words the degree of competition that I might have in the
future.

Is there any technology(ies) within Java that can always obtain the exact state
and current location of any Java based controls? What is the name of this
technology(ies)

 

[Jeffrey Schwab]

Can you answer the questions now?

I'm trying, and so are several other people, just to try to make the
group welcoming. Your questions are extremely vague though; you're not
making yourself easy to help. Do you still have specific questions
you would like answered?

Btw, your tone comes across as argumentative. It's almost like you're
angry with other people for not appreciating the genius that is Peter
Olcott. Most of the people here are also pretty bright, so you might
want to pretend for a moment that you're speaking with your peers
instead of a bunch of hostile inferiors. Catch more software developers
with honey, to paraphrase.

 

[Tom Cole]

The technology that I invented can obtain the current location and state of
graphical user interface controls for literally anything. I want to know the
degree to which another set of methods might be able to accomplish this same
result, in other words the degree of competition that I might have in the
future.

So what if my Swing application had a picture of a button, but no
actual button? Would it think it was a button?

What if my L&F displayed buttons or textfields with no identifying
marks other than just text? How would your technology know the
difference? Would it think it was just text?

You see, my Swing application creates only a single native OS peer, and
that's a window. There are no other component peers. So you can't get
to my other components via the OS, you can only attempt a screen scrape
and hope that my buttons and textfields, etc. look the way you think a
button should look like.

Is there any technology(ies) within Java that can always obtain the exact state
and current location of any Java based controls? What is the name of this
technology(ies)

None.

 

[Michael Rauscher]

Hi,

Is there any technology(ies) within Java that can always obtain the exact state
and current location of any Java based controls?

Perhaps you're more familiar with C.

So, what would you answer if I asked you the same question with respect
to C? Probably you'd tell me, that C doesn't even know anything of
controls. C is a language. Sure, there are libraries like the MFC but
these aren't part of the language C.

The same applies to Java. There are libraries like AWT, Swing and SWT
but they aren't part of the language Java.

If the Java runtime libraries don't offer what you need, you can use JNI
to write Java native methods. From the JNI specification:

The JNI is a native programming interface. It allows Java code that runs
inside a Java Virtual Machine (VM) to interoperate with applications and
libraries written in other programming languages, such as C, C++, and
assembly.

Bye
Michael

 

[Peter Olcott]

I'm trying, and so are several other people, just to try to make the group
welcoming. Your questions are extremely vague though; you're not making
yourself easy to help. Do you still have specific questions you would
like answered?

Btw, your tone comes across as argumentative. It's almost like you're angry
with other people for not appreciating the genius that is Peter Olcott. Most
of the people here are also pretty bright, so you might want to pretend for a
moment that you're speaking with your peers instead of a bunch of hostile
inferiors. Catch more software developers with honey, to paraphrase.

I haven't had nearly the same degree of experience with my interpersonal
communication skills, as I have had with my software development skills. I don't
mean to seem hostile, and I know that my Java development skills are by far
inferior to most everyone here.

I have about 17,000 hours and most of my life savings invested in my current
project, so I tend to have a little emotional attachment to it. I desperately
want this project to succeed. This may tend to make me seem hostile when someone
brings up the possibility that there are already alternatives that would make
this project useless. If there are alternatives that do make this project
useless, then I need to know about these, yet, these must be very rigorously
analyzed.

The primary benefit that my technology provides is the possibility of a
universal scripting language. To the best of my knowledge there is no other
technology that can possibly provide a scripting language that is nearly as
universal as that provided by my technology. Now is the time that I must
definitively ascertain this point.

I did not come to this forum to definitively ascertain this point. I came to
this forum to validate one single aspect of this point. The single aspect of
this point that I came to validate is the difficulty of providing a scripting
language that could completely automate any possible Java program.

From the best that I can tell, from the expert advice that I received here, even
this tiny task would tend to be difficult. The specific aspect of this task that
I must analyze is the difficulty of determining the current state and exact
location of any graphical user interface control. Since Java has at least three
different technologies for creating these controls, a first estimate from my
limited point of view would be that even this tiny subtask would require a hodge
podge conglomeration of these three technologies. Definitively ascertaining the
precise degree of truth of this last statement would fulfill my whole need for
being here.

 

[Peter Olcott]

So what if my Swing application had a picture of a button, but no
actual button? Would it think it was a button?

It would consider anything at all to be a graphical user interface control iff
(if and only if) it was specifically informed that it is to be considered a
graphical user interface control. If you make a bitmap that is identical to the
bitmap of a legitimate graphical user interface control, then it would be
required to use other contextual information to disambiguate this case.

What if my L&F displayed buttons or textfields with no identifying
marks other than just text? How would your technology know the
difference? Would it think it was just text?

Typically textfields are demarked in one way or another. If you don't somehow
identify a textfield as a text field, then a human user will not know that it is
a text field. If there is enough displayed on the screen for a person to tell
that it is a text field, then various methods can be used so that my technology
would identify it as a text field as well. Generally if its good enough for a
person, then its good enough for my technology.

You see, my Swing application creates only a single native OS peer, and
that's a window. There are no other component peers. So you can't get
to my other components via the OS, you can only attempt a screen scrape
and hope that my buttons and textfields, etc. look the way you think a
button should look like.

It would simply look at your specific button, then (when told) would know that
this bitmap in this context is a button.

None.

That is very good news for me. Let's see how many others concur.

 

[Peter Olcott]

Hi,



Perhaps you're more familiar with C.

So, what would you answer if I asked you the same question with respect to C?
Probably you'd tell me, that C doesn't even know anything of controls. C is a
language. Sure, there are libraries like the MFC but these aren't part of the
language C.

The same applies to Java. There are libraries like AWT, Swing and SWT but they
aren't part of the language Java.

If the Java runtime libraries don't offer what you need, you can use JNI to
write Java native methods. From the JNI specification:

The JNI is a native programming interface. It allows Java code that runs
inside a Java Virtual Machine (VM) to interoperate with applications and
libraries written in other programming languages, such as C, C++, and
assembly.

Bye
Michael

This answer seems to indicate that it might be possible within the set of
everything pertaining to Java to always obtain the exact location and state of
any graphical user interface control.

 

[Michael Rauscher]

This answer seems to indicate that it might be possible within the set of
everything pertaining to Java to always obtain the exact location and state of
any graphical user interface control.

Yes. The only thing I'm in doubt is that it's possible just by using the
standard runtime libraries.

 

[Stefan Schmiedl]

Typically textfields are demarked in one way or another. If you don't
somehow identify a textfield as a text field, then a human user will not
know that it is a text field. If there is enough displayed on the screen
for a person to tell that it is a text field, then various methods can
be used so that my technology would identify it as a text field as well.
Generally if its good enough for a person, then its good enough for my
technology.

Then take a look at Squeak's Morphic framework, please. It contains a
sample of a TextMorph presenting rotated text in an arbitrary shape. A
human will recognize that this is a "text field".

How would you recognize a hyperlink if it was presented without decoration
and only revealed itself upon being touched by the mouse pointer? A human
would learn to sweep the mouse over the page to discover active spots. At
least I have done so, as I tend to click on corporate logos in top web
page corners in the hope of being transferred to the home page.

Contextual information is not rectangular.

s.

 

[Peter Olcott]

Yes. The only thing I'm in doubt is that it's possible just by using the
standard runtime libraries.

I need to have a completely definitive answer before this question has been
sufficiently answered for my needs. For example another respondent answered with
the opposite answer. This means that the criterion measure for an affirmative
answer must be the details of the specific means to accomplish this. Hopefully
someone will provide this, or alternatively most everyone else will confirm that
it can't be done.

 

[Peter Olcott]

Then take a look at Squeak's Morphic framework, please. It contains a
sample of a TextMorph presenting rotated text in an arbitrary shape. A
human will recognize that this is a "text field".

My process is deterministic, only a stochastic process could recognize this sort
of thing, but, with stochastic processes, the 100% reliability and real-time
response rate goes away. My system could recognize any possibly rotated text,
iff it was provided an exact sample of the end-result of the rotation, and
informed of the intended value associated with this pixel pattern.

How would you recognize a hyperlink if it was presented without decoration
and only revealed itself upon being touched by the mouse pointer? A human
would learn to sweep the mouse over the page to discover active spots. At
least I have done so, as I tend to click on corporate logos in top web
page corners in the hope of being transferred to the home page.

Same method for my system. The goal here is a universal scripting language.
Scripts written in this scripting language would be written by people that
understand the nature and details of what is being scripted. Because of this
the author would generally know in advance which links are relevant to the task
at hand, and the pixel pattern associated with these links. Alternatively the
system could scan the whole page noting all of the hotspots from changes to the
pixel patterns displayed on the screen.

 

[Stefan Schmiedl]

I need to have a completely definitive answer before this question has
been sufficiently answered for my needs. For example another respondent
answered with the opposite answer. This means that the criterion measure
for an affirmative answer must be the details of the specific means to
accomplish this. Hopefully someone will provide this, or alternatively
most everyone else will confirm that it can't be done.

There are too many variables in your statement of the problem to
provide a "completely definitive" answer:

"might" be possible -- rubber word -> answer yes, probability
low
"within ... Java" -- a real constraint
to "always" obtain -- my programs kill jvms upon startup
-> answer no
"exact location and "state" -- as in 'enabled' or 'disabled'?
of "any ... control" -- invisible ones, too?

There *are* invisible GUI controls, you know. Screen corners have been
used to dynamically (de)activate screen savers on Macintosh way back
in System 6 days.

s.

 

[Stefan Schmiedl]

My process is deterministic, only a stochastic process could recognize
this sort of thing, but, with stochastic processes, the 100% reliability
and real-time response rate goes away. My system could recognize any
possibly rotated text, iff it was provided an exact sample of the
end-result of the rotation, and informed of the intended value
associated with this pixel pattern.

Speaking of reliability ... have you noticed the last sentence of claim 15
on the web page of your patent? Does your software really "invoice" the
second program?

Same method for my system. The goal here is a universal scripting
language. Scripts written in this scripting language would be written by
people that understand the nature and details of what is being scripted.
Because of this the author would generally know in advance which links
are relevant to the task at hand, and the pixel pattern associated with
these links.

Then what's the advantage to just parsing the HTML? In this specific context?

Alternatively the system could scan the whole page noting
all of the hotspots from changes to the pixel patterns displayed on the
screen.

Do you want to move the mouse over every pixel? That would be about a million
on my screen, and you'd have to *wait* for a possibly slow browser (running
on a multitasked system) to really respond to the event. There goes the real
time response.

Maybe we just have different notions of "universal", Horatio.

s

 

[Michael Rauscher]

I need to have a completely definitive answer before this question has been
sufficiently answered for my needs. For example another respondent answered with
the opposite answer. This means that the criterion measure for an affirmative

I said "yes" to "it might be possible". To me "within everything
pertaining to Java" includes the Java language, the Java Platform and in
particular the JNI, too.

So, if one can provide a DLL that is able to "always obtain the exact
location and state of any graphical user interface control" and one can
use the functions of this DLL by wrapping them with Java native methods,
then it might be possible.

With regard to the "opposite answer": this is most probably the
consequence of different interpretations of your question.

E.g. "within Java" is a very unclear statement to me. It could mean
anything from the language to the platform, with JNI/without JNI...

Bye
Michael