How can I trap UI events non-intrusively?

[John]

We have some Computer Based Training (CBT) software for different GUI
business applications on MS windows OS. We currently have Win32 API
and COM based libraries that let us create training applications that
are tightly integrated with the business applications. We do this by
monitoring users mouse and keyboard actions and contents of windows
controls (fields, check boxes, list boxes, etc.). We do not need
access to the source code of the business applications. Through win32
API's we know the menu choice used by the user, the data value entered
in a GUI field, etc.

Now we have to do the same for Java based GUI applications. I was
wondering if I can get any ideas from folks in this group about how I
can achieve similar results for java GUI's. Is there a way to
monitoring java applications without having access to the source code?
Any help will be highly appreciated.

Sincerely,
John

 

[John C. Bollinger]

We have some Computer Based Training (CBT) software for different GUI
business applications on MS windows OS. We currently have Win32 API
and COM based libraries that let us create training applications that
are tightly integrated with the business applications. We do this by
monitoring users mouse and keyboard actions and contents of windows
controls (fields, check boxes, list boxes, etc.). We do not need
access to the source code of the business applications. Through win32
API's we know the menu choice used by the user, the data value entered
in a GUI field, etc.

Now we have to do the same for Java based GUI applications. I was
wondering if I can get any ideas from folks in this group about how I
can achieve similar results for java GUI's. Is there a way to
monitoring java applications without having access to the source code?
Any help will be highly appreciated.

All mouse and keyboard actions in a Java GUI (AWT and/or Swing)
application produce "Events" for keyboard activity and mouse gestures.
You can create software that will receive copies of all such events:
look into the addAWTEventListener() method of class java.awt.Toolkit.
It will be easiest to use this by means of a custom launcher for the
base application being monitored; connecting to an already-running
application is a harder task because of the insulation provided by the JVM.

Hopefully that will get you started in the right direction.


John Bollinger
(e-mail address removed)

 

[John]

Thanks Cid and John for your responses. It gives me an avenue for
investigation.

I also need to understanding what is displayed on the screen so that I
can create context sensitive instructions.

For example consider an Order Entry application. When the user enters
a product number a product unit price will be displayed (say $5000). I
need to capture that amount (which was not entered by keystrokes from
the user, instead was read from a database) so when the user then
enters a product quantity (say 10) the CBT can display a message like
"Sales clerks cannot authorize a transaction exceeding $10,000.

Will I be able to capture screen output like the one above using
'Events'?

John

 

[Sudsy]

John wrote:

For example consider an Order Entry application. When the user enters
a product number a product unit price will be displayed (say $5000). I
need to capture that amount (which was not entered by keystrokes from
the user, instead was read from a database) so when the user then
enters a product quantity (say 10) the CBT can display a message like
"Sales clerks cannot authorize a transaction exceeding $10,000.

<snip>

Yowsa! Sounds like someone's trying to implement business logic at the
GUI layer! Very scary stuff. If you need this kind of capability then
you might want to look at using JavaScript.
Any reason why you can't do this using standard tools and technologies?
It reads like you're trying to wag the dog by its tail.

 

[John C. Bollinger]

Thanks Cid and John for your responses. It gives me an avenue for
investigation.

I also need to understanding what is displayed on the screen so that I
can create context sensitive instructions.

For example consider an Order Entry application. When the user enters
a product number a product unit price will be displayed (say $5000). I
need to capture that amount (which was not entered by keystrokes from
the user, instead was read from a database) so when the user then
enters a product quantity (say 10) the CBT can display a message like
"Sales clerks cannot authorize a transaction exceeding $10,000.

Will I be able to capture screen output like the one above using
'Events'?

Chances are that yes, you can get the information you need one way or
another. I suggest that you might want to start by writing an event
monitor that you can use simply to watch the flow of AWTEvents produced
by an application. (That will be useful to you in itself.) You may be
able to get much of the kind of info you want directly from the events
themselves, but if you run into a situation where you need more info,
many AWTEvent subtypes carry references to GUI widgets that you can
examine for further information.

The details of how you would obtain any particular datum (such as the
unit price in your example) will vary considerably depending on the
application's GUI and control flow. You'll have to work it out per each
target application. I'd be surprised to hear that it's much different
in the Win32 API world that you're more familiar with.


John Bollinger
(e-mail address removed)

 

[Cid]

Thanks Cid and John for your responses. It gives me an avenue for
investigation.

I also need to understanding what is displayed on the screen so that I
can create context sensitive instructions.

For example consider an Order Entry application. When the user enters
a product number a product unit price will be displayed (say $5000). I
need to capture that amount (which was not entered by keystrokes from
the user, instead was read from a database) so when the user then
enters a product quantity (say 10) the CBT can display a message like
"Sales clerks cannot authorize a transaction exceeding $10,000.

Will I be able to capture screen output like the one above using
'Events'?

Sure. You should be able to pull this off. In win32 you might subclass
the control window to monitor events (WM_SETTEXT) and such.

In Swing, it should (generally) be even easier. In this scenario you'd
probably be adding a DocumentListener to the document of a JTextField
(guessing from your description).

You're going to need to read on up Swing to see how it works and what
to look for. Check out sun's tutorial.