How to extract text from windows.

[Kelvin (archive)]

Hi, there:

I have seen some program which extracts text from the computer screen. For example, my English-Chinese dictionary, which extracts a English word from the buttons and menus on other applications.

How does this kind of trick work? How do I program them? What are the keyword to search for information on this topic?

Thanks in advance.

Kelvin

 

[Malcolm Dew-Jones]

Kelvin wrote:
: Hi, there:

I have seen some program which extracts text from the computer screen. For example, my English-Chinese dictionary, which extracts a English word from the buttons and menus on other applications.

How does this kind of trick work? How do I program them? What are the keyword to search for information on this topic?

Look at the windows api. Functions such as GetWindow, EnumWindows, GetWindowText, are required.

Given a top level window, recursively scan through the children. (Each control etc is a window.)

All the data about each window is available by examining the various properties of each window (which are available using functions such as GetWindowText).

For the generic classes provided by the windows api, the data has the "obvious" meaning, i.e. the text for a button window is the text shown on the button.

The only problem is when the window uses a custom class to generate the display. Then you have to know how to interpret the data from that window. Most custom classes are the tools provided by a particular development environment, (e.g. MFC) so if you have the same environment then you can interpret the data by simply passing it to the appropriate api functions of that environment.

Actually, for some things you may have to use other functions, GetMenuString comes to mind, perhaps there are others.

 

[Simon]

Your basic technique is as follows:
* Run a timer that regularly gets updates about the window that is below the cursor.
* At each timer interval you will need to use the APIs WindowFromPoint and ChildWindowFromPoint. These let you know the window handle below the cursor.
* From the window handle you can use the GetWindowText to retrieve the text of the button, edit box, static box or whatever similar item you desire.
* Do a dictionary matching and translation of whatever text you got above.

Truth is that most of this is easy to do. See
http://cuinl.tripod.com/Tips/wincaption.htm for an example. The dictionary part may be a bit more difficult depending on whether you are doing a simply word-for-word match or you are actually interpreting a phrase from English to Chinese.

Simon Hayden
www.AutoUpdatePlus.com Get updates to your clients the quick and easy way.