Java API to control a Real Web Browser

[think300]

Hi Everyone,

I know this has been asked before but most of the threads are old so I
am hopping that there is new solutions, because I have tried Apache
Common's HttpClient and HTTPUNIT on SourceForge which seem to work for
easy web pages, but any thing with JavaScript which gets fired off
before a Form submit is processed tends to give them both trouble. I
would love to just have a Java API which would allow my to print on
"http://mytestsite.com", fill in "id" and "password" and click on
login, let the Java API just follow through all of the redirect and
additional JavaScript gyrations which this site go through and just
eventually end up on the correct page like I was doing it my self with
a web browser. At that point I could grab the web page source from the
Java API and then pull out the data that I was looking for. Again
Apache's HttpClient and HTTPUNIT seems to work on simple forms but not
with any web pages with complex redirects and javascript processing
before the final page is hit. At this point unless someone else has a
better idea I just want to have a Java program bring up FireFox,
simulate my key strokes, and then eventually let me pull the page data
back for processing. At this point I am at a loss. I just figure
there has got to be someone out there doing something similar or they
have heard of a project that is doing this?

Any help would be greatly appreciated,

Thanks,
Roger B.

 

[kevins]

Hi Everyone,
I
would love to just have a Java API which would allow my to print on
"http://mytestsite.com", fill in "id" and "password" and click on
login, let the Java API just follow through all of the redirect and
additional JavaScript gyrations which this site go through and just
eventually end up on the correct page like I was doing it my self with
a web browser.

Hi,

Have you thought about using some sort of bridge between COM objects
and Java?

See http://www.alphaworks.ibm.com/tech/dtjcb

If you can access a Winxp COM object, you can create your own instance
of Internet Explorer. After that, you should be able to click buttons
etc.

Just a suggestion (haven't tried it myself).

K Saunders

 

[think300]

Hi,

Have you thought about using some sort of bridge between COM objects
and Java?

See http://www.alphaworks.ibm.com/tech/dtjcb

If you can access a Winxp COM object, you can create your own instance
of Internet Explorer. After that, you should be able to click buttons
etc.

Just a suggestion (haven't tried it myself).

K Saunders

I did see mention of that in a couple of old threads but to be honest
it looked beyond my skill set to use the COM bridge. It is frustrating
because I actually had this working with the common's HTTPCLIENT api
for months, but then the site I was trying to pull information from
changed and after hours of tinkering with what HTTPCLIENT could do I
could never get it working again (specifically because of some weird
things they are doing with embedded JavaScript and Cookie
manipulation).

-Roger

 

[Andrew Thompson]

...I
would love to just have a Java API which would allow my to print on
"http://mytestsite.com", fill in "id" and "password" and click on
login, let the Java API just follow through all of the redirect and
additional JavaScript gyrations which this site go through and just
eventually end up on the correct page like I was doing it my self with
a web browser.

Embed one?
<https://jdic.dev.java.net/nonav/documentation/javadoc/jdic/org/jdesktop/jdic/browser/WebBrowser.html>

Andrew T.