How to "spoof" a server into believing your app is a browser ?

[Yevgeny Pozin]

I finally managed to connect to the HTTPS site I was struggling with, but
got told by the server to go away since I don't look like a browser.

Is there a way to spoof the server into believing I am a browser and support
javascript ? (My app has no need of any JS or HTML, just need to extract a
single line from the page)

 

[Jarmo]

I finally managed to connect to the HTTPS site I was struggling with, but
got told by the server to go away since I don't look like a browser.

Is there a way to spoof the server into believing I am a browser and support
javascript ? (My app has no need of any JS or HTML, just need to extract a
single line from the page)

Could you capture the traffic between client and HTTPS web-site using
something like Ethereal and then emulate that in your servlet?

 

[Roy Epperson]

I can't remember where the code I wrote is at the moment; but in your HTTP
request header include the User-Agent request-header field and identify your
program as a known browser.

I used the set of classes to go after a number of different websites to find
data....

 

[Roy Epperson]

Whoops forgot the spec http rfc2616

 

[Roedy Green]

Is there a way to spoof the server into believing I am a browser and support
javascript ? (My app has no need of any JS or HTML, just need to extract a
single line from the page)

Marty Hall's book has many examples of doing this sort of thing.
See http://mindprod.com/jgloss/who.html

Also snooping on a real browser is instructive. See
http://mindprod.com/jgloss/sniffer.html

 

[Marko Lahma]

Is there a way to spoof the server into believing I am a browser and support

javascript ? (My app has no need of any JS or HTML, just need to extract a
single line from the page)

A neat library for browser imitation:
http://jakarta.apache.org/commons/httpclient/

-Marko

 

[Anonymous]

A neat library for browser imitation:
http://jakarta.apache.org/commons/httpclient/

-Marko

I agree - I found HttpClient to be very useful if you need to support
cookies or
anything else past simple HTTP.

 

[Mike Baranczak]

I finally managed to connect to the HTTPS site I was struggling with, but
got told by the server to go away since I don't look like a browser.

Is there a way to spoof the server into believing I am a browser and support
javascript ? (My app has no need of any JS or HTML, just need to extract a
single line from the page)

I don't know what kind of logic the website in question uses, but my
guess is you just need to add a "User-agent" header to your HTTP
request.

I don't have the URLs at the moment, but there are several pages on the
web which list the User-agent strings for various popular browsers.

-MB

 

[Yevgeny Pozin]

thanks

extract

I agree - I found HttpClient to be very useful if you need to support
cookies or
anything else past simple HTTP.




----== Posted via Newsfeed.Com - Unlimited-Uncensored-Secure Usenet News==----
http://www.newsfeed.com The #1 Newsgroup Service in the World! >100,000 Newsgroups
---= 19 East/West-Coast Specialized Servers - Total Privacy via Encryption

=---