HTTP tunneling through proxy server

[Alex Molochnikov]

Hello everyone,

Our program connects to the License Generator (the Java-based server running
on our website host) via URLConnection like this:

String _url = "http://gestalt.com/license/";
URL url = new URL(_url);
URLConnection connection = url.openConnection();
connection.setUseCaches(false);
connection.setDoOutput(true);

At the server end, Apache server receives the HTTP request and redirects it
to the License Generator which then responds with the appropriate content.

It always worked, but recently someone complained that the connection,
originating from his laptop, cannot go through the proxy server that his
laptop connects to. Unfortunately, I could not get any detail on this
incident, but it left me wondering: what could possibly go wrong with the
connection?

Should I have used HttpURLConnection class instead? And, for this matter,
when would one use HttpURLConnection over URLConnection ?

TIA

Alex Molochnikov
Gestalt Corporation
www.gestalt.com

 

[Yu SONG]

Hello everyone,

Our program connects to the License Generator (the Java-based server running
on our website host) via URLConnection like this:

String _url = "http://gestalt.com/license/";
URL url = new URL(_url);
URLConnection connection = url.openConnection();
connection.setUseCaches(false);
connection.setDoOutput(true);

At the server end, Apache server receives the HTTP request and redirects it
to the License Generator which then responds with the appropriate content.

It always worked, but recently someone complained that the connection,
originating from his laptop, cannot go through the proxy server that his
laptop connects to. Unfortunately, I could not get any detail on this
incident, but it left me wondering: what could possibly go wrong with the
connection?

Should I have used HttpURLConnection class instead? And, for this matter,
when would one use HttpURLConnection over URLConnection ?

TIA

Alex Molochnikov
Gestalt Corporation
www.gestalt.com

You'd better add some options to let the user select his connection(direct,
http proxy, Socks ...) and set appropriate properties in the "System".

 

[Alex Molochnikov]

You'd better add some options to let the user select his connection(direct,
http proxy, Socks ...) and set appropriate properties in the "System".

That would be nice, but the HTTP tunneling is supposed to be the lowest
common denominator that works through firewalls (as long as they let
outgoing connections on port 80) and that is also platform-independent.

So, my original question still stands: what could cause the URLConnection to
fail when going though a proxy server?

Alex.

 

[Roger]

So, my original question still stands: what could cause the URLConnection to
fail when going though a proxy server?

Authentication by the proxy? The proxy I'm forced to use at work
rquires Windows NTLM Authentication which requires additional
considerations when establishing the connection.

Regards
Roger

 

[iksrazal]

That would be nice, but the HTTP tunneling is supposed to be the lowest
common denominator that works through firewalls (as long as they let
outgoing connections on port 80) and that is also platform-independent.

So, my original question still stands: what could cause the URLConnection to
fail when going though a proxy server?

Alex.

NTLM for one. As I understand it, java 1.4.x only supports NTLM on
Windows. You might try HttpClient from Apache Commons as an
alternative to URLConnection - it has more features, ntlm works
everywhere, and I for one have had a lot of luck with.

iksrazal

 

[Chris Smith]

So, my original question still stands: what could cause the URLConnection to
fail when going though a proxy server?

Lots of possibilities.

Ditto the suggestion on HttpClient. That's going to end up being pretty
much a lifesaver if you intend to get this to work really reliably.

That said, I still recall fighting proxy issues back when we first
released the Design-a-Course client, and the hoops we jumped through
trying to make that work reliably. Some things to keep in mind:

* Many firewalls are transparent, but many more are not. A LOT of
popular outgoing firewall software products require use of a proxy
server. You will need to provide the option to configure a proxy
server.

* Firewall companies do the darnedest things in the name of security.
We found stuff as ludicrous as filtering outgoing requests based on the
User-Agent header, assuming that anything that doesn't claim to be
Mozilla *must* be malicious, right?. So we had to add an option to fake
the user-agent header as a popular browser (I copied from IE 6).

* There are two related standards for auto-detection or auto-
configuration of proxies. They are called, respectively, PAC and WPAD.
They are absolutely silly standards, and you'll need to embed a
JavaScript interpreter into your application to implement either one.
However, if you don't do so, your application will mysteriously fail and
your user won't even be at all aware that they need to configure a
proxy! So they'll blame you, of course.

* A lot of proxies have trouble with certain combinations of HTTP
features. We found very frequent problems with combining HTTPS and host
authentication; with combining host authentication with proxy
authentication; and with pretty much any use of the 101-continue header.
The latter can IIRC be disabled with current versions of HttpClient, and
you should do so unless you have a good reason not to. If you need to
do so with an older HttpClient version (though I can't imagine why...),
I've got some code around somewhere to subclass some of the method
classes for that purpose, so just ask and it's yours.

* Proxy and firewall vendors will always blame your application or the
client's configuration (even if the client never touched the default
configuration since installation). They will never try to solve your
problem. The presumption is that you're at fault.

That's all I can remember of this fiasco right now.

--
www.designacourse.com
The Easiest Way to Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation

 

[Alex Molochnikov]

I was somewhat distracted by the MySQL licensing debate, and could not
respond to this sooner.

Lots of possibilities.

Ditto the suggestion on HttpClient. That's going to end up being pretty
much a lifesaver if you intend to get this to work really reliably.

Are you referring to Yu Song's post? Rather than trying to second-guess the
user's environment, give him a way to specify the proxy server name, port
and possibly the authentication data?

That said, I still recall fighting proxy issues back when we first
released the Design-a-Course client, and the hoops we jumped through
trying to make that work reliably. Some things to keep in mind:

* Many firewalls are transparent, but many more are not. A LOT of
popular outgoing firewall software products require use of a proxy
server. You will need to provide the option to configure a proxy
server.

* Firewall companies do the darnedest things in the name of security.
We found stuff as ludicrous as filtering outgoing requests based on the
User-Agent header, assuming that anything that doesn't claim to be
Mozilla *must* be malicious, right?. So we had to add an option to fake
the user-agent header as a popular browser (I copied from IE 6).

* There are two related standards for auto-detection or auto-
configuration of proxies. They are called, respectively, PAC and WPAD.
They are absolutely silly standards, and you'll need to embed a
JavaScript interpreter into your application to implement either one.
However, if you don't do so, your application will mysteriously fail and
your user won't even be at all aware that they need to configure a
proxy! So they'll blame you, of course.

* A lot of proxies have trouble with certain combinations of HTTP
features. We found very frequent problems with combining HTTPS and host
authentication; with combining host authentication with proxy
authentication; and with pretty much any use of the 101-continue header.
The latter can IIRC be disabled with current versions of HttpClient, and
you should do so unless you have a good reason not to. If you need to
do so with an older HttpClient version (though I can't imagine why...),
I've got some code around somewhere to subclass some of the method
classes for that purpose, so just ask and it's yours.

* Proxy and firewall vendors will always blame your application or the
client's configuration (even if the client never touched the default
configuration since installation). They will never try to solve your
problem. The presumption is that you're at fault.

That's all I can remember of this fiasco right now.

I doubt if we can find automated solution(s) for all these circumstances.

One possible way out of this situation for us could be telling the users to
contact us by e-mail when everything else fails.

Thank you for the input.

Regards,

Alex.

 

[Marat]

Check this site out (www.jproxy.com). They claim that you can basically work
with any J2EE API and the calls are automatically HTTP tunneled for you.

Cheers!