J2ME, HttpConnection and Proxy

[Marko]

I'm having big trouble with a small J2ME app. In a nutshell.. MIDlet
just doesn't want to pass HTTP requests through a proxy.

Needless to say I've set up correctly all proxy's parameters in the
Toolkit's preferences. As a matter of fact, I'm using the same proxy
to post this message.

I'm using standard J2ME code for sending HTTP GET and POST requests.
Nothing fancy. Also, my proxy doesn't require authentication.

To make the matters worse, the same small J2ME app. won't work on my
Siemens S55 either. I've defined correctly Telecom's proxy parameters
in my mobile phone. For example, phone's WAP browser works perfectly
while MIDlet can't send a simple HTTP request.

This is the code I'm using:

import javax.microedition.midlet.*;
import javax.microedition.lcdui.*;
import javax.microedition.io.*;
import java.io.*;

....

private void loadText(String sURL) {

HttpConnection c;
DataInputStream s;

String strMethodName = "loadText().";

try {
c = (HttpConnection)Connector.open(sURL, Connector.READ);
} catch (Exception ex) {
System.out.println("Error: " + strMethodName + "1");
System.out.println(ex.toString());
return;
}

try {
c.setRequestProperty("Content-Type","text/html"); // not neccessary
c.setRequestProperty("Content-Language", "en-US"); // not neccessary
c.setRequestProperty("Accept", "*/*"); // not neccessary
c.setRequestMethod(HttpConnection.GET);
} catch (Exception ex) {
System.out.println("Error: " + strMethodName + "2");
System.out.println(ex.toString());
return;
}

try {
s = new DataInputStream(c.openDataInputStream());
} catch (Exception ex) {
System.out.println("Error: " + strMethodName + "3");
System.out.println(ex.toString());
return;
}

try {
s.close();
c.close();
}
catch (Exception ex) {
System.out.println("Error: " + strMethodName + "4");
System.out.println(ex.toString());
return;
}

}


I know the code is OK because when I'm using direct connection to the
Internet, without proxy, J2ME HTTP request works just fine. I can send
GET and POST requests without any problems.

Error always occurs in:
s = new DataInputStream(c.openDataInputStream());

The error description is:
Exception: java/lang/ClassNotFoundException
Initializing class: 'com/sun/cldc/io/NetworkConnectionBase'
Initializing class: 'com/sun/cldc/io/j2me/socket/Protocol'
Exception: java/lang/ClassNotFoundException
Exception: java/io/IOException
Exception: java/io/IOException
Error: loadText().3
java.io.IOException: Error initializing HTTP tunnel connection:
HTTP/1.0 403 Forbidden
Server: squid/2.5.STABLE1
Mime-Version: 1.0
Date: Mon, 11 Oct 2004 11:13:21 GMT
Content-Type: text/html
Content-Length: 1053
Expires: Mon, 11 Oct 2004 11:13:21 GMT
X-Squid-Error: ERR_ACCESS_DENIED 0
X-Cache: MISS from wonderland.hznet.hr
X-Cache-Lookup: NONE from wonderland.hznet.hr:8001
Proxy-Connection: close
http://www.google.com/index.html


I've browsed through Google groups several times but I haven't found
any meaningful answers to this conundrum.

Please help!

Marko

 

[Yu SONG]

...I know the code is OK because when I'm using direct connection to the
Internet, without proxy, J2ME HTTP request works just fine. I can send
GET and POST requests without any problems.

Error always occurs in:
s = new DataInputStream(c.openDataInputStream());

The error description is:
Exception: java/lang/ClassNotFoundException
Initializing class: 'com/sun/cldc/io/NetworkConnectionBase'
Initializing class: 'com/sun/cldc/io/j2me/socket/Protocol'
Exception: java/lang/ClassNotFoundException
Exception: java/io/IOException
Exception: java/io/IOException
Error: loadText().3
java.io.IOException: Error initializing HTTP tunnel connection:
HTTP/1.0 403 Forbidden
Server: squid/2.5.STABLE1...

Before "Error: loadText().3", there have been few lines of errors
already. And you received response from proxy server, which means the
request reached the server.


--
Song

/* E-mail.c */
#define User "Yu.Song"
#define At '@'
#define Warwick "warwick.ac.uk"
int main() {
printf("Yu Song's E-mail: %s%c%s", User, At, Warwick);
return 0;}

Further Info. : http://www.dcs.warwick.ac.uk/~esubbn/
_______________________________________________________

 

[Marko]

Before "Error: loadText().3", there have been few lines of errors
already. And you received response from proxy server, which means the
request reached the server.


--
Song

/* E-mail.c */
#define User "Yu.Song"
#define At '@'
#define Warwick "warwick.ac.uk"
int main() {
printf("Yu Song's E-mail: %s%c%s", User, At, Warwick);
return 0;}

Further Info. : http://www.dcs.warwick.ac.uk/~esubbn/
_______________________________________________________

You are 100% right. Before the main exception (main error) in:
s = new DataInputStream(c.openDataInputStream());

I'm getting:
Exception: java/lang/ClassNotFoundException
Initializing class: 'com/sun/cldc/io/NetworkConnectionBase'
Initializing class: 'com/sun/cldc/io/j2me/socket/Protocol'
Exception: java/lang/ClassNotFoundException
Exception: java/io/IOException
Exception: java/io/IOException

But I just can't pinpoint where these exceptions take place and what
is causing them. In my question here I have posted the entire trace
after I invoke my public void loadText(String) function.

Any ideas or hunches?

Thanks,
Marko

 

[Darryl L. Pierce]

Marko wrote:

You are 100% right. Before the main exception (main error) in:
s = new DataInputStream(c.openDataInputStream());

Why are you doing the above? You're unnecessarily creating objects that will
only slow down your system with garbgae collection. Try:

s = c.openDataInputStream();

You don't need to wrap the DataInputStream returned with *another*
DataInputStream. That's awaste of resources.

--
/**
* @author Darryl L. Pierce <[email protected]>
* @see The Infobahn Offramp <http://mcpierce.mypage.org>
* @quote "Lobby, lobby, lobby, lobby, lobby, lobby..." - Adrian Monk
*/

 

[Marko]

OK. Right.

But I'm afraid that I'm still getting the same error from the MIDlet:
"proxy cannot initialize HTTP tunnel connection".

I've changed the line you specified, and also I've added some old
fashioned tracing used before the invention of trace levels. And as
you can see in the output, four exceptions (2
java/lang/ClassNotFoundException and 2 java/io/IOException) are raised
by calling:
s = c.openDataInputStream();

J2ME code in question:
--------------------------------------------------------
sURL = "http://www.google.com/index.html";

System.out.println(strMethodName + "1");

try {
c = (HttpConnection)Connector.open(sURL, Connector.READ);
} catch (Exception ex) {
System.out.println("Error: " + strMethodName + "1");
System.out.println(ex.toString());
return;
}

System.out.println(strMethodName + "2");

try {
s = c.openDataInputStream();
} catch (Exception ex) {
System.out.println("Error: " + strMethodName + "3");
System.out.println(ex.toString());
return;
}

System.out.println(strMethodName + "3");
--------------------------------------------------------

And again, the output:
--------------------------------------------------------
ucitajTekst().1
ucitajTekst().2
Exception: java/lang/ClassNotFoundException
Exception: java/lang/ClassNotFoundException
Exception: java/io/IOException
Exception: java/io/IOException
Error: ucitajTekst().3
java.io.IOException: Error initializing HTTP tunnel connection:
HTTP/1.0 403 Forbidden
Server: squid/2.5.STABLE1
Mime-Version: 1.0
Date: Thu, 14 Oct 2004 10:59:09 GMT
Content-Type: text/html
Content-Length: 1053
Expires: Thu, 14 Oct 2004 10:59:09 GMT
X-Squid-Error: ERR_ACCESS_DENIED 0
X-Cache: MISS from wonderland.hznet.hr
X-Cache-Lookup: NONE from wonderland.hznet.hr:8001
Proxy-Connection: close
http://www.google.com/index.html
--------------------------------------------------------

I've event tried fooling around with J2ME Toolkit's Preferences. I've
typed in different (wrong) proxy port numbers. The result is I had one
java/io/IOException and HTTPConnection wasn't established. Then I've
typed in the Preferences the address of the Web server itself and port
80 (port of the HTTP protocol) and still no budge.

I'd like to reiterate that I can establish HTTP connections in Visual
Studio .NET, Visual Basic, ANSI C++ using standard libraries in the
same (network) environment I'm trying to use MIDlet and Sun's J2ME
emulator. MIDlet can't establish HTTP connection throught a proxy,
while other technologies can.

Thanks again for any help,
Marko

 

[Bruno Grieder]

Marko,

I do not know about J2ME but I can testify you can establish an HTTP and
HTTPS connection through a squid proxy using a jdk 1.4.

With a 403, it sounds to me that you have:
-either your squid proxy preventing you to post on the port you want to use
-or you have not declared to the JVM you are using a proxy,
-or there is a proxy authentication issue.

Proxy is kind of black magic, particularly in Java and you can read
about anything wrt it. I am waiting for the day where SUN is going to
clarify how this is supposed to *really* work.

To declare the proxy, do this (that always works):

System.setProperty("http.proxyHost","my.proxy.com");
System.setProperty("http.proxyPort","proxyport");

Replace http with https for an HTTPS connection.


Where the thing becomes fuzzy, it is with authentication:

The official methods is this
System.setProperty( "http.proxyUser", proxyUser);
System.setProperty( "http.proxyPassword", proxyPassword);
Replace http with https for an https connection.

Sounds simple but this never worked for me, at least with https.

What works is the old fashioned method on manipulating the HTTP headers:

String encodedAuth = (new sun.misc.BASE64Encoder()).encode(
(proxyUser + ":" + ((proxyPassword==null) ? "" :
proxyPassword)).getBytes());

conx.setRequestProperty("Proxy-Authorization", "Basic "+encodedAuth);

Sometimes (not for Squid I believe), you will need to do this:

conx.setRequestProperty( "Https-Proxy-Authorization", "Basic " +
encodedAuth );

Cheers

Bruno

 

[Darryl L. Pierce]

But I'm afraid that I'm still getting the same error from the MIDlet:
"proxy cannot initialize HTTP tunnel connection".

That's a problem outside of the scope of the device. But, you're having this
problem on the emulator and not one a device, right??

try {
s = c.openDataInputStream();

Try, before this line, doing:

int rc = c.getResponseCode();
String rm = c.getResponseMessage();

and see what happens.

I've event tried fooling around with J2ME Toolkit's Preferences. I've
typed in different (wrong) proxy port numbers. The result is I had one
java/io/IOException and HTTPConnection wasn't established. Then I've
typed in the Preferences the address of the Web server itself and port
80 (port of the HTTP protocol) and still no budge.

Is your proxy trying to force you into an HTTPS connection? There's no
reason for a ClassNotFoundException unless the proxy's doing a redirect and
it's trying to force a protocol other than HTTP down on the handset.

I'd like to reiterate that I can establish HTTP connections in Visual
Studio .NET, Visual Basic, ANSI C++ using standard libraries in the
same (network) environment I'm trying to use MIDlet and Sun's J2ME
emulator. MIDlet can't establish HTTP connection throught a proxy,
while other technologies can.

It's not the MIDlet, it's the emulator. MIDlet's depend on their environment
to handle negotiating through firewalls and proxies.

--
/**
* @author Darryl L. Pierce <[email protected]>
* @see The Infobahn Offramp <http://mcpierce.multiply.com>
* @quote "Lobby, lobby, lobby, lobby, lobby, lobby..." - Adrian Monk
*/

 

[Darryl L. Pierce]

Bruno Grieder wrote:

To declare the proxy, do this (that always works):

System.setProperty("http.proxyHost","my.proxy.com");
System.setProperty("http.proxyPort","proxyport");

You can't do this in MIDP. They will either cause RuntimeErrors to be
thrown, or they'll just be ignored by the system.

Replace http with https for an HTTPS connection.

MIDP 1.0 does not support HTTPS. That would cause a ClassNotFoundException
or else a protocol exception to get through.

--
/**
* @author Darryl L. Pierce <[email protected]>
* @see The Infobahn Offramp <http://mcpierce.multiply.com>
* @quote "Lobby, lobby, lobby, lobby, lobby, lobby..." - Adrian Monk
*/

 

[Marko]

That's a problem outside of the scope of the device. But, you're having this
problem on the emulator and not one a device, right??


Try, before this line, doing:

int rc = c.getResponseCode();
String rm = c.getResponseMessage();

and see what happens.

OK. I'm getting the same error ("java.io.IOException: Error
initializing HTTP tunnel connection: HTTP/1.0 403 Forbidden Server:
squid/2.5.STABLE1") and the same two ClassNotFoundExceptions and two
IOExceptions.

More precisely, c.getResponseCode() function call is causing the
error.

I have also used ContentConnection insted of HTTPConnection, and the
(same) error is still there.

Is your proxy trying to force you into an HTTPS connection?

No. But - good point. I have three proxies at my disposal. Two of them
do indeed force HTTPS, but I'm working with the proxy that doesn't
force HTTPS.

When I'm using a proxy that forces HTTPS I'm getting this error:
"java.io.IOException: Error initializing HTTP tunnel connection:
HTTP/1.1 502 Proxy Error ( The specified Secure Sockets Layer (SSL)
port is not allowed. ISA Server is not configured to allow SSL
requests from this port. Most Web browsers use port 443 for SSL
requests. )"

There's no
reason for a ClassNotFoundException unless the proxy's doing a redirect and
it's trying to force a protocol other than HTTP down on the handset.

OK.
But how would you explain that when I download MIDlet on my mobile
phone and run it I get exactly the same error? It seems virtually
impossible that Telecom would force all HTTP requests to HTTPS. They
have more than one million customers, in other words one million
potentially unsatisfied customers.

It's not the MIDlet, it's the emulator. MIDlet's depend on their environment
to handle negotiating through firewalls and proxies.

Right.

You see I'm trying to make a MIDlet that can be a client in a multi
tier client/server application. Web services and HTTP are two key
technologies, key connection points, in this architecture. Nowadays
mobile phones are virtually everywhere, a lot of them have J2ME
Virtual Machine, and it would be very interesting to use them for apps
other than calculators, currency converters and 80s-like-games (ATARI,
Commodore 64, ZX Spectrum style games).

So enabling HTTP connectivity is the key.

 

[Darryl L. Pierce]

OK. I'm getting the same error ("java.io.IOException: Error
initializing HTTP tunnel connection: HTTP/1.0 403 Forbidden Server:
squid/2.5.STABLE1") and the same two ClassNotFoundExceptions and two
IOExceptions.

More precisely, c.getResponseCode() function call is causing the
error.

Okay, now it's more clear. The 403 from your server is from your server's
understanding the request but it is refusing to fulfill it. Check your
server's logs and see what exactly the request was that it was trying to
process. We can't do anything more to help until we know *what* your server
is refusing to fulfill.

I have also used ContentConnection insted of HTTPConnection, and the
(same) error is still there.

It's not an issue on the emulator side, it's an issue on the server side.

OK.
But how would you explain that when I download MIDlet on my mobile
phone and run it I get exactly the same error? It seems virtually
impossible that Telecom would force all HTTP requests to HTTPS. They
have more than one million customers, in other words one million
potentially unsatisfied customers.

My previous theory about being forced to HTTPS was soley about running on
an emulator on a LAN.

What is the actual URL you're trying to open?

Right.

Everything you described above (other than the MIDP emulator) relies
directly on your PC's TCP/IP stack. The MIDP emulator tries to emulator a
real device.

--
/**
* @author Darryl L. Pierce <[email protected]>
* @see The Infobahn Offramp <http://mcpierce.multiply.com>
* @quote "Lobby, lobby, lobby, lobby, lobby, lobby..." - Adrian Monk
*/

 

[Darryl L. Pierce]

Marko wrote:
<snip>

I'm interested now, though, in why your proxy was returning a 403. What was
it that caused the proxy to reject your MIDlet's communications request?

--
/**
* @author Darryl L. Pierce <[email protected]>
* @see The Infobahn Offramp <http://mcpierce.multiply.com>
* @quote "Lobby, lobby, lobby, lobby, lobby, lobby..." - Adrian Monk
*/

 

[Marko]

Okay, now it's more clear. The 403 from your server is from your server's
understanding the request but it is refusing to fulfill it. Check your
server's logs and see what exactly the request was that it was trying to
process. We can't do anything more to help until we know *what* your server
is refusing to fulfill.


It's not an issue on the emulator side, it's an issue on the server side.

OK. It's good to know that the programming actually works. I was using
standard solutions for enabling HTTP communication in J2ME as
described here (as an example to others who might read this article):

Basic Network Programming in J2ME MIDP:
http://www.samspublishing.com/articles/article.asp?p=131116&seqNum=7

Network Programming with J2ME Wireless Devices:
http://www.wirelessdevnet.com/channels/java/features/j2me_http.phtml

Bill Day's J2ME Archive:
http://www.billday.com/j2me/index.html

<snip>

What is the actual URL you're trying to open?

One very "common" URL:
http://www.google.com/index.html

Everything you described above (other than the MIDP emulator) relies
directly on your PC's TCP/IP stack. The MIDP emulator tries to emulator a
real device.

OK. As I've said, as long as J2ME code that I was using is OK -- I'm
satisfied as a programmer.