URLEncoder -- When Is It Really Needed? (I Seem to Misunderstand It.)

[Hal Vaughan]

I am getting different results by using URLEncoder than what I expect and
what my research says should happen. Naturally, since Java can't be wrong
(okay -- stop laughing!), I must not understand this correctly. I'd really
like some clarification.

I have a few programs, in Perl, that go on my web site for uploading and
downloading data between programs. The data is transferred in name/value
pairs using POST. In some cases there are longer values, such as small
files that are being transferred. (It's easier to transfer the files like
regular data than using any up/download methods since I needed to parse the
data from name/value pairs anyway.)

My Java program will run on several different computers, some Windows, some
Linux. To communicate with the web site (and the Perl scripts o it), I use
the routine at the end of this post. The text, messageText, is the
name/value pairs, something like this:

source=server2&destination=server3&filename=myfile&filedata=ThisIsMyfile

As I understand it, the text I send should be encoded using URLEncoder so
any disallowed characters are replaced with codes. For most of the values
this wasn't an issue, since the values were not going to use anything other
than alphanumeric characters. I tried sending data by using URLEncoder to
encode the value for filedata, since that could include any text character
and newlines. When I did this, the data the Perl script saved on its end
was still encoded.

I remember my first test didn't work unless I encoded the data, and now when
I'm sending data to the server, it works fine unencoded and if I encoded
it, it's never decoded on the other end. (I realize that may be a Perl
problem -- some setting in Perl's CGI.pm, but since this may be a Java
issue, I want to understand both ends of the process.)

When I'm sending data up to the server for POST, is it standard to and
should I be encoding the data? Are there times when I should do it and
times I should not? From what I've read, the data should always be
encoded, but that would not explain why it's working now without any
encoding.

Thanks for any help on this!

Hal


-----HTTP/HTTPS connection method------

//sURL = "http://myhost.com/perl/upload.pl"
// or, if using secure connections:
//sURL = "https://myhost.com/perl/upload.pl"

public String connect(String sURL, String messageText) {
String sLine, resultPage = "";
URL uPage;
URLConnection ucPage;
BufferedReader inRead;
PrintWriter outPrint;

try {
uPage = new URL(sURL);
ucPage = uPage.openConnection();
ucPage.setDoOutput(true);
outPrint = new PrintWriter(ucPage.getOutputStream());
outPrint.print(messageText);
outPrint.close();
inRead = new BufferedReader(new
InputStreamReader(ucPage.getInputStream()));
resultPage = "";
while ((sLine = inRead.readLine()) != null) {
resultPage = resultPage + sLine + "\n";
}
System.out.println(resultPage);
inRead.close();
} catch (Exception e) {
System.out.println("Error: " + e);
e.printStackTrace();
resultPage = "error: connection incomplete";
}
return resultPage;
}

 

[Sandeep]

When I'm sending data up to the server for POST, is it standard to and
should I be encoding the data? Are there times when I should do it and
times I should not? From what I've read, the data should always be
encoded, but that would not explain why it's working now without any
encoding.

- Try decoding the data explicity on the server side. It is _safer_ not
to rely on automatic converion on the server side (not sure if it
happens though ).

 

[Missaka Wijekoon]

Hal Vaughan wrote:
snip

outPrint = new PrintWriter(ucPage.getOutputStream());
outPrint.print(messageText);
outPrint.close();

Hal,

When sending data over a URL connection, the PrintWriter might not be
ideal. These classes may alter the data depending on the default
character set of the system (especially if binary data is involved in
file transfer). Try simply doing something like:

OutputStream os = ucPage.getOutputStream();
os.write(messageText.getBytes());
os.close();

You might also want to make sure that any file data when converted to a
String does not get munged due to character set conversion, etc.

Cheers,
Misk

 

[Hal Vaughan]

Hal Vaughan wrote:
snip
Hal,

When sending data over a URL connection, the PrintWriter might not be
ideal. These classes may alter the data depending on the default
character set of the system (especially if binary data is involved in
file transfer). Try simply doing something like:

OutputStream os = ucPage.getOutputStream();
os.write(messageText.getBytes());
os.close();

I took that into consideration. I'm writing a class that needs to be used
in a new program, but ALSO needs to serve as a drop in replacement for a
class in another program. The older program used e-mail to transfer data
instead of a web app, so all data being transferred is going to be MIME
encoded anyway.

With that in mind, won't PrintWriter handle the standard control characters
(like /n and possibly /t) that might be found in messages that are mostly
MIME encoded?

And thanks for pointing that out. I was just recently writing a port
forwarder, and one thing I found VERY frustrating is that 95% of all the
examples I could find in my own books and online never handled binary data
-- only using PrinterWriter and text reading classes.

Thanks!

Hal