SocketServer and a Java applet listener

[google]

Dear newsgroup,

I give up, I must be overseeing something terribly trivial, but I can't
get a simple (Java) applet to react to incoming (python) SocketServer
messages.

Without boring you with the details of my code (on request available,
though), here is what I do :

I have a TCPServer and BaseRequestHandler .
Connecting via telnet : everything goes OK.

Connecting from Applet :
problem 1 (worked around it) : java has some 'propietary' UTF-8 format,
python's unicode doesn't seem to handle it correctly and I have to
strip the first two bytes/chars , then all goes OK .

problem 2:
I have tried IMHO everything.
In the BaseRequestHandler.handle() method, I want to update a list of
clients in the server, i.e.:

self.server.players[username] = self

self := instance of the BaseRequestHandler, I only do this after
succesfull connect , i.e. first time socket. I assume (wrongfully?)
that I can now use the self.request socket for future transmissions to
the client.

In the applet, I start a thread that listens to the socket by eternally
looping over:
String line = self.din.readUTF()
if (line == null)
break;
handle(line);

self := instance of Thread
self.din := DataInputStream(socket.getInputStream());

(It's a bit simplistic, but I am quite sure (well...) I got those
things right, the issue seems to me to lie in some weird
java-python-socket anomaly having to do with close()/flush() etc. ...)

However, the handle(line) method only seems to get called when I
destroy (close ?) the socket on the server side. I tried making it a
wfile (socket.makefile) and calling the flush() method.
Moreover, I searched and googled and couldn't find reference to a
working implementation of a python SocketServer sending to a java
Applet (socket listener).


Would be much appreciated if anyone knows such a reference ?

Any pointers to correct handling of the 'java propietary UTF-8 format'
in python (xml.sax) would also be appreciated. Skipping the first two
bytes really is a smelly workaround, I know, *deep sigh*...

 

[Steve Horsley]

Dear newsgroup,

I give up, I must be overseeing something terribly trivial, but I can't
get a simple (Java) applet to react to incoming (python) SocketServer
messages.

Without boring you with the details of my code (on request available,
though), here is what I do :

I have a TCPServer and BaseRequestHandler .
Connecting via telnet : everything goes OK.

Connecting from Applet :
problem 1 (worked around it) : java has some 'propietary' UTF-8 format,
python's unicode doesn't seem to handle it correctly and I have to
strip the first two bytes/chars , then all goes OK .

Those 2 bytes are important! They are a string length indicator.
Here are the docs that tell you that it puts a 2-byte length on
the front:
http://java.sun.com/j2se/1.5.0/docs/api/java/io/DataOutputStream.html#writeUTF(java.lang.String)
If you are ignoring these bytes, then how can you be sure you
have received the whole string? Please don't tell me you "just
hope" that the whole string will always arrive in a single read()
call. There is no guarantee of that. TCP does NOT have message
boundaries, and TCP packets can be both fragmented and coalesced.
E.g. if you do:
out.write('Steve was here")
out.flush()
out.write("Bilbo Baggins wasn't")
out.flush()

it is entirely legal for two successive read() calls to retrieve
"steve was " and "hereBilbo Baggins wasn't". Although in
practice, fragmentation won't normally happen until strings reach
around 1500 bytes.

writeUTF tries to fix the problem by telling the receive how much
string to expect.

problem 2:
I have tried IMHO everything.
In the BaseRequestHandler.handle() method, I want to update a list of
clients in the server, i.e.:

self.server.players[username] = self

self := instance of the BaseRequestHandler, I only do this after
succesfull connect , i.e. first time socket. I assume (wrongfully?)
that I can now use the self.request socket for future transmissions to
the client.

In the applet, I start a thread that listens to the socket by eternally
looping over:
String line = self.din.readUTF()
if (line == null)
break;
handle(line);

Probably the same problem. If you didn't send a 2 byte length
indicator first, then java's readUTF() will have tried to
interpret the first 2 bytes that you did actually send as the
string length, and may well simply be waiting patiently for the
rest to arrive.

HTH
Steve

 

[google]

Steve Horsley schreef:

Those 2 bytes are important! They are a string length indicator.
Here are the docs that tell you that it puts a 2-byte length on
the front:
http://java.sun.com/j2se/1.5.0/docs/api/java/io/DataOutputStream.html#writeUTF(java.lang.String)
If you are ignoring these bytes, then how can you be sure you
have received the whole string? Please don't tell me you "just
hope" that the whole string will always arrive in a single read()
call. There is no guarantee of that. TCP does NOT have message
boundaries, and TCP packets can be both fragmented and coalesced.

Ah, I see... I worked around this (see below), the workaround is
consistent with what you assume.
My question "How to send/receive between python SocketServer and java
Applet (SocketListener)" then seems to be boiling down to: "How to
interface between python unicode and java read/writeUTF?"

Probably the same problem. If you didn't send a 2 byte length
indicator first, then java's readUTF() will have tried to
interpret the first 2 bytes that you did actually send as the
string length, and may well simply be waiting patiently for the
rest to arrive.

I just couldn't get read/writeUTF and python unicode to interface, so I
refactored the applet's socketlistener to convert the
socket.getInputStream to a BufferedInputReader on which I call the
readline() method.
I still skip the first two bytes in the python receiver which seems
harmless since python doesn't use the byte length.
On both sides I have other way to know when the end-of-message has been
reached. A timeout mechanism provides some safety for partial
transmission handling.


Thanks !



Thijs

 

[Steve Horsley]

Steve Horsley schreef:



I just couldn't get read/writeUTF and python unicode to interface, so I
refactored the applet's socketlistener to convert the
socket.getInputStream to a BufferedInputReader on which I call the
readline() method.
I still skip the first two bytes in the python receiver which seems
harmless since python doesn't use the byte length.
On both sides I have other way to know when the end-of-message has been
reached. A timeout mechanism provides some safety for partial
transmission handling.

I would encourage you to re-think this.

There are two normal ways to delineate messages in the
byte-stream: An explicit length indication up front (which java
read/writeUTF chooses), or a unique end-of-message indication at
the end such as your readline() for strings that end in linefeed.

HTTP is an interesting mixture of these - the header contains a
content-length line telling you how long the payload is, but the
header itslef is variable length, terminated by a blank line. In
chunked encoding, there is a variable number of chunks, and the
last chunk is marked as such in the header.

Anyway, either approach is good. But using timing to separate
messages is Bad. There is always the chance that you will get
bitten by strange timings happening later on.

If you choose to go for the java read/writeUTF approach, the
2-byte length indicator goes Most Significant Byte first, so a
100 char string would be preceded by 00 64 ... Also, the
indicator gives the number of bytes after encoding, not the
number of characters before encoding.

Steve

 

[google]

Steve Horsley schreef:

There are two normal ways to delineate messages in the
byte-stream: An explicit length indication up front (which java
read/writeUTF chooses), or a unique end-of-message indication at
the end such as your readline() for strings that end in linefeed.

If you choose to go for the java read/writeUTF approach, the
2-byte length indicator goes Most Significant Byte first, so a
100 char string would be preceded by 00 64 ... Also, the
indicator gives the number of bytes after encoding, not the
number of characters before encoding.

You are right, Steven. I invested some time and isolated the problem of
transferring UTF-8 strings between a python SocketServer and a Java
applet. In a very, very draft version, hereby surrounded by every
disclaimer imagineable (as in "Don't try this at home!") I have put the
result on a webpage:
http://www.phaedro.com/javapythonutf8/

Not only I tried to give a more or less 'generic' solution to the UTF-8
interface (reversing the leading bytes using python's struct module,
and based on SocketServer.StreamingRequestHandler), maybe the
draft-craft helps others looking for (rather scarce) examples of
SocketServer implementations - this one is very trivial so maybe others
can learn more efficiently than I had to do.