SocketServer and a Java applet listener

Discussion in 'Python' started by google@phaedro.com, Aug 24, 2005.

  1. Guest

    Dear newsgroup,

    I give up, I must be overseeing something terribly trivial, but I can't
    get a simple (Java) applet to react to incoming (python) SocketServer
    messages.

    Without boring you with the details of my code (on request available,
    though), here is what I do :

    I have a TCPServer and BaseRequestHandler .
    Connecting via telnet : everything goes OK.

    Connecting from Applet :
    problem 1 (worked around it) : java has some 'propietary' UTF-8 format,
    python's unicode doesn't seem to handle it correctly and I have to
    strip the first two bytes/chars , then all goes OK .

    problem 2:
    I have tried IMHO everything.
    In the BaseRequestHandler.handle() method, I want to update a list of
    clients in the server, i.e.:

    self.server.players[username] = self

    self := instance of the BaseRequestHandler, I only do this after
    succesfull connect , i.e. first time socket. I assume (wrongfully?)
    that I can now use the self.request socket for future transmissions to
    the client.

    In the applet, I start a thread that listens to the socket by eternally
    looping over:
    String line = self.din.readUTF()
    if (line == null)
    break;
    handle(line);

    self := instance of Thread
    self.din := DataInputStream(socket.getInputStream());

    (It's a bit simplistic, but I am quite sure (well...) I got those
    things right, the issue seems to me to lie in some weird
    java-python-socket anomaly having to do with close()/flush() etc. ...)

    However, the handle(line) method only seems to get called when I
    destroy (close ?) the socket on the server side. I tried making it a
    wfile (socket.makefile) and calling the flush() method.
    Moreover, I searched and googled and couldn't find reference to a
    working implementation of a python SocketServer sending to a java
    Applet (socket listener).


    Would be much appreciated if anyone knows such a reference ?

    Any pointers to correct handling of the 'java propietary UTF-8 format'
    in python (xml.sax) would also be appreciated. Skipping the first two
    bytes really is a smelly workaround, I know, *deep sigh*...

    --
    Thijs Cobben
    www.phaedro.com
     
    , Aug 24, 2005
    #1
    1. Advertising

  2. wrote:
    > Dear newsgroup,
    >
    > I give up, I must be overseeing something terribly trivial, but I can't
    > get a simple (Java) applet to react to incoming (python) SocketServer
    > messages.
    >
    > Without boring you with the details of my code (on request available,
    > though), here is what I do :
    >
    > I have a TCPServer and BaseRequestHandler .
    > Connecting via telnet : everything goes OK.
    >
    > Connecting from Applet :
    > problem 1 (worked around it) : java has some 'propietary' UTF-8 format,
    > python's unicode doesn't seem to handle it correctly and I have to
    > strip the first two bytes/chars , then all goes OK .
    >


    Those 2 bytes are important! They are a string length indicator.
    Here are the docs that tell you that it puts a 2-byte length on
    the front:
    http://java.sun.com/j2se/1.5.0/docs/api/java/io/DataOutputStream.html#writeUTF(java.lang.String)
    If you are ignoring these bytes, then how can you be sure you
    have received the whole string? Please don't tell me you "just
    hope" that the whole string will always arrive in a single read()
    call. There is no guarantee of that. TCP does NOT have message
    boundaries, and TCP packets can be both fragmented and coalesced.
    E.g. if you do:
    out.write('Steve was here")
    out.flush()
    out.write("Bilbo Baggins wasn't")
    out.flush()

    it is entirely legal for two successive read() calls to retrieve
    "steve was " and "hereBilbo Baggins wasn't". Although in
    practice, fragmentation won't normally happen until strings reach
    around 1500 bytes.

    writeUTF tries to fix the problem by telling the receive how much
    string to expect.


    > problem 2:
    > I have tried IMHO everything.
    > In the BaseRequestHandler.handle() method, I want to update a list of
    > clients in the server, i.e.:
    >
    > self.server.players[username] = self
    >
    > self := instance of the BaseRequestHandler, I only do this after
    > succesfull connect , i.e. first time socket. I assume (wrongfully?)
    > that I can now use the self.request socket for future transmissions to
    > the client.
    >
    > In the applet, I start a thread that listens to the socket by eternally
    > looping over:
    > String line = self.din.readUTF()
    > if (line == null)
    > break;
    > handle(line);
    >


    Probably the same problem. If you didn't send a 2 byte length
    indicator first, then java's readUTF() will have tried to
    interpret the first 2 bytes that you did actually send as the
    string length, and may well simply be waiting patiently for the
    rest to arrive.

    HTH
    Steve
     
    Steve Horsley, Aug 27, 2005
    #2
    1. Advertising

  3. Guest

    Steve Horsley schreef:

    > wrote:
    > > Dear newsgroup,
    > >
    > > I give up, I must be overseeing something terribly trivial, but I can't
    > > get a simple (Java) applet to react to incoming (python) SocketServer
    > > messages.
    > >
    > > Without boring you with the details of my code (on request available,
    > > though), here is what I do :
    > >
    > > I have a TCPServer and BaseRequestHandler .
    > > Connecting via telnet : everything goes OK.
    > >
    > > Connecting from Applet :
    > > problem 1 (worked around it) : java has some 'propietary' UTF-8 format,
    > > python's unicode doesn't seem to handle it correctly and I have to
    > > strip the first two bytes/chars , then all goes OK .
    > >

    >
    > Those 2 bytes are important! They are a string length indicator.
    > Here are the docs that tell you that it puts a 2-byte length on
    > the front:
    > http://java.sun.com/j2se/1.5.0/docs/api/java/io/DataOutputStream.html#writeUTF(java.lang.String)
    > If you are ignoring these bytes, then how can you be sure you
    > have received the whole string? Please don't tell me you "just
    > hope" that the whole string will always arrive in a single read()
    > call. There is no guarantee of that. TCP does NOT have message
    > boundaries, and TCP packets can be both fragmented and coalesced.


    Ah, I see... I worked around this (see below), the workaround is
    consistent with what you assume.
    My question "How to send/receive between python SocketServer and java
    Applet (SocketListener)" then seems to be boiling down to: "How to
    interface between python unicode and java read/writeUTF?"

    >
    > Probably the same problem. If you didn't send a 2 byte length
    > indicator first, then java's readUTF() will have tried to
    > interpret the first 2 bytes that you did actually send as the
    > string length, and may well simply be waiting patiently for the
    > rest to arrive.
    >

    I just couldn't get read/writeUTF and python unicode to interface, so I
    refactored the applet's socketlistener to convert the
    socket.getInputStream to a BufferedInputReader on which I call the
    readline() method.
    I still skip the first two bytes in the python receiver which seems
    harmless since python doesn't use the byte length.
    On both sides I have other way to know when the end-of-message has been
    reached. A timeout mechanism provides some safety for partial
    transmission handling.


    Thanks !



    Thijs
     
    , Aug 29, 2005
    #3
  4. wrote:
    > Steve Horsley schreef:
    >
    >
    >>Probably the same problem. If you didn't send a 2 byte length
    >>indicator first, then java's readUTF() will have tried to
    >>interpret the first 2 bytes that you did actually send as the
    >>string length, and may well simply be waiting patiently for the
    >>rest to arrive.
    >>

    >
    > I just couldn't get read/writeUTF and python unicode to interface, so I
    > refactored the applet's socketlistener to convert the
    > socket.getInputStream to a BufferedInputReader on which I call the
    > readline() method.
    > I still skip the first two bytes in the python receiver which seems
    > harmless since python doesn't use the byte length.
    > On both sides I have other way to know when the end-of-message has been
    > reached. A timeout mechanism provides some safety for partial
    > transmission handling.
    >


    I would encourage you to re-think this.

    There are two normal ways to delineate messages in the
    byte-stream: An explicit length indication up front (which java
    read/writeUTF chooses), or a unique end-of-message indication at
    the end such as your readline() for strings that end in linefeed.

    HTTP is an interesting mixture of these - the header contains a
    content-length line telling you how long the payload is, but the
    header itslef is variable length, terminated by a blank line. In
    chunked encoding, there is a variable number of chunks, and the
    last chunk is marked as such in the header.

    Anyway, either approach is good. But using timing to separate
    messages is Bad. There is always the chance that you will get
    bitten by strange timings happening later on.

    If you choose to go for the java read/writeUTF approach, the
    2-byte length indicator goes Most Significant Byte first, so a
    100 char string would be preceded by 00 64 ... Also, the
    indicator gives the number of bytes after encoding, not the
    number of characters before encoding.

    Steve
     
    Steve Horsley, Aug 29, 2005
    #4
  5. Guest

    Steve Horsley schreef:

    > wrote:
    > > Steve Horsley schreef:
    > >
    > >
    > >>Probably the same problem. If you didn't send a 2 byte length
    > >>indicator first, then java's readUTF() will have tried to
    > >>interpret the first 2 bytes that you did actually send as the
    > >>string length, and may well simply be waiting patiently for the
    > >>rest to arrive.
    > >>

    > >
    > > I just couldn't get read/writeUTF and python unicode to interface, so I
    > > refactored the applet's socketlistener to convert the
    > > socket.getInputStream to a BufferedInputReader on which I call the
    > > readline() method.

    <snap>

    > There are two normal ways to delineate messages in the
    > byte-stream: An explicit length indication up front (which java
    > read/writeUTF chooses), or a unique end-of-message indication at
    > the end such as your readline() for strings that end in linefeed.
    >


    <snap>

    >
    > If you choose to go for the java read/writeUTF approach, the
    > 2-byte length indicator goes Most Significant Byte first, so a
    > 100 char string would be preceded by 00 64 ... Also, the
    > indicator gives the number of bytes after encoding, not the
    > number of characters before encoding.
    >


    You are right, Steven. I invested some time and isolated the problem of
    transferring UTF-8 strings between a python SocketServer and a Java
    applet. In a very, very draft version, hereby surrounded by every
    disclaimer imagineable (as in "Don't try this at home!") I have put the
    result on a webpage:
    http://www.phaedro.com/javapythonutf8/

    Not only I tried to give a more or less 'generic' solution to the UTF-8
    interface (reversing the leading bytes using python's struct module,
    and based on SocketServer.StreamingRequestHandler), maybe the
    draft-craft helps others looking for (rather scarce) examples of
    SocketServer implementations - this one is very trivial so maybe others
    can learn more efficiently than I had to do.

    --
    Thijs Cobben
    Explicit typing sux.
     
    , Aug 29, 2005
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Krista
    Replies:
    3
    Views:
    2,358
    Andrew Thompson
    Sep 15, 2004
  2. Liam Slusser

    socketserver and postgres

    Liam Slusser, Oct 6, 2003, in forum: Python
    Replies:
    0
    Views:
    370
    Liam Slusser
    Oct 6, 2003
  3. Milo K. Piffenpauffer

    Examples and Tutorials of SocketServer?

    Milo K. Piffenpauffer, Dec 13, 2003, in forum: Python
    Replies:
    4
    Views:
    483
    Fredrik Lundh
    Dec 14, 2003
  4. alf

    SocketServer and timers

    alf, Jul 28, 2006, in forum: Python
    Replies:
    5
    Views:
    336
  5. theneb
    Replies:
    4
    Views:
    673
    theneb
    May 5, 2007
Loading...

Share This Page