Partial messages returned over socket

Discussion in 'Java' started by final74, Jul 12, 2005.

  1. final74

    final74 Guest

    I have written a Java applicaion that connects via a socket to a
    third-party server. The server comms are byte-oriented and I've been
    having problems with partial messages being returned by from it. I have
    therefore used DataInputStream and DataOutputStream with no buffering
    or filter wrappers so as to properly monitor the byte streams to and
    from the server.

    As soon as I connect, the server sends a response to which I must
    reply. Everything proceeds correctly until a point where I expect a
    'prompt' from the sever. However, my client actually only receives the
    first 8 bytes of the prompt. If I subsequently send an arbitrary
    message to the server the client then
    receives the remainder of the message. If I then respond to this
    prompt, I get no response whatsoever from the server.

    The forms of the 'send' and the 'receives' I use are as follows:
    [RECEIVE]
    byte[] m;
    m = new byte[300];
    count = in.read(m);

    ...where in is a DataInputStream. I check the return value,
    count.
    [SEND]
    out.write(m,0,m.length);
    out.flush();
    ..where out is a DataOutputStream, and m is byte[];

    Judging by the symptoms, it looks like the stream/socket is becoming
    'clogged', somehow. I thought I would avoid this by invoking flush()
    during a 'send', thus clearing the socket prior to a response from the
    server being transmitted over it. I also always understood that the
    underlying TCP/IP would guarantee that the entire message would be sent
    but not necessarily in one read. At first sight this appears to be what
    is happening, hoever I thought TCP/IP's 'chunking' was dependent upon
    an internal limit. Eight bytes seems to be too short for this, however.

    I have considered re-designing the app. to use threads and
    synchronization techniques. However, this seemed like overkill
    given my assumptions regarding flush(), above. I have a feeling I'm
    missing some other point, so any insights would
    be gratefully received. Thanks.
    final74, Jul 12, 2005
    #1
    1. Advertising

  2. final74 wrote:
    > Judging by the symptoms, it looks like the stream/socket is becoming
    > 'clogged', somehow.


    Start with the usual thing in networking: Gather facts by using a
    network sniffer. You need to find out if the problem is on the sending
    side, the network in between (unlikely from your description), or the
    receiving side.

    > I thought I would avoid this by invoking flush()
    > during a 'send', thus clearing the socket prior to a response from the
    > server being transmitted over it. I also always understood that the
    > underlying TCP/IP would guarantee that the entire message would be sent
    > but not necessarily in one read.


    (a) TCP has no notations of messages at all. It is a stream protocol. It
    doesn't send messages, it doesn't care about messages. It sends a data
    stream.

    (b) If you talk about sending, that would be a write, not a read. A
    single write might result in the data being send immediately, not send
    (buffered) or partly send - almost entirely at the discretion of the
    TCP/IP stack.

    > At first sight this appears to be what
    > is happening, hoever I thought TCP/IP's 'chunking' was dependent upon
    > an internal limit. Eight bytes seems to be too short for this, however.


    There is no such thing like like chunks in TCP. Maybe you mean segments.

    > I have considered re-designing the app. to use threads and
    > synchronization techniques.


    This is a waste of time if you don't know the source of your problems.
    There is no point in churning out random code if you don't know what is
    wrong.

    > However, this seemed like overkill
    > given my assumptions regarding flush(), above. I have a feeling I'm
    > missing some other point, so any insights would
    > be gratefully received.


    Yes, you are don't base your decisions on facts. Also, from your wording
    you seem to miss some understanding of TCP. A good textbook could help here.

    /Thomas

    --
    The comp.lang.java.gui FAQ:
    ftp://ftp.cs.uu.nl/pub/NEWS.ANSWERS/computer-lang/java/gui/faq
    http://www.uni-giessen.de/faq/archiv/computer-lang.java.gui.faq/
    Thomas Weidenfeller, Jul 12, 2005
    #2
    1. Advertising

  3. final74 wrote:

    > I have written a Java applicaion that connects via a socket to a
    > third-party server. The server comms are byte-oriented and I've been
    > having problems with partial messages being returned by from it. I have
    > therefore used DataInputStream and DataOutputStream with no buffering
    > or filter wrappers so as to properly monitor the byte streams to and
    > from the server.
    >
    > As soon as I connect, the server sends a response to which I must
    > reply. Everything proceeds correctly until a point where I expect a
    > 'prompt' from the sever. However, my client actually only receives the
    > first 8 bytes of the prompt. If I subsequently send an arbitrary
    > message to the server the client then
    > receives the remainder of the message. If I then respond to this
    > prompt, I get no response whatsoever from the server.
    >
    > The forms of the 'send' and the 'receives' I use are as follows:
    > [RECEIVE]
    > byte[] m;
    > m = new byte[300];
    > count = in.read(m);
    >
    > ...where in is a DataInputStream. I check the return value,
    > count.
    > [SEND]
    > out.write(m,0,m.length);
    > out.flush();
    > ..where out is a DataOutputStream, and m is byte[];
    >
    > Judging by the symptoms, it looks like the stream/socket is becoming
    > 'clogged', somehow. I thought I would avoid this by invoking flush()
    > during a 'send', thus clearing the socket prior to a response from the
    > server being transmitted over it. I also always understood that the
    > underlying TCP/IP would guarantee that the entire message would be sent
    > but not necessarily in one read. At first sight this appears to be what
    > is happening, hoever I thought TCP/IP's 'chunking' was dependent upon
    > an internal limit. Eight bytes seems to be too short for this, however.
    >
    > I have considered re-designing the app. to use threads and
    > synchronization techniques. However, this seemed like overkill
    > given my assumptions regarding flush(), above. I have a feeling I'm
    > missing some other point, so any insights would
    > be gratefully received. Thanks.
    >


    You will have problems reading an unknown number of bytes in this
    manner. You need to know when you've read the complete prompt.
    DataInputStream.read(byte[]) can read any number of bytes from 0 to the
    length of the byte[]. Is the data you are reading a string of ascii
    characters with a terminator (carriage return or line feed)? If so, use
    a Reader and read the String and then respond. If the data is fixed
    length, then use a buffer of that length and completely read the response.

    byte[] buf = new byte[response length];
    int bytesRead = 0;
    do {
    bytesRead = in.read(buf,bytesRead,buf.length);
    } while (bytesRead < buf.length) ;

    Unless your communication is really simple the client should be threaded.

    --

    Knute Johnson
    email s/nospam/knute/
    Knute Johnson, Jul 12, 2005
    #3
  4. final74 wrote:
    > I have written a Java applicaion that connects via a socket to a
    > third-party server. The server comms are byte-oriented and I've been
    > having problems with partial messages being returned by from it. I have
    > therefore used DataInputStream and DataOutputStream with no buffering
    > or filter wrappers so as to properly monitor the byte streams to and
    > from the server.
    >
    > As soon as I connect, the server sends a response to which I must
    > reply. Everything proceeds correctly until a point where I expect a
    > 'prompt' from the sever. However, my client actually only receives the
    > first 8 bytes of the prompt. If I subsequently send an arbitrary
    > message to the server the client then
    > receives the remainder of the message. If I then respond to this
    > prompt, I get no response whatsoever from the server.
    >
    > The forms of the 'send' and the 'receives' I use are as follows:
    > [RECEIVE]
    > byte[] m;
    > m = new byte[300];
    > count = in.read(m);
    >
    > ...where in is a DataInputStream. I check the return value,
    > count.
    > [SEND]
    > out.write(m,0,m.length);
    > out.flush();
    > ..where out is a DataOutputStream, and m is byte[];
    >
    > Judging by the symptoms, it looks like the stream/socket is becoming
    > 'clogged', somehow. I thought I would avoid this by invoking flush()
    > during a 'send', thus clearing the socket prior to a response from the
    > server being transmitted over it. I also always understood that the
    > underlying TCP/IP would guarantee that the entire message would be sent
    > but not necessarily in one read.


    It guarantees that it will TRY to send all bytes given it when
    you call flush. Barring network problems or window congestion, it
    will send them. There is no guarantee how many packets it will
    use to send the byte stream, or what the timing will be.


    > At first sight this appears to be what
    > is happening, hoever I thought TCP/IP's 'chunking' was dependent upon
    > an internal limit. Eight bytes seems to be too short for this, however.


    I repeat - There is no guarantee how many packets it will use to
    send the byte stream, or what the timing will be. (Actually, you
    are guaranteed that you will never recieve _part_ of an byte.)
    Even if the server sends a "message" all in one packet, there is
    no guarantee that the receiving IP stack will deliver it to your
    application in one read call.

    Don't assume what might and might not happen - write to cope with
    the guaranteed behaviour. You have a major error above in
    assuming that a single read() call will get everything you are
    expecting. This is not guaranteed, as you seem to have
    discovered. I am betting that the server sends the "prompt" in at
    least 2 packets - maybe a generic linefeed, then a specific text
    prompt, message of the day or server version - I dunno. You
    should ALWAYS keep calling read() until you are SURE you have all
    of the message you expect, to avoid disappointment.

    A protocol analyser will prove the point, but could just
    encourage you to assume that the server will always behave that
    way, which is not guaranteed of course. How the server packetizes
    its stream may depend very much on load and process timings.

    > I have considered re-designing the app. to use threads and
    > synchronization techniques. However, this seemed like overkill
    > given my assumptions regarding flush(), above. I have a feeling I'm
    > missing some other point, so any insights would
    > be gratefully received. Thanks.
    >


    Not necessary really. If you can tell when the end of the
    "message" has arrived, just keep reading until it has. If you
    can't, then you're screwed and all the threads in the world won't
    help figure it out (except maybe a timer thread, but that's
    unreliable because network delays can fool you).
    Steve Horsley, Jul 12, 2005
    #4
  5. final74

    final74 Guest

    Apologies for letting this thread slide: I was diverted onto some
    'real' work and had to drop the telnet experiment for a while.

    @Thomas I know a little about TCP/IP but I'm no expet. Hence this
    project as a starting point so, yeah, chunks probably isnt the offical
    term but means the same as segement in my description.

    I have no access to network sniffers or any of the usual tools for
    debuging such problems. They are locked down and are usable by root
    user's only in this installation. The suggestion of a threaded approach
    - far from being 'random' code - was an attempt to determine the
    problem from another angle. Indeed, I _dont_ know what is wrong which
    is exactly why I am trying a number of approaches and posting here. It
    is an experiment only; a leraning exercise for me which I do not
    consider a waste of time at all! Far from it.

    As for basing a diagnosis on facts. I have implemented apps in the past
    that communicate over sockets in a variety of languages. When TCP/IP
    has segmented the data the number of bytes delivered has always been
    numbered in hundreds of bytes. This has been observable over a number
    of years. So, I was concerend when I received 'segements' in much
    smaller quantities.

    @Steve/Knute

    I've implemented an algorithm that attempts to determine the end of and
    command sequences. Raw text (message of the day) I ignore. This works
    but not consistently.
    In some instances the call to read the stream appears to stop dead.
    There is no opportunity to read agin as the Java method that has been
    invoked just does not return though it's process appears to be still
    executing.

    I have no idea what is happening here. I am specualting as to whether
    I've got ahead of telnetd before it's marshalled its responses. So, I
    think some further TCP/IP research wopuld be beneficial at this point.

    Regards
    final74, Jul 26, 2005
    #5
  6. final74 wrote:

    > @Steve/Knute
    >
    > I've implemented an algorithm that attempts to determine the end of and
    > command sequences. Raw text (message of the day) I ignore. This works
    > but not consistently.
    > In some instances the call to read the stream appears to stop dead.
    > There is no opportunity to read agin as the Java method that has been
    > invoked just does not return though it's process appears to be still
    > executing.
    >
    > I have no idea what is happening here. I am specualting as to whether
    > I've got ahead of telnetd before it's marshalled its responses. So, I
    > think some further TCP/IP research wopuld be beneficial at this point.
    >
    > Regards
    >


    This is one of those times when you really NEED an analyser. The
    next best thing is to add a routine that logs the length and
    content of every read() call. You CAN be sure that the underlying
    TCP will not lose data. The most common problem is either
    ignoring the length returned from read(), assuming the entire
    buffer has been filled or that the entire sent message arrives in
    one lump.

    Not having a firm way (except time) to know for sure that the
    sender has finished is a bind. I have seen servers that upon
    receiving a connection, put CR FL in the first packet, then some
    telnet negotiation in the second (DO ECHO etc) and a login prompt
    in the third.

    I have also seen a server that took 60.2 seconds to send the
    login prompt after accepting a connection (turned out to be a DNS
    configuration error - go figure).

    The Cog
    Cantankerous Old Git, Jul 26, 2005
    #6
  7. final74

    final74 Guest

    Thanks 'Cog'

    I've been logging the content all along as you adivsed. It's awkward
    because it's intermittent. Once I've read the stream I pass the buffer
    to my logging routine which loops through buf.length bytes and writes
    them off to a file. If I have determined there is more data to come, or
    I want to issue a shell command over the telnet connection, I send data
    over the stream, flush() the stream then start reading again. It's at
    this point that the read just sits there.

    I'll post again if, when I hit upon the answer!

    Rgds
    final74, Jul 27, 2005
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Billy
    Replies:
    2
    Views:
    493
    Billy
    Feb 1, 2006
  2. Bryan
    Replies:
    1
    Views:
    429
  3. Thomas Heller
    Replies:
    13
    Views:
    843
    Michele Simionato
    Feb 8, 2007
  4. J. Clifford Dyer

    Re: Partial 1.0 - Partial classes for Python

    J. Clifford Dyer, Feb 8, 2007, in forum: Python
    Replies:
    0
    Views:
    511
    J. Clifford Dyer
    Feb 8, 2007
  5. Ashwin Rao
    Replies:
    2
    Views:
    1,038
    Martien Verbruggen
    Dec 3, 2009
Loading...

Share This Page