Repeated read From Socket is Truncated

[Roedy Green]

It doesn't matter how often people
claim that there are messages.

underneath of course there are packets, but TCP/IP presents as two
continuous streams of characters one in each direction with no hint as
to where the packet boundaries were.

If you want messages, you need to invent your own way of doing them in
a continuous stream. The traditional way is to start each header with
two binary shorts:

1. message type
2. message length

You then can use the message type as in a case switch, an enum
values() or a HashMap lookup to get the code to deal with the
message. You can hop over to the next message by reading length
bytes.

if you have less than 255 types of message, you can shrink that to a
ubyte and if your messages are bigger than 32K, you can expand the
length to an ushort or an int. Leave yourself some breathing room.

..You can also do it by sending serialised objects which have their own
internal length tracking.

 

[Thomas Weidenfeller]

underneath of course there are packets,

There are packets, fragments, segments. And non of this makes up a
message. You can try as hard as you want, but you can't squeeze messages
out of the TCP protocol as such. You need a higher layer protocol.

If you want messages, you need to invent your own way of doing them in
a continuous stream.

This is what we try to tell the OP for the Nth time. Invent your own way
or use an existing higher layer protocol which already does what he want.

But the OP keeps claiming that his TCP is special, and that his Unix has
a very special recv() system call, which does things the way he wants
it, and not like they are. And he keeps telling us that Java is broken,
because it does not work with his imaginary TCP.

/Thomas

 

[Rogan Dawes]

Trouble@Mill wrote On 02/02/06 17:07,:

In response to Eric and Lothar:
[...]
00000720: 6669 6C65 206E 616D 653D 222E 6D61 696C 'file name=".mail'
00000730: 6361 7022 2074 7970 653D 2266 696C 6522 'cap" type="file"'
00000740: 202F 3E3C 2F63 7572 7265 6E74 3E3C 2F64 ' /></current></d'
00000750: 6972 6563 746F 7279 3E0A 'irectory>. '
Feb 02 13:34:38.642 Timed out waiting for reply
Feb 02 13:34:38.642 Got back: []
Feb 02 13:34:38.642 recv 0
Data:
00000000: ' '
TCP Socket closed 1888


From this, you can see that the returned data was not received in a
single buffer, and that the recv() DID understand that the data was
complete, and posted a 0 length reply to indicate this.

No; recv() did not understand that the "data was
complete." Rather, select() understood that there was
a ten-second interval of silence; after select() timed
out, recv() wasn't even called again.

Sorry Eric, you missed some pieces in your reply:

Feb 02 13:34:28.617 bytesReceived 0
Feb 02 13:34:28.617 Got back: [<directory><current
[..]
Feb 02 13:34:28.617 recv 1882
Data:
00000000: 3C64 6972 6563 746F 7279 3E3C 6375 7272 '<directory><curr'
[..]
00000750: 6972 6563 746F 7279 3E0A 'irectory>. '
Feb 02 13:34:38.642 Timed out waiting for reply

From that you can see that the data was received and printed BEFORE
the select() was re-entered, not after the select() timed out.

Cheers,
Eddie

It seems to me that recv returned immediately when there were 0 bytes
available for reading, since you do a non-blocking read. You are just
lucky that this occurs at the right time in your C code, as opposed to
your Java code. Your assumption that that a 0 byte read indicates the
end of the "message" is invalid, as mentioned numerous times already, it
simply indicates that there was nothing ready at that particular time.

I'd suggest doing something like:

// write your message to the server
OutputStream os = socket.getOutputStream();
os.write(message.getBytes());
os.flush();

InputStream is = socket.getInputStream();
// read the response from the server
byte[] buff = new byte[1024];
int got;
boolean complete = false;
ByteArrayOutputStream baos = new ByteArrayOutputStream();
while ((got=is.read(buff))>=0 && ! complete) {
baos.write(buff,0,got);
// this is VERY inefficient, a better way would be to maintain
// some kind of state machine, or possibly use SAX to parse
// the XML as it comes in, so you know when you end the response.
if (new String(baos.getBytes()).endsWith("</directory>"))
complete = true;
}
// process the message

Regards,

Rogan

 

[Roedy Green]

But the OP keeps claiming that his TCP is special, and that his Unix has
a very special recv() system call, which does things the way he wants
it, and not like they are. And he keeps telling us that Java is broken,
because it does not work with his imaginary TCP.

He is being seduced by the timing. The stream is coming at him in
bursts. IF the packets were flowing slowly with say 0.01 second
between them, it would LOOK as if you had access to the individual
packet boundaries in the stream. But sooner or late the packets will
come in a burst and you won't see those boundaries. Just think about
what happens on retransmission. The stream gets held up by the oldest
unsuccessful packet. When it gets through BRRMM. It can charge ahead
become the windowing allows it be several packets ahead. It would
look like to this OP, that suddenly the underlying packets got huge.

This is not something you can rely on to mean anything.

 

[Trouble@Mill]

OK, OK, You've all beaten me into submission. I'll crawl gracefully
back into my corner and promise to do more research next time.

But, I'd just like to add a couple of random muses on some of the

Well, it was select() that was the gatekeeper, so it thought there was
"something" to read.

Hey, I've been "lucky" on every single recv() that this client
application has done for the past 2 years. Maybe it's time to go buy
Lotto tickets. Or have I used up *all* my luck here.

Except it wasn't, otherwise the next select() would have failed, which
it didn't. Plus, there was more code that shown in the snippet which
continues to use the socket.

See above.

Awww, is that the only seduction being offered. <Wide Grin>

But seriously. Thank you all for the various lessons learned. Maybe
I should patent this:because, as pointed out, it's been working fine for over 2 years.

Cheers,
Eddie