Receiver in Outputstream.read() stops after 2735 bytes

[Bart Friederichs]

Hi,

I have written a sender-receiver and the receiver stops receiving any
data after 2735 bytes. The sender seems to be fine, because when
connecting with a telnet session, it sends all the bytes.

I have tried to send the data in 100 byte pieces and flush() afterwards,
to no avail.

Am I missing someting?

TIA
Bart

 

[Knute Johnson]

Hi,

I have written a sender-receiver and the receiver stops receiving any
data after 2735 bytes. The sender seems to be fine, because when
connecting with a telnet session, it sends all the bytes.

I have tried to send the data in 100 byte pieces and flush() afterwards,
to no avail.

Am I missing someting?

Yes, showing us the code .

 

[Bart Friederichs]

Yes, showing us the code .

Obviously.

This is the sender:

socket.sock.getOutputStream().write(chunk);

socket is my own class, sock inside it is a java.net.Socket, chunk is a
byte[].

This is the receiver:

int bytesleft = length;
int bytesread = 0;
while (bytesleft > 0 && bytesread > -1) {
bytesread = socket.sock.getInputStream().read(theChunk, length -
bytesleft, 1);
bytesleft -= bytesread;
}

theChunk is a byte[] of size 'length'

Bart

 

[markspace]

Hi,

I have written a sender-receiver and the receiver stops receiving any
data after 2735 bytes. The sender seems to be fine, because when
connecting with a telnet session, it sends all the bytes.

To elaborate on Knute's comment, I've used OutputStream.read() in plenty
of places where it read more than 2735 bytes. Therefore, there must be
some mistake in your own code. Show us an SSCCE and we might be able to
help.

http://sscce.org/

 

[markspace]

This is the sender:

Just to add a bit to my previous post: This code isn't complete. We
need a complete example that reproduces the problem. Otherwise, not
much we can do.

 

[Eric Sosman]

Yes, showing us the code .

Obviously.

This is the sender:

socket.sock.getOutputStream().write(chunk);

socket is my own class, sock inside it is a java.net.Socket, chunk is a
byte[].

This is the receiver:

int bytesleft = length;
int bytesread = 0;
while (bytesleft> 0&& bytesread> -1) {
bytesread = socket.sock.getInputStream().read(theChunk, length -
bytesleft, 1);
bytesleft -= bytesread;
}

theChunk is a byte[] of size 'length'

... whose first `bytesread' elements will hold the
values from the *last* call to read(), if I'm not mistaken.
That is, if you try to read 1000 bytes and get them in two
chunks of 600 and 400 bytes each,

- The first read() deposits input bytes 0-599 in
theChunk[0] through theChunk[599],

- The next read() deposits input bytes 600-999 in
theChunk[0] through theChunk[399], wiping out
bytes 0-399,

- At the end, theChunk[] holds input bytes 600-999,
followed by input bytes 400-599, followed by zeroes
(or old garbage),

- And, as a special bonus, you have no way of knowing
how many bytes were received or where they were stored.

 

[Bart Friederichs]

theChunk is a byte[] of size 'length'

... whose first `bytesread' elements will hold the
values from the *last* call to read(), if I'm not mistaken.
That is, if you try to read 1000 bytes and get them in two
chunks of 600 and 400 bytes each,

- The first read() deposits input bytes 0-599 in
theChunk[0] through theChunk[599],

- The next read() deposits input bytes 600-999 in
theChunk[0] through theChunk[399], wiping out
bytes 0-399,

Not according to the documentation:

"...The first byte read is stored into element b[off], the next one into
b[off+1], and so on. The number of bytes read is, at most, equal to len.
Let k be the number of bytes actually read; these bytes will be stored
in elements b[off] through b[off+k-1], leaving elements b[off+k] through
b[off+len-1] unaffected.

In every case, elements b[0] through b[off] and elements b[off+len]
through b[b.length-1] are unaffected. ..."

- At the end, theChunk[] holds input bytes 600-999,
followed by input bytes 400-599, followed by zeroes
(or old garbage),

- And, as a special bonus, you have no way of knowing
how many bytes were received or where they were stored.

Of course, read() returns that.

Bart

 

[markspace]

Of course, read() returns that.

I think Eric means that read() also returns -1 if it reaches
end-of-stream, which you don't test for. Therefore, bytesleft will be
off by -1 if you reach end of stream. But you have no way of knowing
that it did, so....

 

[Bart Friederichs]

I think Eric means that read() also returns -1 if it reaches
end-of-stream, which you don't test for. Therefore, bytesleft will be
off by -1 if you reach end of stream. But you have no way of knowing
that it did, so....

True, but my code echoes me the return value, which is 35 in the last
call. After that, the read() blocks.

According to the process monitor, the processes are waiting in the
futex_wait channel (whatever that is; seems to have to do something with
threading).

i posted the code as I have it now here:
http://www.friesoft.nl/software/MammothCopy-0.1.tar.gz

which is a NetBeans project.

I am sorry it is not short, but it is what I could do fast, and if
someone is willing to take a look at it, I will be grateful. The code
that seems to have problems is in MammothReceiver::ReceiveChunks (line 191).

Bart

 

[Martin Gregorie]

Yes, showing us the code .

Obviously.

This is the sender:

socket.sock.getOutputStream().write(chunk);

socket is my own class, sock inside it is a java.net.Socket, chunk is a
byte[].

This is the receiver:

int bytesleft = length;
int bytesread = 0;
while (bytesleft > 0 && bytesread > -1) {
bytesread = socket.sock.getInputStream().read(theChunk, length -
bytesleft, 1);
bytesleft -= bytesread;
}

theChunk is a byte[] of size 'length'

Since you don't, apparently, send the receiver any indication of how many
bytes you intend to send it, how does 'length' get set to a sensible
value?

 

[Bart Friederichs]

I have written a sender-receiver and the receiver stops receiving any
data after 2735 bytes. The sender seems to be fine, because when
connecting with a telnet session, it sends all the bytes.

I have been experimenting a little bit more and found that when sending
smaller pieces of data (I used 41 bytes now), AND finishing the exact
amount of bytes, the entire chunk gets sent correctly.

Bart

 

[Bart Friederichs]

You are correct, however, that the OP's code is flawed in its use of
"bytesread". He should only be subtracting the number from "bytesleft"
if "bytesread" is positive. An easy way to do that is either to move
the call to read() and assignment to bytesread into the while loop's
condition expression, or simply to move the statement decrementing
bytesleft to precede the call to read() in the loop (a harmless 0
decrement happens the first time through in that approach).

Yes, that's a good one, and I will implement the correct checking of the
-1 return value.

Other than the off-by-one bug, the biggest issue with the loop is that
the OP is reading one byte at a time! Duh.

Which is the result of my own debugging efforts .

Bart

 

[Bart Friederichs]

Since you don't, apparently, send the receiver any indication of how many
bytes you intend to send it, how does 'length' get set to a sensible
value?

I do send the receiver the size of the chunk, before sending the chunk
itself. The code is part of a protocol implementation.

The code is part of a large file copying program. See also
http://blog.friesoft.nl/2009/10/24/sending-really-big-files/

Bart

 

[Bart Friederichs]

I have been experimenting a little bit more and found that when sending
smaller pieces of data (I used 41 bytes now), AND finishing the exact
amount of bytes, the entire chunk gets sent correctly.

Hm..

No idea why I got into this mess. When sending and receiving exactly
'length' bytes, it works fine. When 'length' is bug (around 1MB) I get a
'connection reset' exception, so I guess I should chop the chunk up in
smaller pieces to send.

Anyway, I am on my way again.

Bart

 

[Martin Gregorie]

I do send the receiver the size of the chunk, before sending the chunk
itself. The code is part of a protocol implementation.

The code is part of a large file copying program. See also
http://blog.friesoft.nl/2009/10/24/sending-really-big-files/

Thats why posting an SSCE is a good idea. You don't forget rather
important details like that and may well see the flaw when you're making
the SSCE. Another thing you haven't shown us or said you've checked: are
you sure the sender always reports the number of bytes its going to send
correctly?

 

[markspace]

No idea why I got into this mess. When sending and receiving exactly
'length' bytes, it works fine. When 'length' is bug (around 1MB) I get a
'connection reset' exception, so I guess I should chop the chunk up in
smaller pieces to send.

Anyway, I am on my way again.

This is all so wrong, I don't know where to start. Have you considered
posting that SSCCE, so you can get some advice how to read and write
streams correctly, without doing silly things like breaking your data
into chunks less than 1MB for no obvious reason?

 

[EJP]

while (bytesleft > 0 && (bytesread =
socket.sock.getInputStream().read(theChunk, length-bytesleft,
bytesleft)) > -1)
bytesleft -= bytesread;
}

 

[Eric Sosman]

theChunk is a byte[] of size 'length'

... whose first `bytesread' elements will hold the
values from the *last* call to read(), if I'm not mistaken.
That is, if you try to read 1000 bytes and get them in two
chunks of 600 and 400 bytes each,

- The first read() deposits input bytes 0-599 in
theChunk[0] through theChunk[599],

- The next read() deposits input bytes 600-999 in
theChunk[0] through theChunk[399], wiping out
bytes 0-399,

Not according to the documentation:

"...The first byte read is stored into element b[off], the next one into
b[off+1], and so on. The number of bytes read is, at most, equal to len.
Let k be the number of bytes actually read; these bytes will be stored
in elements b[off] through b[off+k-1], leaving elements b[off+k] through
b[off+len-1] unaffected.

Good thing I wrote "if I'm not mistaken," isn't it? ;-)

Of course, read() returns that.

Not my day, was it? You don't have a direct indication of
how many bytes were read, but you *can* compute the count:

int totalbytes = length - bytesleft
+ (bytesread < 0 ? 1 : 0);

Definitely not my day ...

 

[Tom Anderson]

I do send the receiver the size of the chunk, before sending the chunk
itself. The code is part of a protocol implementation.

The code is part of a large file copying program. See also
http://blog.friesoft.nl/2009/10/24/sending-really-big-files/

That's called 'scp' or 'bittorrent'. Running that between two
NATted/firewalled boxes is called 'NAT traversal' or 'hole punching'.
There are techniques and tools for doing these things - google will help
you find them.

Or you could just use GBridge or similar:

http://www.gbridge.com/

tom

 

[Bart Friederichs]

I took one of his previous replies to mean that he had considered a
SSCCE and decided he would rather than each of us individually distill
his code into a usable sample we can debug, rather than doing that work
himself.

In that case, you misunderstood me.

I have considered the SSCCE and also decided that I wasn't going to make
it yesterday (perhaps I will make it later this week, or month). I
posted the entire code, because of the possibility that someone *would*
take the time or effort to look into it. If nobody does that, fine. I
also don't expect anybody who specifically asks for an SSCCE to look
into it.

Bart