has someone seen this error before?

[Jeff]

I would greatly appreciate some good advice on a problem that has eluded us.

Our server has what appears to be a network IO problem under load. We have
seen one instance of an error message that we think is a strong clue:
java.net.SocketException: Insufficient buffer space

However, we can't find any documentation on what this error means? Which
buffer space?

ARCHITECTURE
1. our probe sends large volumes of data to our server
2. the probe-server connection is tcp carrying proprietary data
3. at times, our probe definitely sends more data than the server can
process. We expected tcp congestion control to throttle the probe.

OPTIMIZATIONS COMPLETED
1. set the socket's receiveBufferSize to 32768
2. set the the buffered input stream's buffer size to 1024000

SYMPTOMS
1. data in buffer returned by inputStream.read() appears to be off by 2 or 3
bytes
2. no tcp congestion control (advertised window does not decrease)
3. according to the server's operating system stats, the receiver queue
builds up
4. data input rate is 7 to 8 Meg continuous.

ENVIRONMENT
- running BEA's JRockit JVM
- both SuSE 8.2 and WinXP.
- Lot of processing on the buffer returned by the read call is available for
garbage collection

SERVER'S SOCKET CODE
Socket probeSocket = sock.accept();
if ( probeSocket.getReceiveBufferSize() < 32768 )
probeSocket.setReceiveBufferSize(32768);

BufferedInputStream is = new BufferedInputStream(
probeSocket.getInputStream(), 1024000 );

Thanks

 

[VK]

Seems more like a queue limitation, rather than buffer.
Do you have a setting like "tcp_conn_request_max" and what (if any) does it
say?

 

[Louis]

(e-mail address removed) wrote...

Seems more like a queue limitation, rather than buffer.
Do you have a setting like "tcp_conn_request_max" and what (if any) does it
say?

I believe the OP stated it was a single connection that was
saturated.

 

[John C. Bollinger]

I would greatly appreciate some good advice on a problem that has eluded us.

Our server has what appears to be a network IO problem under load. We have
seen one instance of an error message that we think is a strong clue:
java.net.SocketException: Insufficient buffer space

However, we can't find any documentation on what this error means? Which
buffer space?

I don't _know_, but a good guess would be that the exception wraps an
error signaled by the OS' native socket implementation. The buffer
would be one of the socket's native I/O buffers. Such an error probably
means that you have too many active sockets, whether because there are
many sockets simultaneously in use or because discarded sockets are not
cleaned up. How many is too many would depend in part on system
resources, including file descriptors and kernel memory among other things.

It might not be the core problem, but you should be sure _always_ to
close() sockets when you are done with them. Typically you would ensure
this by putting the close() invocation in a finally {} block associated
with a suitably scoped try {} block.

ARCHITECTURE
1. our probe sends large volumes of data to our server
2. the probe-server connection is tcp carrying proprietary data
3. at times, our probe definitely sends more data than the server can
process. We expected tcp congestion control to throttle the probe.

OPTIMIZATIONS COMPLETED
1. set the socket's receiveBufferSize to 32768

I don't know how that compares to the system default, but you cannot
hope to make the situation (running out of buffer space) better by
setting this larger than the system's default.

2. set the the buffered input stream's buffer size to 1024000

This is unlikely to be the problem. On the other hand, it's also very
much larger than is likely to be useful.

Did you make these "optimizations" in response to actual system
performance problems? Did they demonstrably help? If not, then I'd
tear them back out until (at least) you have fixed your immediate problem.

SYMPTOMS
1. data in buffer returned by inputStream.read() appears to be off by 2 or 3
bytes

I'm not sure what that means. Bytes are being dropped? A few bytes are
incorrect? Those would be very bad signs, but probably indications of a
native system-level problem. On the other hand, if you just mean that
you are receiving fewer bytes at a time than the receive buffer size
(but no bytes are dropped) then that is perfectly reasonable.

2. no tcp congestion control (advertised window does not decrease)

Whether or not congestion control goes into effect is definitely in the
realm of the system's network stack. I don't know offhand what specific
conditions would be expected to bring it into play. I assume you have
already confirmed that that feature is in fact built into your network
stack and enabled. It is conceivable, though not required (as far as I
can tell), that some particular TCP implementation might disable
congestion control on a socket after the receive buffer size is manually
set on it.

3. according to the server's operating system stats, the receiver queue
builds up

That shows that the server is not keeping up with the data stream, but
I'm not sure what other conclusions we are supposed to draw from it.

4. data input rate is 7 to 8 Meg continuous.

That's the rate at which the server is inputting the data or the
utilization level of the wire? Not that it much matters -- without
hardware and OS details the number is not especially meaningful.

ENVIRONMENT
- running BEA's JRockit JVM
- both SuSE 8.2 and WinXP.
- Lot of processing on the buffer returned by the read call is available for
garbage collection

None of a BufferedInputStream's read() methods returns a buffer. Two of
them store bytes in a buffer provided to them. Whether or not this
buffer is eligible for garbage collection is not likely to be directly
relevant to the question, but you might want to recognize that
performance will be negatively affected by frequent allocation (and
concomitant initialization) of medium- to large-size arrays. These
buffers are unrelated to a socket's native I/O buffers.

SERVER'S SOCKET CODE
Socket probeSocket = sock.accept();
if ( probeSocket.getReceiveBufferSize() < 32768 )
probeSocket.setReceiveBufferSize(32768);

I'd recommend instead that you set the default receive buffer size on
your ServerSocket, and then live with what you actually get instead of
testing each accepted socket. Note also that although a large buffer
helps with throughput if you have high volume over individual
connections, but it may not help (and might even hurt) if you have a
large number of connections with small average volume. For instance, if
the average amount of data transmitted over each socket connection is 1K
then over 95% of your buffer space is going unused. That buffer space
will live in kernel memory, which may be considerably more restricted
than user memory.

BufferedInputStream is = new BufferedInputStream(
probeSocket.getInputStream(), 1024000 );

As I wrote above, that's a _much_ larger buffer than typically gives
advantage. If you have many connections and you do this for each then
you are probably reducing your throughput.

If you have performance problems then _test_ (by profiling) where the
bottlenecks are before you set out to improve matters, and _test_ after
tuning to verify that you have in fact helped things.


John Bollinger
(e-mail address removed)

 

[Jeff]

(e-mail address removed) wrote...

I believe the OP stated it was a single connection that was
saturated.

Correct - it's a single connection that generates the error

 

[John Harlow]

Our server has what appears to be a network IO problem under load.

Can you try smaller (say 16k) "chunks"? I have a feeling you are hitting a
hard limit on some underlying layer that may have issues with large buffer
sizes.

 

[John C. Bollinger]

(e-mail address removed) wrote...



I believe the OP stated it was a single connection that was
saturated.

He had not previously so stated, but he now has done. All the same I
find the assertion surprising.


John Bollinger
(e-mail address removed)

 

[John C. Bollinger]

Correct - it's a single connection that generates the error

That's still not clear. I _think_ you mean that your "probe" connects
just once to the server during a test run, and while connected sends
data over the one connection at a sustained rate of about 7.5 MB/s (from
the original message) for an extended time, and at some point during
this procedure the server fails. You _could_ mean, however, that the
probe makes some number of connections serially, sending some
unspecified amount of data over each one at an overall rate of about 7.5
MB/s, and most of the transmissions complete successfully, but some
random one occasionally fails.

Either way, the exception reported appears to be signaling a resource
exhaustion condition in the network stack. I really don't have enough
information to diagnose it any more specifically, and I certainly
couldn't say whether the Java server implementation has anything to do
with the problem.


John Bollinger
(e-mail address removed)