has someone seen this error before?

Discussion in 'Java' started by Jeff, Nov 3, 2004.

  1. Jeff

    Jeff Guest

    I would greatly appreciate some good advice on a problem that has eluded us.

    Our server has what appears to be a network IO problem under load. We have
    seen one instance of an error message that we think is a strong clue:
    java.net.SocketException: Insufficient buffer space

    However, we can't find any documentation on what this error means? Which
    buffer space?

    ARCHITECTURE
    1. our probe sends large volumes of data to our server
    2. the probe-server connection is tcp carrying proprietary data
    3. at times, our probe definitely sends more data than the server can
    process. We expected tcp congestion control to throttle the probe.

    OPTIMIZATIONS COMPLETED
    1. set the socket's receiveBufferSize to 32768
    2. set the the buffered input stream's buffer size to 1024000

    SYMPTOMS
    1. data in buffer returned by inputStream.read() appears to be off by 2 or 3
    bytes
    2. no tcp congestion control (advertised window does not decrease)
    3. according to the server's operating system stats, the receiver queue
    builds up
    4. data input rate is 7 to 8 Meg continuous.

    ENVIRONMENT
    - running BEA's JRockit JVM
    - both SuSE 8.2 and WinXP.
    - Lot of processing on the buffer returned by the read call is available for
    garbage collection

    SERVER'S SOCKET CODE
    Socket probeSocket = sock.accept();
    if ( probeSocket.getReceiveBufferSize() < 32768 )
    probeSocket.setReceiveBufferSize(32768);

    BufferedInputStream is = new BufferedInputStream(
    probeSocket.getInputStream(), 1024000 );

    Thanks
    --
    Jeff
     
    Jeff, Nov 3, 2004
    #1
    1. Advertising

  2. Jeff

    VK Guest

    Seems more like a queue limitation, rather than buffer.
    Do you have a setting like "tcp_conn_request_max" and what (if any) does it
    say?
     
    VK, Nov 3, 2004
    #2
    1. Advertising

  3. Jeff

    Louis Guest

    SocketException: Insufficient buffer space (Re: has someone seen this error before?)

    wrote...
    > Seems more like a queue limitation, rather than buffer.
    > Do you have a setting like "tcp_conn_request_max" and what (if any) does it
    > say?


    I believe the OP stated it was a single connection that was
    saturated.
     
    Louis, Nov 3, 2004
    #3
  4. Jeff wrote:

    > I would greatly appreciate some good advice on a problem that has eluded us.
    >
    > Our server has what appears to be a network IO problem under load. We have
    > seen one instance of an error message that we think is a strong clue:
    > java.net.SocketException: Insufficient buffer space
    >
    > However, we can't find any documentation on what this error means? Which
    > buffer space?


    I don't _know_, but a good guess would be that the exception wraps an
    error signaled by the OS' native socket implementation. The buffer
    would be one of the socket's native I/O buffers. Such an error probably
    means that you have too many active sockets, whether because there are
    many sockets simultaneously in use or because discarded sockets are not
    cleaned up. How many is too many would depend in part on system
    resources, including file descriptors and kernel memory among other things.

    It might not be the core problem, but you should be sure _always_ to
    close() sockets when you are done with them. Typically you would ensure
    this by putting the close() invocation in a finally {} block associated
    with a suitably scoped try {} block.

    > ARCHITECTURE
    > 1. our probe sends large volumes of data to our server
    > 2. the probe-server connection is tcp carrying proprietary data
    > 3. at times, our probe definitely sends more data than the server can
    > process. We expected tcp congestion control to throttle the probe.
    >
    > OPTIMIZATIONS COMPLETED
    > 1. set the socket's receiveBufferSize to 32768


    I don't know how that compares to the system default, but you cannot
    hope to make the situation (running out of buffer space) better by
    setting this larger than the system's default.

    > 2. set the the buffered input stream's buffer size to 1024000


    This is unlikely to be the problem. On the other hand, it's also very
    much larger than is likely to be useful.

    Did you make these "optimizations" in response to actual system
    performance problems? Did they demonstrably help? If not, then I'd
    tear them back out until (at least) you have fixed your immediate problem.

    > SYMPTOMS
    > 1. data in buffer returned by inputStream.read() appears to be off by 2 or 3
    > bytes


    I'm not sure what that means. Bytes are being dropped? A few bytes are
    incorrect? Those would be very bad signs, but probably indications of a
    native system-level problem. On the other hand, if you just mean that
    you are receiving fewer bytes at a time than the receive buffer size
    (but no bytes are dropped) then that is perfectly reasonable.

    > 2. no tcp congestion control (advertised window does not decrease)


    Whether or not congestion control goes into effect is definitely in the
    realm of the system's network stack. I don't know offhand what specific
    conditions would be expected to bring it into play. I assume you have
    already confirmed that that feature is in fact built into your network
    stack and enabled. It is conceivable, though not required (as far as I
    can tell), that some particular TCP implementation might disable
    congestion control on a socket after the receive buffer size is manually
    set on it.

    > 3. according to the server's operating system stats, the receiver queue
    > builds up


    That shows that the server is not keeping up with the data stream, but
    I'm not sure what other conclusions we are supposed to draw from it.

    > 4. data input rate is 7 to 8 Meg continuous.


    That's the rate at which the server is inputting the data or the
    utilization level of the wire? Not that it much matters -- without
    hardware and OS details the number is not especially meaningful.

    > ENVIRONMENT
    > - running BEA's JRockit JVM
    > - both SuSE 8.2 and WinXP.
    > - Lot of processing on the buffer returned by the read call is available for
    > garbage collection


    None of a BufferedInputStream's read() methods returns a buffer. Two of
    them store bytes in a buffer provided to them. Whether or not this
    buffer is eligible for garbage collection is not likely to be directly
    relevant to the question, but you might want to recognize that
    performance will be negatively affected by frequent allocation (and
    concomitant initialization) of medium- to large-size arrays. These
    buffers are unrelated to a socket's native I/O buffers.

    > SERVER'S SOCKET CODE
    > Socket probeSocket = sock.accept();
    > if ( probeSocket.getReceiveBufferSize() < 32768 )
    > probeSocket.setReceiveBufferSize(32768);


    I'd recommend instead that you set the default receive buffer size on
    your ServerSocket, and then live with what you actually get instead of
    testing each accepted socket. Note also that although a large buffer
    helps with throughput if you have high volume over individual
    connections, but it may not help (and might even hurt) if you have a
    large number of connections with small average volume. For instance, if
    the average amount of data transmitted over each socket connection is 1K
    then over 95% of your buffer space is going unused. That buffer space
    will live in kernel memory, which may be considerably more restricted
    than user memory.

    > BufferedInputStream is = new BufferedInputStream(
    > probeSocket.getInputStream(), 1024000 );


    As I wrote above, that's a _much_ larger buffer than typically gives
    advantage. If you have many connections and you do this for each then
    you are probably reducing your throughput.

    If you have performance problems then _test_ (by profiling) where the
    bottlenecks are before you set out to improve matters, and _test_ after
    tuning to verify that you have in fact helped things.


    John Bollinger
     
    John C. Bollinger, Nov 3, 2004
    #4
  5. Jeff

    Jeff Guest

    Re: SocketException: Insufficient buffer space (Re: has someone seen this error before?)

    "Louis" <> wrote in message
    news:...
    > wrote...
    >> Seems more like a queue limitation, rather than buffer.
    >> Do you have a setting like "tcp_conn_request_max" and what (if any) does
    >> it
    >> say?

    >
    > I believe the OP stated it was a single connection that was
    > saturated.
    >


    Correct - it's a single connection that generates the error
     
    Jeff, Nov 3, 2004
    #5
  6. Jeff

    John Harlow Guest

    > Our server has what appears to be a network IO problem under load.

    Can you try smaller (say 16k) "chunks"? I have a feeling you are hitting a
    hard limit on some underlying layer that may have issues with large buffer
    sizes.
     
    John Harlow, Nov 3, 2004
    #6
  7. Re: SocketException: Insufficient buffer space (Re: has someone seenthis error before?)

    Louis wrote:

    > wrote...
    >
    >>Seems more like a queue limitation, rather than buffer.
    >>Do you have a setting like "tcp_conn_request_max" and what (if any) does it
    >>say?

    >
    >
    > I believe the OP stated it was a single connection that was
    > saturated.


    He had not previously so stated, but he now has done. All the same I
    find the assertion surprising.


    John Bollinger
     
    John C. Bollinger, Nov 3, 2004
    #7
  8. Re: SocketException: Insufficient buffer space (Re: has someone seenthis error before?)

    Jeff wrote:

    > "Louis" <> wrote in message
    > news:...
    >
    >> wrote...
    >>
    >>>Seems more like a queue limitation, rather than buffer.
    >>>Do you have a setting like "tcp_conn_request_max" and what (if any) does
    >>>it
    >>>say?

    >>
    >>I believe the OP stated it was a single connection that was
    >>saturated.
    >>

    >
    >
    > Correct - it's a single connection that generates the error


    That's still not clear. I _think_ you mean that your "probe" connects
    just once to the server during a test run, and while connected sends
    data over the one connection at a sustained rate of about 7.5 MB/s (from
    the original message) for an extended time, and at some point during
    this procedure the server fails. You _could_ mean, however, that the
    probe makes some number of connections serially, sending some
    unspecified amount of data over each one at an overall rate of about 7.5
    MB/s, and most of the transmissions complete successfully, but some
    random one occasionally fails.

    Either way, the exception reported appears to be signaling a resource
    exhaustion condition in the network stack. I really don't have enough
    information to diagnose it any more specifically, and I certainly
    couldn't say whether the Java server implementation has anything to do
    with the problem.


    John Bollinger
     
    John C. Bollinger, Nov 3, 2004
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Rob G
    Replies:
    2
    Views:
    2,291
    Rob G
    May 24, 2004
  2. David
    Replies:
    2
    Views:
    930
    David
    Jul 6, 2004
  3. Replies:
    3
    Views:
    493
    Juan T. Llibre
    Nov 4, 2005
  4. jens Jensen

    Has anyone seen this before?

    jens Jensen, May 2, 2006, in forum: ASP .Net
    Replies:
    0
    Views:
    307
    jens Jensen
    May 2, 2006
  5. Replies:
    8
    Views:
    877
Loading...

Share This Page