Question about non-blocking NIO and Selection Keys

[Zachary Turner]

I am new to non-blocking socket i/o so please go easy on my terrible
code below

That being said, I have a socket that must be used for reading and
writing, but nothing else. So I initialize my socket early on as:

asyncSelector = new Selector();
sockC.configureBlocking(false);
sockC.register(asyncSelector, SelectionKey.OP_WRITE |
SelectionKey.OP_READ);

Now, whenever I need to write to my socket, I do the following:

public void write(ByteBuffer buf) throws ServerException {
int bytesWritten = 0;
int bytesToWrite = buf.remaining();

while (bytesWritten < bytesToWrite) {
limitBuffer(buf, bytesToWrite, bytesWritten,
MAX_BYTES_PER_BUF_WRITE);
waitForOperation(SelectionKey.OP_WRITE);

int bytesWrittenThisTime = sockC.write(buf);
if (bytesWrittenThisTime == -1)
throw new ServerException("Remote host unexpectedly closed
the connection");
else
bytesWritten += bytesWrittenThisTime;
}
}

public void waitForOperation(int operation) throws ServerException {
while (true) {
try {
if (asyncSelector.select() == 0)
continue;
Set<SelectionKey> selected = asyncSelector.selectedKeys();
for (Iterator<SelectionKey> i = selected.iterator();
i.hasNext(); ) {
if ((i.next().readyOps() & operation) != 0)
return;
}
}
catch (IOException e) {
throw new ServerException("An error occured while performing
a socket operation.");
}
}
}


When I need to read from the socket, I do something very similar.

Now first let me say I know this is poorly designed, but all of this
code is contained in a class that was originally designed to
encapsulate a blocking socket. Then an entire application was built
around this class, and then a specific need was encountered to make it
non-blocking, and short of re-designing the entire application (which
is not an option at all) this seemed like the best way to go about
doing it.

The ultimate problem I was trying to solve is that I had a blocking
socket that would start sending tons of data, gigabytes even. At any
point in time, the client might say "I don't want this data anymore",
and instead of waiting for 17 gigabytes fo data to be sent, which
could take quite a while, it should be able to send the server a code
which would make the server stop sending data. Before we had this
need, a blocking socket was fine. But now, the server breaks the data
up into chunks and sends it in a tight loop. The first line of the
loop simply checks if SelectionKey.OP_READ is selected and if so, it
can read the code from the client and possibly stop sending data.

The problem I am having is that if I run a previous version of the
application it is much, much, much faster. I'm not sure how to tell
if this NIO is even the culprit, as a few other changes were made as
well, but this one is the most fundamental and highest visibility, and
as such I think it's most likely to be the culprit. Can anyone
provide any observations or suggestions as how to improve this code?

Thanks

 

[Mark Space]

I am new to non-blocking socket i/o so please go easy on my terrible
code below

That being said, I have a socket that must be used for reading and
writing, but nothing else. So I initialize my socket early on as:

asyncSelector = new Selector();
sockC.configureBlocking(false);
sockC.register(asyncSelector, SelectionKey.OP_WRITE |
SelectionKey.OP_READ);

Now, whenever I need to write to my socket, I do the following:

public void write(ByteBuffer buf) throws ServerException {
int bytesWritten = 0;
int bytesToWrite = buf.remaining();

while (bytesWritten < bytesToWrite) {
limitBuffer(buf, bytesToWrite, bytesWritten,
MAX_BYTES_PER_BUF_WRITE);
waitForOperation(SelectionKey.OP_WRITE);

int bytesWrittenThisTime = sockC.write(buf);
if (bytesWrittenThisTime == -1)
throw new ServerException("Remote host unexpectedly closed
the connection");
else
bytesWritten += bytesWrittenThisTime;
}
}

public void waitForOperation(int operation) throws ServerException {
while (true) {
try {
if (asyncSelector.select() == 0)
continue;
Set<SelectionKey> selected = asyncSelector.selectedKeys();
for (Iterator<SelectionKey> i = selected.iterator();
i.hasNext(); ) {
if ((i.next().readyOps() & operation) != 0)
return;
}
}
catch (IOException e) {
throw new ServerException("An error occured while performing
a socket operation.");
}
}
}

as such I think it's most likely to be the culprit. Can anyone
provide any observations or suggestions as how to improve this code?

No, but I'm no NIO expert so don't dispair yet. However I do have a
couple of observations.

First, writting a small number of bytes seems counter productive with
NIO. If you are blocking, it make sense to limit the number of bytes
written so you don't block too long. With NIO, this shouldn't be an
issue, so for efficiency (speed) you should just hand NIO as much data
as you have and let it deal with sending it out.

> limitBuffer(buf, bytesToWrite, bytesWritten,
> MAX_BYTES_PER_BUF_WRITE);

The above line makes no sense to me. Why limit the bytes if the
operation won't block?

Second, you say your IO needs to be non-blocking, but you have this
method call with the word "wait" in it right in the middle of your
routine, which seems suspect. If you don't want to block, why wait? It
makes no sense. In particular, this line:

> if (asyncSelector.select() == 0)

will block, so you might be slowing yourself down there. It's worth
looking into, I think.

Good luck.

 

[Zachary Turner]

Second, you say your IO needs to be non-blocking, but you have this
method call with the word "wait" in it right in the middle of your
routine, which seems suspect.  If you don't want to block, why wait?  It
makes no sense.  In particular, this line:

 >          if (asyncSelector.select() == 0)

will block, so you might be slowing yourself down there.  It's worth
looking into, I think.

Good luck.- Hide quoted text -

Well, most of this is due to the fact that it was originally designed
around blocking sockets, and without a complete re-write it's hard to
do it correctly. The idea is that I want to wait *until the socket is
ready for writing*, then I want to write N bytes of data and return
immediately.

The sole reason this was changed from blocking to non-blocking is
because we need to support the case where the client can
asynchronously send a message to the server telling it to stop sending
more data, so the server needs to be able to do a non-blocking read,
because otherwise there's no way to say "is there data waiting? if so
read it, if not let me do other stuff".

I've looked into the possibility of converting everything back to
blocking sockets. Then I would have to have one thread writing and
one thread reading from the same SocketChannel instance, which is
apparently supported according to the docs. The problem though is now
I have all this complex synchronization between the two threads,
because after writing a certain sequence of bytes, I then need to read
a certain response, then send other bytes. Currently I just have
reads and writes interspersed sequentially in the order that I need
them, it becomes more difficult if I have to force this synchronicity
across multiple threads, but I guess it's an option if I have to.

 

[Christian]

I am new to non-blocking socket i/o so please go easy on my terrible
code below

That being said, I have a socket that must be used for reading and
writing, but nothing else. So I initialize my socket early on as:

asyncSelector = new Selector();
sockC.configureBlocking(false);
sockC.register(asyncSelector, SelectionKey.OP_WRITE |
SelectionKey.OP_READ);

Now, whenever I need to write to my socket, I do the following:

public void write(ByteBuffer buf) throws ServerException {
int bytesWritten = 0;
int bytesToWrite = buf.remaining();

while (bytesWritten < bytesToWrite) {
limitBuffer(buf, bytesToWrite, bytesWritten,
MAX_BYTES_PER_BUF_WRITE);
waitForOperation(SelectionKey.OP_WRITE);

int bytesWrittenThisTime = sockC.write(buf);
if (bytesWrittenThisTime == -1)
throw new ServerException("Remote host unexpectedly closed
the connection");
else
bytesWritten += bytesWrittenThisTime;
}
}

public void waitForOperation(int operation) throws ServerException {
while (true) {
try {
if (asyncSelector.select() == 0)
continue;
Set<SelectionKey> selected = asyncSelector.selectedKeys();
for (Iterator<SelectionKey> i = selected.iterator();
i.hasNext(); ) {
if ((i.next().readyOps() & operation) != 0)
return;
}
}
catch (IOException e) {
throw new ServerException("An error occured while performing
a socket operation.");
}
}
}


When I need to read from the socket, I do something very similar.

Now first let me say I know this is poorly designed, but all of this
code is contained in a class that was originally designed to
encapsulate a blocking socket. Then an entire application was built
around this class, and then a specific need was encountered to make it
non-blocking, and short of re-designing the entire application (which
is not an option at all) this seemed like the best way to go about
doing it.

The ultimate problem I was trying to solve is that I had a blocking
socket that would start sending tons of data, gigabytes even. At any
point in time, the client might say "I don't want this data anymore",
and instead of waiting for 17 gigabytes fo data to be sent, which
could take quite a while, it should be able to send the server a code
which would make the server stop sending data. Before we had this
need, a blocking socket was fine. But now, the server breaks the data
up into chunks and sends it in a tight loop. The first line of the
loop simply checks if SelectionKey.OP_READ is selected and if so, it
can read the code from the client and possibly stop sending data.

The problem I am having is that if I run a previous version of the
application it is much, much, much faster. I'm not sure how to tell
if this NIO is even the culprit, as a few other changes were made as
well, but this one is the most fundamental and highest visibility, and
as such I think it's most likely to be the culprit. Can anyone
provide any observations or suggestions as how to improve this code?

Thanks

If you need to send data while reading from the socket nothing stops you
from Using a second thread to write the stop signal. The writing on the
other side can then be interrupted by the reading thread.


Seems to be easier than what you did.
NIO is only really useful if you use just one thread for reading from
several sockets.. using one Thread per socket with NIO spoils the benefits.


Christian

also If you ever do a rewrite I just recently found a library by Apache
called Mina which seems very promissing and easy to use.

 

[Arne Vajhøj]

I thought NIO was for reading several connections to the same socket,
using one thread per socket instead of the traditional one thread per
connection.

Considering that a single ServerSocket(Channel) listen and
accepts multiple Socket(Channel) one for each connection, then
there are no difference.

Arne

 

[Neil Coffey]

Well, most of this is due to the fact that it was originally designed
around blocking sockets, and without a complete re-write it's hard to
do it correctly. The idea is that I want to wait *until the socket is
ready for writing*, then I want to write N bytes of data and return
immediately.

The sole reason this was changed from blocking to non-blocking is
because we need to support the case where the client can
asynchronously send a message to the server telling it to stop sending
more data, so the server needs to be able to do a non-blocking read,
because otherwise there's no way to say "is there data waiting? if so
read it, if not let me do other stuff".

Isn't your communication protocol just broken, then? What you
seem to be saying is that sometimes you need write/reads to be
sequenced, but sometimes you don't... Possibly things could be improved
by doing one of the following:

- use a separate connection for the asynchronous control message
(obviously wasteful, but OK for a small number of connections)
- make everything synchronous: after every portion of data sent, you
expect a response from the client "OK got that" or "OK got that but
don't send more"; if there's no data for you to send in X seconds,
send some "dummy" message (and expect a response) just to check if
the client is wanting to stop communication
- Of course, the client could also signal to the server that it
didn't want more data by closing the connection...

Otherwise, I'd suggest trying to write things so that you have
(a) a single selector thread, and (b) for each connection, a 'state'
object which encompasses things like the data you're currently
trying to send, and whether the next data you expect to read is
in response to previously sent data. You can then attach your state
object to the selection key (check the SelectionKey.attach() method if
you haven't come across it). In your single selector loop, when you
get a 'ready to write', you send the next bit of data from your
state object; when you get a 'ready to read', you check if you're
expecting a response and process it, else assume it's one of your
asynchronous control messages.

It's not entirely clear to me that this is any easier or more difficult
than the synchronising you'd need between threads (each time your
read/write thread came back, you'd effectively then synchronize
and update a similar state object...). But if you want to stick to
the selector route, I think it'll be easier sticking to the standard
single-selector pattern I've outlined above.

Neil

 

[Zachary Turner]

If you need to send data while reading from the socket nothing stops you
from Using a second thread to write the stop signal. The writing on the
other side can then be interrupted by the reading thread.

However, interrupting a blocked read operation causes the channel to
be closed if I understand correctly. This is not an option, as I am
not done with the channel. Furthermore, reads and writes have to be
sequenced with each other. I need to write something that tells the
client i'm about to send data, then only after that is done i have to
wait for the client to send me something saying ok, and only after
that is done do i have to actually send the data to the client. It
really seems easier just to use a non-blocking socket. I know that
NIO is "most useful" when used in conjuction with selecting off of
lots of different sockets and handling tons of connections at the same
time, but making a separate thread only to completely serialize 90% of
their execution with each other is just as using NIO the way I
proposed.

Of course, I could be misunderstanding something. If the write thread
really could interrupt the read thread and not have my SocketChannel
be closed as a result, a separate thread would trivialize this
approach and make it easily superior. This way I could block in read,
have the write thread interrupt it if it gets all the way to the end
of the file successfully, then continue about my work sending the next
file to the same client.

 

[Zachary Turner]

Isn't your communication protocol just broken, then? What you
seem to be saying is that sometimes you need write/reads to be
sequenced, but sometimes you don't... Possibly things could be improved
by doing one of the following:

- use a separate connection for the asynchronous control message
   (obviously wasteful, but OK for a small number of connections)

This could be an option, I considered it initially but wrote it off
without much thought because non-blockign sockets seemed like the way
to go.

- make everything synchronous: after every portion of data sent, you
   expect a response from the client "OK got that" or "OK got that but
   don't send more"; if there's no data for you to send in X seconds,
   send some "dummy" message (and expect a response) just to check if
   the client is wanting to stop communication

Actually tried this, but considering the volume of data being sent,
the constant round trips back to the server were having a noticeable
performance impact. I got about 10% higher throughput just by
removing a 4 byte acknowledgement from client -> server after every
chunk of data.

- Of course, the client could also signal to the server that it
   didn't want more data by closing the connection...

Well, it's really more just like "I don't want any more of THAT
data." But the server still has lots of other data to send.


The problem is that whatever I do has to fit nicely and transparently
into the implementation of a class that used to represent a blocking
socket, and is now scattered across 10s of thousands of lines of code
of our server. So what you said at the very beginning is pretty much
spot on. It needs to be blockign most of the time, but sometimes it
doesn't. I know it sounds stupid, but that's what happens when stuff
is designed poorly in the beginning and you just have to make
something work.

What I'm after is whatever is the closest I can possibly get to a
blocking socket with the added ability to "peek" at the socket's recv
buffer.

 

[Christian]

However, interrupting a blocked read operation causes the channel to
be closed if I understand correctly. This is not an option, as I am
not done with the channel. Furthermore, reads and writes have to be
sequenced with each other. I need to write something that tells the
client i'm about to send data, then only after that is done i have to
wait for the client to send me something saying ok, and only after
that is done do i have to actually send the data to the client. It
really seems easier just to use a non-blocking socket. I know that
NIO is "most useful" when used in conjuction with selecting off of
lots of different sockets and handling tons of connections at the same
time, but making a separate thread only to completely serialize 90% of
their execution with each other is just as using NIO the way I
proposed.

Of course, I could be misunderstanding something. If the write thread
really could interrupt the read thread and not have my SocketChannel
be closed as a result, a separate thread would trivialize this
approach and make it easily superior. This way I could block in read,
have the write thread interrupt it if it gets all the way to the end
of the file successfully, then continue about my work sending the next
file to the same client.

No interrupting will close the socket.
Though what you can do is ser some com variable and check that. Like
this you can send some "stop after next read" .. or specify that the
next read data is actually already for the next destination file...

Christian

 

[Zachary Turner]

Zachary Turner schrieb:







No interrupting will close the socket.
Though what you can do is ser some com variable and check that. Like
this you can send some "stop after next read" .. or specify that the
next read data is actually already for the next destination file...

Christian- Hide quoted text -

- Show quoted text -

FWIW I fixed the performance problem (which only occured on windows
for some odd reason) by disabling nagle algorithm on the socket. I'm
at a little bit of a loss as to why this would fix anything. My
understanding of nagle is that it's good in a situation where you have
multiple small writes followed by a read. or even just lots of small
writes. But I don't, I have like one small write followed by lots of
huge writes, with no reads anywhere in sight.

 

[Mark Space]

FWIW I fixed the performance problem (which only occured on windows
for some odd reason) by disabling nagle algorithm on the socket. I'm

That's very strange to me. I was certain that it was going to be
because of blocking or buffer manipulation. Interesting, Nagle on
Windows = suspect.

I'm glad you got it sorted, and thanks very much for report back what
the solution turned out to be.

 

[Zachary Turner]

That's very strange to me.  I was certain that it was going to be
because of blocking or buffer manipulation.  Interesting, Nagle on
Windows = suspect.

I'm glad you got it sorted, and thanks very much for report back what
the solution turned out to be.

Well it's even stranger now. Up until I disabled Nagle it was slow
(around 600K / second throughput) every single time I ran this code
path, probably 20 times total over the course of 2 days. Disabled
nagle, instant fix. Re-enabled it, slow. Went back and forth like
this about 5 times to make sure the results were consistent. Now all
of a sudden the performance problems just disappeared no matter how I
set Nagle. Been testing it for about 2 hours and no matter what I do
I actually can't get it to be slow again. These are the most annoying
types of problems That being said, I did re-organize the socket
code and fixed some inefficiencies in that area, but the basic model
of how I hacked the asynchronicity into the blocking socket is still
pretty much the same. Guess I'll shoot it run a stress test and see
if the results stay good across multiple platforms / physical
networks. /shrug

 

[Arne Vajhøj]

The general term "socket" applies to both ends, but I admit I was
thinking only of the server side in my response. Of course, you're
absolutely spot on - there would be more than one client socket per NIO
thread.

I was talking about the classes and instances of those used server side.

Arne