There simply has to be some multithreaded paradigm/pattern knowledge
that goes along io_service/bind/callback type situations.
I've written listeners that I've tested with more than thousand
connections. In retrospect (on looking at the code), I can see
potential deadlocks if the servers to the wrong thing, but luckily my
servers and listeners interact within my control.
You concerns aren't separated well in my opinion. My listener service
would accept a descriptors requiring accepting and publish/dispatch
them to the interested server associated with the listening
descriptor. It would then simply continue listening on the same
descriptor.
The server/subscriber would become responsible for the socket. In my
case I had a further one to many from Server to ServerSockets. The
listener was therefore opaque.
- You would create a server servicing a port/ip combination.
- You would create a socket and associate it with the applicable
server.
- The server would, upon receiving the accepted socket, create a
Connection object and hook it up with all the relevant services (such
as the service responsible for receiving data).
- It would then publish the Connection to interested/associated
servers.
- The first server "accepting" the connection (where accepting implies
an accept on application level, than might be based on an application
layer protocol) would steal ownership of the connection. Unaccepted
connections would be discarded/closed after some time be the server.
The server loop looked something to this effect...
void TcpSvrImpl::serviceConnections()
{
std::list<IfToSock*>::iterator socket;
std::list<TcpSvrConnection*>::iterator connection;
for( socket = connectNotifyList_.begin();
socket != connectNotifyList_.end();
++socket )
{
for( connection = connectionList_.begin();
connection != connectionList_.end();
++connection )
{
if( (*socket)->processingConnection() )
{
//Current socket is processing some connection -
// chances are allmost 100% it will continue
// doing so during this loop...This socket cannot
// be serviced further as it is still busy
// processing a connection
break;
}
else if( (*socket)->hasAcceptedConnection() )
{
//Current Socket has accepted a connection
// - remove from notification list
(*socket)->resetSvr();
//remove the connection itself from connection list as we
allow only
// one TcpSvrConnection to be assigned to one TcpSvrSock
if( (*socket)->isAssociatedWith( *connection ) )
{
connection = connectionList_.erase( connection );
}
socket = connectNotifyList_.erase( socket );
break;
}
else if( (*socket)->rejectedConnection( **connection ) )
{
//Current socket has rejected this connection previously, move
on to
// evaluate if this socket has processed the next connection
in loop
continue;
}
else if( !(*connection)->hasBeenAccepted() )
{
//Attempt to associate this connection with the socket
(*socket)->setConnection( **connection );
break;
}
}
//Handle the case where the only socket was erased, meaning socket
// is pointing to end.
if( socket == connectNotifyList_.end() )
{
break; //Breaking from outer loop.
}
}
}
I can mention that I had to handle the case of sockets being erased
while connections were being dispatched. This I did by using a concept
called a rendezvous. The context (or thread of execution) from where
sockets would dissociate themselves from the server would block until
the server would complete its loop. The code looked something like
this:
void TcpSvrImpl::detachFromConnectNotify( IfToSock& socket )
{
//- Create an asynchronous callback with argument (socket)
// that will execute in the context of thread associated
// with "cmdSvr_".
//- Wait for this callback to complete (synchCmd...) before
// continuing this threads execution.
cmdRetArg1StoreArg(
*this, &TcpSvrImpl::detachSockFromTcpSvrImpl, cmdSvr_ ).synchCmd(
syncDetachFromConnNotify_ )( &socket );
}
.... and the asynchronous call made into the servicing thread...
void TcpSvrImpl::detachSockFromTcpSvrImpl( IfToSock* socket )
{
if( RtsCurrent_TcpSvrImpl ==
RtsTcpSvrImplTcpSvrImpl_States_Created )
{
socket->resetSvr();
connectNotifyList_.remove( socket );
}
else
{
/* Event not handled */
}
RtsRunToCompletion();
}
Once the connection is the responsibility of the socket, things are
easier. You still require a way for the asynchronous reading service
to notify the socket when the peer disconnects though...
