start a multi-sockets server (a socket/per thread) with different ports but same host

Discussion in 'Python' started by zxo102, Aug 12, 2006.

  1. zxo102

    zxo102 Guest

    Hi,
    I am doing a small project using socket server and thread in python.
    This is first time for me to use socket and thread things.
    Here is my case. I have 20 socket clients. Each client send a set
    of sensor data per second to a socket server. The socket server will
    do two things: 1. write data into a file via bsddb; 2. forward the data
    to a GUI written in wxpython.
    I am thinking the code should work as follow (not sure it is
    feasible)
    20 threads, each thread takes care of a socket server with a
    different port.
    I want all socket servers start up and wait for client connection.
    In the attached demo code, It stops at the startup of first socket
    server somewhere in the following two lines and waits for client call:

    lstn.listen(5)
    (clnt,ap) = lstn.accept()

    Any ideas how to handle these 20 clients? Really appreciate your
    suggestions.

    Thanks a lot.

    Ouyang


    import socket
    import sys
    import threading
    class srvr(threading.Thread):
    v = ''
    vlock = threading.Lock()
    id = 0 # next available thread number
    def __init__(self,clntsock):
    threading.Thread.__init__(self)
    self.myid = srvr.id
    srvr.id += 1
    self.myclntsock = clntsock
    def run(self):
    while 1:
    k = self.myclntsock.recv(1)
    if k == '': break
    # update v in an atomic manner
    srvr.vlock.acquire()
    srvr.v += k
    srvr.vlock.release()
    self.myclntsock.send(srvr.v)
    self.myclntsock.close()

    #lstn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    #port = int(sys.argv[1]) # server port number
    #lstn.bind(('', port))
    #lstn.listen(5)
    nclnt = 20
    mythreads = [] # list of all the threads

    for i in range(nclnt):
    lstn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    lstn.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    lstn.bind(('', 2000+i+1))
    lstn.listen(5)
    (clnt,ap) = lstn.accept()
    s = srvr(clnt)
    mythreads.append(s)
    s.start()

    # shut down the server socket, since it's not needed anymore
    #lstn.close()

    # wait for all threads to finish
    for s in mythreads:
    s.join()

    print 'the final value of v is', srvr.v
     
    zxo102, Aug 12, 2006
    #1
    1. Advertising

  2. zxo102

    zxo102 Guest

    Jean-Paul,
    Thanks a lot. The code is working. The python twisted is new to me too.
    Here are my three more questions:
    1. Since the code need to be started in a wxpyhon GUI (either by
    clicking a button or up with the GUI), do I have to run the code in a
    thread (sorry, I have not tried it yet)?
    2. How can I grab the client data in the code? Can you write two lines
    for that? I really appreciate that.
    3. After I change
    self.transport.write(''.join(self.data))
    to
    self.transport.write(''.join(data))
    and scan all the ports with the following code twice (run twice).
    First round scanning says "succefully connected". But second round
    scanning says "failed". I have to restart your demo code to make it
    work.

    Ouyang


    import sys, threading, socket

    class scanner(threading.Thread):
    tlist = [] # list of all current scanner threads
    maxthreads = int(sys.argv[2]) # max number of threads we're
    allowing
    evnt = threading.Event() # event to signal OK to create more
    threads
    lck = threading.Lock() # lock to guard tlist
    def __init__(self,tn,host):
    threading.Thread.__init__(self)
    #self.threadnum = tn # thread ID/port number
    self.threadnum = 2000+tn # thread ID/port number
    self.host = host # checking ports on this host
    def run(self):
    s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    try:
    s.connect((self.host, self.threadnum))
    print "%d: successfully connected" % self.threadnum
    s.close()
    except:
    print "%d: connection failed" % self.threadnum
    # thread is about to exit; remove from list, and signal OK if we
    # had been up against the limit
    scanner.lck.acquire()
    scanner.tlist.remove(self)
    print "%d: now active --" % self.threadnum, scanner.tlist
    if len(scanner.tlist) == scanner.maxthreads-1:
    scanner.evnt.set()
    scanner.evnt.clear()
    scanner.lck.release()
    def newthread(pn,hst):
    scanner.lck.acquire()
    sc = scanner(pn,hst)
    scanner.tlist.append(sc)
    scanner.lck.release()
    sc.start()
    print "%d: starting check" % pn
    print "%d: now active --" % pn, scanner.tlist
    newthread = staticmethod(newthread)

    def main():
    host = sys.argv[1]
    #for i in range(1,100):
    for i in range(20):
    scanner.lck.acquire()
    print "%d: attempting check" % i
    # check to see if we're at the limit before starting a new thread
    if len(scanner.tlist) >= scanner.maxthreads:
    # too bad, need to wait until not at thread limit
    print "%d: need to wait" % i
    scanner.lck.release()
    scanner.evnt.wait()
    else:
    scanner.lck.release()
    scanner.newthread(i,host)
    for sc in scanner.tlist:
    sc.join()

    if __name__ == '__main__':
    main()






    Jean-Paul Calderone 写é“:

    > On 12 Aug 2006 09:00:02 -0700, zxo102 <> wrote:
    > >Hi,
    > > I am doing a small project using socket server and thread in python.
    > > This is first time for me to use socket and thread things.
    > > Here is my case. I have 20 socket clients. Each client send a set
    > >of sensor data per second to a socket server. The socket server will
    > >do two things: 1. write data into a file via bsddb; 2. forward the data
    > >to a GUI written in wxpython.
    > > I am thinking the code should work as follow (not sure it is
    > >feasible)
    > > 20 threads, each thread takes care of a socket server with a
    > >different port.
    > > I want all socket servers start up and wait for client connection.
    > >In the attached demo code, It stops at the startup of first socket
    > >server somewhere in the following two lines and waits for client call:
    > >

    >
    > Threads aren't the best way to manage the concurrency present in this
    > application. Instead, consider using non-blocking sockets with an
    > event notification system. For example, using Twisted, your program
    > might look something like this:
    >
    > from twisted.internet import reactor, protocol, defer
    >
    > class CumulativeEchoProtocol(protocol.Protocol):
    > def connectionMade(self):
    > # Stop listening on the port which accepted this connection
    > self.factory.port.stopListening()
    >
    > # Set up a list in which to collect the bytes which we receive
    > self.received = []
    >
    >
    > def connectionLost(self, reason):
    > # Notify the main program that this connection has been lost, so
    > # that it can exit the process when there are no more connections.
    > self.factory.onConnectionLost.callback(self)
    >
    > def dataReceived(self, data):
    > # Accumulate the new data in our list
    > self.received.append(data)
    > # And then echo the entire list so far back to the client
    > self.transport.write(''.join(self.data))
    >
    > def allConnectionsLost():
    > # When all connections have been dropped, stop the reactor so the
    > # process can exit.
    > reactor.stop()
    >
    > def main():
    > # Set up a list to collect Deferreds in. When all of these Deferreds
    > # have had callback() invoked on them, the reactor will be stopped.
    > completionDeferreds = []
    > for i in xrange(20):
    > # Make a new factory for this port
    > f = protocol.ServerFactory()
    >
    > # Make a Deferred for this port's connection-lost event and make
    > # it available to the protocol by way of the factory.
    > d = defer.Deferred()
    > f.onConnectionLost = d
    > completionDeferreds.append(d)
    > f.protocol = CumulativeEchoProtocol
    >
    > # Start listening on a particular port number with this factory
    > port = reactor.listenTCP(2000 + i + 1, f)
    >
    > # Make the port object available to the protocol as well, so that
    > # it can be shut down when a connection is made.
    > f.port = port
    >
    > # Create a Deferred which will only be called back when all the other
    > # Deferreds in this list have been called back.
    > d = defer.DeferredList(completionDeferreds)
    >
    > # And tell it to stop the reactor when it fires
    > d.addCallback(lambda result: allConnectionsLost())
    >
    > # Start the reactor so things can start happening
    > reactor.run()
    >
    > if __name__ == '__main__':
    > main()
    >
    > Hope this helps,
    >
    > Jean-Paul
     
    zxo102, Aug 12, 2006
    #2
    1. Advertising

  3. zxo102

    zxo102 Guest

    Jean-Paul,
    I just start to learn Twisted. Here is my simple case: I can find
    the data sent by clients in dataReceived but I don't know which
    client/which port the data is from. After I know where the data comes
    from, I can do different things there, for example, write them into
    different files via bsddb. I am not sure if it is the correct way to
    do it.


    def dataReceived(self, data):
    # Accumulate the new data in our list
    self.received.append(data)
    # And then echo the entire list so far back to the client
    self.transport.write(''.join(data))

    print "============> data: ", data
    print " which Port? : ", self.factory.port # unforunately it is
    an object here.

    # if Port == 2001:
    # write the data into a file via bsddb
    # if Port == 2002:
    # write the data into another file via bsddb
    # etc .....


    Ouyang

    Jean-Paul Calderone 写é“:

    > On 12 Aug 2006 10:44:29 -0700, zxo102 <> wrote:
    > >Jean-Paul,
    > >Thanks a lot. The code is working. The python twisted is new to me too.
    > >Here are my three more questions:
    > >1. Since the code need to be started in a wxpyhon GUI (either by
    > >clicking a button or up with the GUI), do I have to run the code in a
    > >thread (sorry, I have not tried it yet)?

    >
    > You can try to use Twisted's wxPython integration support:
    >
    > http://twistedmatrix.com/projects/core/documentation/howto/choosing-reactor.html#auto15
    >
    > But note the warnings about how well it is likely to work. Using a separate
    > thread might be the best solution.
    >
    > >2. How can I grab the client data in the code? Can you write two lines
    > >for that? I really appreciate that.

    >
    > I'm not sure what you mean. The data is available in the `received' attribute
    > of the protocol instance. Any code which needs to manipulate the data can get
    > that list and do whatever it likes with it.
    >
    > >3. After I change
    > >self.transport.write(''.join(self.data))
    > > to
    > >self.transport.write(''.join(data))
    > > and scan all the ports with the following code twice (run twice).
    > >First round scanning says "succefully connected". But second round
    > >scanning says "failed". I have to restart your demo code to make it
    > >work.

    >
    > I intentionally added code which shuts the server off after the first round
    > of connections is completed, since that seemed to be what your example
    > program was doing. If you don't want this, just remove the shutdown code.
    >
    > Jean-Paul
     
    zxo102, Aug 13, 2006
    #3
  4. zxo102

    Bryan Olson Guest

    Re: start a multi-sockets server (a socket/per thread) with differentports but same host

    zxo102 wrote:
    > I am doing a small project using socket server and thread in python.
    > This is first time for me to use socket and thread things.
    > Here is my case. I have 20 socket clients. Each client send a set
    > of sensor data per second to a socket server. The socket server will
    > do two things: 1. write data into a file via bsddb; 2. forward the data
    > to a GUI written in wxpython.
    > I am thinking the code should work as follow (not sure it is
    > feasible)
    > 20 threads, each thread takes care of a socket server with a
    > different port.
    > I want all socket servers start up and wait for client connection.
    > In the attached demo code, It stops at the startup of first socket
    > server somewhere in the following two lines and waits for client call:
    >
    > lstn.listen(5)
    > (clnt,ap) = lstn.accept()


    It will block there, waiting for connection.

    > Any ideas how to handle these 20 clients? Really appreciate your
    > suggestions.


    One reserved port for each client strikes me as whacked,
    as does coding a server to handle exactly 20 of them. Since
    you say this is your first socket server, maybe you just
    haven't seen the usual techniques.

    Normally, one listener socket accepts all the connections.
    Each call to accept() returns a new, independent socket for the
    connection. You can then start a thread to handle the new
    socket. Untested:


    listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    listener.bind(('', 2000))
    listener.listen(5)
    while True: # or some should_continue() thing
    sock, _ = listener.accept()
    thread.start_new_thread(service_function, (sock,))
    # Or start threads via class Threading


    To update the GUI, you could use the Queue from the Python
    library, and call wxPostEvent to tell the GUI go wake up and
    check the queue.


    --
    --Bryan
     
    Bryan Olson, Aug 13, 2006
    #4
  5. zxo102

    zxo102 Guest

    Bryan,
    Thanks for your note. Finally, I have made "one listener socket for
    all the connections" work plus Queue-communication between the threads
    in wxpython Gui and the threads for socket connections.
    Trying to make that twisted example code in this topic for "one
    listener socket-all the connections" but failed. That twisted example
    only accepts one client connection. I have printed out the Twisted help
    file (256 pages). Too much to read.

    Ouyang


    Bryan Olson 写é“:

    > zxo102 wrote:
    > > I am doing a small project using socket server and thread in python.
    > > This is first time for me to use socket and thread things.
    > > Here is my case. I have 20 socket clients. Each client send a set
    > > of sensor data per second to a socket server. The socket server will
    > > do two things: 1. write data into a file via bsddb; 2. forward the data
    > > to a GUI written in wxpython.
    > > I am thinking the code should work as follow (not sure it is
    > > feasible)
    > > 20 threads, each thread takes care of a socket server with a
    > > different port.
    > > I want all socket servers start up and wait for client connection.
    > > In the attached demo code, It stops at the startup of first socket
    > > server somewhere in the following two lines and waits for client call:
    > >
    > > lstn.listen(5)
    > > (clnt,ap) = lstn.accept()

    >
    > It will block there, waiting for connection.
    >
    > > Any ideas how to handle these 20 clients? Really appreciate your
    > > suggestions.

    >
    > One reserved port for each client strikes me as whacked,
    > as does coding a server to handle exactly 20 of them. Since
    > you say this is your first socket server, maybe you just
    > haven't seen the usual techniques.
    >
    > Normally, one listener socket accepts all the connections.
    > Each call to accept() returns a new, independent socket for the
    > connection. You can then start a thread to handle the new
    > socket. Untested:
    >
    >
    > listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    > listener.bind(('', 2000))
    > listener.listen(5)
    > while True: # or some should_continue() thing
    > sock, _ = listener.accept()
    > thread.start_new_thread(service_function, (sock,))
    > # Or start threads via class Threading
    >
    >
    > To update the GUI, you could use the Queue from the Python
    > library, and call wxPostEvent to tell the GUI go wake up and
    > check the queue.
    >
    >
    > --
    > --Bryan
     
    zxo102, Aug 15, 2006
    #5
  6. zxo102

    zxo102 Guest

    "That twisted example only accepts one client connection" if only one
    port is available.


    zxo102 写é“:

    > Bryan,
    > Thanks for your note. Finally, I have made "one listener socket for
    > all the connections" work plus Queue-communication between the threads
    > in wxpython Gui and the threads for socket connections.
    > Trying to make that twisted example code in this topic for "one
    > listener socket-all the connections" but failed. That twisted example
    > only accepts one client connection. I have printed out the Twisted help
    > file (256 pages). Too much to read.
    >
    > Ouyang
    >
    >
    > Bryan Olson 写é“:
    >
    > > zxo102 wrote:
    > > > I am doing a small project using socket server and thread in python.
    > > > This is first time for me to use socket and thread things.
    > > > Here is my case. I have 20 socket clients. Each client send a set
    > > > of sensor data per second to a socket server. The socket server will
    > > > do two things: 1. write data into a file via bsddb; 2. forward the data
    > > > to a GUI written in wxpython.
    > > > I am thinking the code should work as follow (not sure it is
    > > > feasible)
    > > > 20 threads, each thread takes care of a socket server with a
    > > > different port.
    > > > I want all socket servers start up and wait for client connection.
    > > > In the attached demo code, It stops at the startup of first socket
    > > > server somewhere in the following two lines and waits for client call:
    > > >
    > > > lstn.listen(5)
    > > > (clnt,ap) = lstn.accept()

    > >
    > > It will block there, waiting for connection.
    > >
    > > > Any ideas how to handle these 20 clients? Really appreciate your
    > > > suggestions.

    > >
    > > One reserved port for each client strikes me as whacked,
    > > as does coding a server to handle exactly 20 of them. Since
    > > you say this is your first socket server, maybe you just
    > > haven't seen the usual techniques.
    > >
    > > Normally, one listener socket accepts all the connections.
    > > Each call to accept() returns a new, independent socket for the
    > > connection. You can then start a thread to handle the new
    > > socket. Untested:
    > >
    > >
    > > listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    > > listener.bind(('', 2000))
    > > listener.listen(5)
    > > while True: # or some should_continue() thing
    > > sock, _ = listener.accept()
    > > thread.start_new_thread(service_function, (sock,))
    > > # Or start threads via class Threading
    > >
    > >
    > > To update the GUI, you could use the Queue from the Python
    > > library, and call wxPostEvent to tell the GUI go wake up and
    > > check the queue.
    > >
    > >
    > > --
    > > --Bryan
     
    zxo102, Aug 15, 2006
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Alexander N. Spitzer
    Replies:
    21
    Views:
    1,843
    Randolf Richardson
    Aug 16, 2004
  2. cos
    Replies:
    0
    Views:
    500
  3. Tim Black
    Replies:
    1
    Views:
    1,374
    Alan Kennedy
    Aug 3, 2004
  4. liu yang
    Replies:
    4
    Views:
    2,043
    Antoninus Twink
    Jul 28, 2008
  5. mreister
    Replies:
    1
    Views:
    3,386
    mreister
    May 25, 2010
Loading...

Share This Page