How to initialize each multithreading Pool worker with an individualvalue?

Discussion in 'Python' started by Valery Khamenya, Nov 30, 2010.

  1. Hi,

    multithreading.pool Pool has a promissing initializer argument in its
    constructor.
    However it doesn't look possible to use it to initialize each Pool's
    worker with some individual value (I'd wish to be wrong here)

    So, how to initialize each multithreading Pool worker with the
    individual values?

    The typical use case might be a connection pool, say, of 3 workers,
    where each of 3 workers has its own TCP/IP port.

    from multiprocessing.pool import Pool

    def port_initializer(_port):
    global port
    port = _port

    def use_connection(some_packet):
    global _port
    print "sending data over port # %s" % port

    if __name__ == "__main__":
    ports=((4001,4002, 4003), )
    p = Pool(3, port_initializer, ports) # oops... :)
    some_data_to_send = range(20)
    p.map(use_connection, some_data_to_send)


    best regards
     
    Valery Khamenya, Nov 30, 2010
    #1
    1. Advertisements

  2. Valery Khamenya

    Aahz Guest

    You probably can't use initargs here. Your port_initializer needs to be
    some kind of class instance that works with multiprocessing and emits one
    port number when its __call__() method gets invoked. (There may be other
    ways to accomplish the same effect, but that's what springs to mind.)
     
    Aahz, Dec 30, 2010
    #2
    1. Advertisements

  3. Maybe this is obvious; but it is possible to create a pool of workers
    all listening on the same socket. An idle worker will pick-up the
    connection. Just open the socket in the initial process and then fork
    your workers - they will inherit the file handle and can accept() on it.
     
    Adam Tauno Williams, Dec 30, 2010
    #3
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.