How to initialize each multithreading Pool worker with an individualvalue?

[Valery Khamenya]

Hi,

multithreading.pool Pool has a promissing initializer argument in its
constructor.
However it doesn't look possible to use it to initialize each Pool's
worker with some individual value (I'd wish to be wrong here)

So, how to initialize each multithreading Pool worker with the
individual values?

The typical use case might be a connection pool, say, of 3 workers,
where each of 3 workers has its own TCP/IP port.

from multiprocessing.pool import Pool

def port_initializer(_port):
global port
port = _port

def use_connection(some_packet):
global _port
print "sending data over port # %s" % port

if __name__ == "__main__":
ports=((4001,4002, 4003), )
p = Pool(3, port_initializer, ports) # oops...
some_data_to_send = range(20)
p.map(use_connection, some_data_to_send)


best regards

 

[Aahz]

However it doesn't look possible to use it to initialize each Pool's
worker with some individual value (I'd wish to be wrong here)

So, how to initialize each multithreading Pool worker with the
individual values?

The typical use case might be a connection pool, say, of 3 workers,
where each of 3 workers has its own TCP/IP port.

from multiprocessing.pool import Pool

def port_initializer(_port):
global port
port = _port

def use_connection(some_packet):
global _port
print "sending data over port # %s" % port

if __name__ == "__main__":
ports=((4001,4002, 4003), )
p = Pool(3, port_initializer, ports) # oops...

You probably can't use initargs here. Your port_initializer needs to be
some kind of class instance that works with multiprocessing and emits one
port number when its __call__() method gets invoked. (There may be other
ways to accomplish the same effect, but that's what springs to mind.)

 

[Adam Tauno Williams]

You probably can't use initargs here. Your port_initializer needs to be
some kind of class instance that works with multiprocessing and emits one
port number when its __call__() method gets invoked. (There may be other
ways to accomplish the same effect, but that's what springs to mind.)

Maybe this is obvious; but it is possible to create a pool of workers
all listening on the same socket. An idle worker will pick-up the
connection. Just open the socket in the initial process and then fork
your workers - they will inherit the file handle and can accept() on it.