How to initialize each multithreading Pool worker with an individualvalue?

Discussion in 'Python' started by Valery Khamenya, Nov 30, 2010.

  1. Hi,

    multithreading.pool Pool has a promissing initializer argument in its
    constructor.
    However it doesn't look possible to use it to initialize each Pool's
    worker with some individual value (I'd wish to be wrong here)

    So, how to initialize each multithreading Pool worker with the
    individual values?

    The typical use case might be a connection pool, say, of 3 workers,
    where each of 3 workers has its own TCP/IP port.

    from multiprocessing.pool import Pool

    def port_initializer(_port):
    global port
    port = _port

    def use_connection(some_packet):
    global _port
    print "sending data over port # %s" % port

    if __name__ == "__main__":
    ports=((4001,4002, 4003), )
    p = Pool(3, port_initializer, ports) # oops... :)
    some_data_to_send = range(20)
    p.map(use_connection, some_data_to_send)


    best regards
    --
    Valery A.Khamenya
     
    Valery Khamenya, Nov 30, 2010
    #1
    1. Advertising

  2. Valery Khamenya

    Aahz Guest

    In article <>,
    Valery Khamenya <> wrote:
    >
    >However it doesn't look possible to use it to initialize each Pool's
    >worker with some individual value (I'd wish to be wrong here)
    >
    >So, how to initialize each multithreading Pool worker with the
    >individual values?
    >
    >The typical use case might be a connection pool, say, of 3 workers,
    >where each of 3 workers has its own TCP/IP port.
    >
    >from multiprocessing.pool import Pool
    >
    >def port_initializer(_port):
    > global port
    > port = _port
    >
    >def use_connection(some_packet):
    > global _port
    > print "sending data over port # %s" % port
    >
    >if __name__ == "__main__":
    > ports=((4001,4002, 4003), )
    > p = Pool(3, port_initializer, ports) # oops... :)


    You probably can't use initargs here. Your port_initializer needs to be
    some kind of class instance that works with multiprocessing and emits one
    port number when its __call__() method gets invoked. (There may be other
    ways to accomplish the same effect, but that's what springs to mind.)
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Think of it as evolution in action." --Tony Rand
     
    Aahz, Dec 30, 2010
    #2
    1. Advertising

  3. Re: How to initialize each multithreading Pool worker with anindividual value?

    On Thu, 2010-12-30 at 08:01 -0800, Aahz wrote:
    > In article <>,
    > Valery Khamenya <> wrote:
    > >However it doesn't look possible to use it to initialize each Pool's
    > >worker with some individual value (I'd wish to be wrong here)
    > >So, how to initialize each multithreading Pool worker with the
    > >individual values?
    > >The typical use case might be a connection pool, say, of 3 workers,
    > >where each of 3 workers has its own TCP/IP port.
    > >from multiprocessing.pool import Pool
    > >def port_initializer(_port):
    > > global port
    > > port = _port
    > >def use_connection(some_packet):
    > > global _port
    > > print "sending data over port # %s" % port
    > >if __name__ == "__main__":
    > > ports=((4001,4002, 4003), )
    > > p = Pool(3, port_initializer, ports) # oops... :)

    > You probably can't use initargs here. Your port_initializer needs to be
    > some kind of class instance that works with multiprocessing and emits one
    > port number when its __call__() method gets invoked. (There may be other
    > ways to accomplish the same effect, but that's what springs to mind.)


    Maybe this is obvious; but it is possible to create a pool of workers
    all listening on the same socket. An idle worker will pick-up the
    connection. Just open the socket in the initial process and then fork
    your workers - they will inherit the file handle and can accept() on it.
     
    Adam Tauno Williams, Dec 30, 2010
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. alex
    Replies:
    1
    Views:
    691
    Lau Lei Cheong
    Feb 4, 2005
  2. Gordon Smith
    Replies:
    1
    Views:
    3,600
    Curt_C [MVP]
    Mar 1, 2005
  3. Jeremy S.
    Replies:
    2
    Views:
    4,191
    Brock Allen
    Mar 28, 2005
  4. J-T
    Replies:
    5
    Views:
    9,086
    Karl Seguin
    Dec 5, 2005
  5. Jordan Tan
    Replies:
    0
    Views:
    530
    Jordan Tan
    May 10, 2006
Loading...

Share This Page