Re: Memory consumption of multiprocessing.Pool

Discussion in 'Python' started by Aahz, Dec 31, 2009.

  1. Aahz

    Aahz Guest

    In article <>,
    Wolodja Wentland <-heidelberg.de> wrote:
    >
    >I have a problem with the memory consumption of multiprocessing.Pool()'s
    >worker processes. I have a parent process that has to handle big data
    >structures and would like to use a pool of processes for computations.
    >
    >The problem is, that all worker processes have the same memory
    >requirement as the parent one, although they do *not* use any the parent
    >processes data structures.


    What I would do is put your current parent process into its own
    subprocess and have the main process handle communication between the
    memory-heavy subprocess and the worker processes.
    --
    Aahz () <*> http://www.pythoncraft.com/

    Weinberg's Second Law: If builders built buildings the way programmers wrote
    programs, then the first woodpecker that came along would destroy civilization.
    Aahz, Dec 31, 2009
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    3
    Views:
    2,040
    Aaron Brady
    May 1, 2009
  2. Replies:
    1
    Views:
    327
    Jesse Noller
    Apr 30, 2009
  3. masher
    Replies:
    4
    Views:
    854
    ryles
    Jul 3, 2009
  4. Rick Lawson
    Replies:
    8
    Views:
    786
    Graham Dumpleton
    Jul 17, 2009
  5. Allen Fowler
    Replies:
    1
    Views:
    1,194
Loading...

Share This Page