Advice regarding multiprocessing module

Discussion in 'Python' started by Abhinav M Kulkarni, Mar 11, 2013.

  1. Dear all,

    I need some advice regarding use of the multiprocessing module.
    Following is the scenario:

    * I am running gradient descent to estimate parameters of a pairwise
    grid CRF (or a grid based graphical model). There are 106 data
    points. Each data point can be analyzed in parallel.
    * To calculate gradient for each data point, I need to perform
    approximate inference since this is a loopy model. I am using Gibbs
    sampling.
    * My grid is 9x9 so there are 81 variables that I am sampling in one
    sweep of Gibbs sampling. I perform 1000 iterations of Gibbs sampling.
    * My laptop has quad-core Intel i5 processor, so I thought using
    multiprocessing module I can parallelize my code (basically
    calculate gradient in parallel on multiple cores simultaneously).
    * I did not use the multi-threading library because of GIL issues, GIL
    does not allow multiple threads to run at a time.
    * As a result I end up creating a process for each data point (instead
    of a thread that I would ideally like to do, so as to avoid process
    creation overhead).
    * I am using basic NumPy array functionalities.

    Previously I was running this code in MATLAB. It runs quite faster, one
    iteration of gradient descent takes around 14 sec in MATLAB using parfor
    loop (parallel loop - data points is analyzed within parallel loop).
    However same program takes almost 215 sec in Python.

    I am quite amazed at the slowness of multiprocessing module. Is this
    because of process creation overhead for each data point?

    Please keep my email in the replies as I am not a member of this mailing
    list.

    Thanks,
    Abhinav
     
    Abhinav M Kulkarni, Mar 11, 2013
    #1
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. sturlamolden

    multiprocessing module (PEP 371)

    sturlamolden, Jun 4, 2008, in forum: Python
    Replies:
    6
    Views:
    599
    John Nagle
    Jun 8, 2008
  2. Replies:
    9
    Views:
    1,235
  3. Graham Dumpleton
    Replies:
    4
    Views:
    821
    Graham Dumpleton
    Feb 22, 2009
  4. dmitrey
    Replies:
    1
    Views:
    421
    Terry Reedy
    Mar 14, 2009
  5. Replies:
    1
    Views:
    901
    Tennessee
    Oct 14, 2009
  6. Sebastien Binet
    Replies:
    4
    Views:
    1,259
    Sebastien Binet
    Dec 16, 2009
  7. Matt Newville
    Replies:
    0
    Views:
    148
    Matt Newville
    Mar 24, 2014
  8. Chris Angelico
    Replies:
    3
    Views:
    229
    Chris Angelico
    Mar 25, 2014
Loading...