Advice regarding multiprocessing module

Discussion in 'Python' started by Abhinav M Kulkarni, Mar 11, 2013.

  1. Dear all,

    I need some advice regarding use of the multiprocessing module.
    Following is the scenario:

    * I am running gradient descent to estimate parameters of a pairwise
    grid CRF (or a grid based graphical model). There are 106 data
    points. Each data point can be analyzed in parallel.
    * To calculate gradient for each data point, I need to perform
    approximate inference since this is a loopy model. I am using Gibbs
    sampling.
    * My grid is 9x9 so there are 81 variables that I am sampling in one
    sweep of Gibbs sampling. I perform 1000 iterations of Gibbs sampling.
    * My laptop has quad-core Intel i5 processor, so I thought using
    multiprocessing module I can parallelize my code (basically
    calculate gradient in parallel on multiple cores simultaneously).
    * I did not use the multi-threading library because of GIL issues, GIL
    does not allow multiple threads to run at a time.
    * As a result I end up creating a process for each data point (instead
    of a thread that I would ideally like to do, so as to avoid process
    creation overhead).
    * I am using basic NumPy array functionalities.

    Previously I was running this code in MATLAB. It runs quite faster, one
    iteration of gradient descent takes around 14 sec in MATLAB using parfor
    loop (parallel loop - data points is analyzed within parallel loop).
    However same program takes almost 215 sec in Python.

    I am quite amazed at the slowness of multiprocessing module. Is this
    because of process creation overhead for each data point?

    Please keep my email in the replies as I am not a member of this mailing
    list.

    Thanks,
    Abhinav
     
    Abhinav M Kulkarni, Mar 11, 2013
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Dave Angel
    Replies:
    0
    Views:
    107
    Dave Angel
    Mar 11, 2013
  2. Abhinav M Kulkarni

    Re: Advice regarding multiprocessing module

    Abhinav M Kulkarni, Mar 11, 2013, in forum: Python
    Replies:
    0
    Views:
    122
    Abhinav M Kulkarni
    Mar 11, 2013
  3. Oscar Benjamin

    Re: Advice regarding multiprocessing module

    Oscar Benjamin, Mar 11, 2013, in forum: Python
    Replies:
    0
    Views:
    151
    Oscar Benjamin
    Mar 11, 2013
  4. Matt Newville
    Replies:
    0
    Views:
    78
    Matt Newville
    Mar 24, 2014
  5. Chris Angelico
    Replies:
    3
    Views:
    85
    Chris Angelico
    Mar 25, 2014
Loading...

Share This Page