numarray and SMP

Discussion in 'Python' started by Christopher T King, Jul 1, 2004.

  1. In a quest to speed up numarray computations, I tried writing a 'threaded
    array' class for use on SMP systems that would distribute its workload
    across the processors. I hit a snag when I found out that since the Python
    interpreter is not reentrant, this effectively disables parallel
    processing in Python. I've come up with two solutions to this problem,
    both involving numarray's C functions that perform the actual vector
    operations:

    1) Surround the C vector operations with Py_BEGIN_ALLOW_THREADS and
    Py_END_ALLOW_THREADS, thus allowing the vector operations (which don't
    access Python structures) to run in parallel with the interpreter.
    Python glue code would take care of threading and locking.

    2) Move the parallelization into the C vector functions themselves. This
    would likely get poorer performance (a chain of vector operations
    couldn't be combined into one threaded operation).

    I'd much rather do #1, but will playing around with the interpreter state
    like that cause any problems?
    Christopher T King, Jul 1, 2004
    #1
    1. Advertising

  2. Christopher T King wrote:

    > In a quest to speed up numarray computations, I tried writing a 'threaded
    > array' class for use on SMP systems that would distribute its workload
    > across the processors. I hit a snag when I found out that since the Python
    > interpreter is not reentrant, this effectively disables parallel
    > processing in Python. I've come up with two solutions to this problem,
    > both involving numarray's C functions that perform the actual vector
    > operations:


    [...]

    I suggest you repost this to the numpy list as well. Not only are the
    developers there, but this issue interests many of us, so you'd get an eager
    audience and more discussion. Not that I don't think c.l.py is a good forum,
    quite the contrary: many threading experts live here and not in numpy. I
    meant posting to both places, since the combined expertise of 'generic
    threading' experts from c.l.py and numeric/numarray users/developers will
    likely be a good thing.

    Best,

    f
    Fernando Perez, Jul 1, 2004
    #2
    1. Advertising

  3. On Thu, 1 Jul 2004, Fernando Perez wrote:

    > I suggest you repost this to the numpy list as well. Not only are the
    > developers there, but this issue interests many of us, so you'd get an eager
    > audience and more discussion. Not that I don't think c.l.py is a good forum,
    > quite the contrary: many threading experts live here and not in numpy. I
    > meant posting to both places, since the combined expertise of 'generic
    > threading' experts from c.l.py and numeric/numarray users/developers will
    > likely be a good thing.


    Thanks, and done.
    Christopher T King, Jul 1, 2004
    #3
  4. Christopher T King wrote:

    > On Thu, 1 Jul 2004, Fernando Perez wrote:
    >
    >> I suggest you repost this to the numpy list as well. Not only are the

    [...]
    >
    > Thanks, and done.


    Just saw it. I don't really have an answer for you, but I'm curious about what
    others on that list may say. We'll see.

    Cheers,

    f
    Fernando Perez, Jul 1, 2004
    #4
  5. Christopher T King

    Aahz Guest

    In article <>,
    Christopher T King <> wrote:
    >
    >1) Surround the C vector operations with Py_BEGIN_ALLOW_THREADS and
    > Py_END_ALLOW_THREADS, thus allowing the vector operations (which don't
    > access Python structures) to run in parallel with the interpreter.
    > Python glue code would take care of threading and locking.
    >
    >2) Move the parallelization into the C vector functions themselves. This
    > would likely get poorer performance (a chain of vector operations
    > couldn't be combined into one threaded operation).
    >
    >I'd much rather do #1, but will playing around with the interpreter state
    >like that cause any problems?


    Not at all -- that's precisely what they're there for. All you have to
    do is make sure you're not calling back into Python before doing
    Py_END_ALLOW_THREADS. Traditionally, Python has only done this for I/O
    operations; I'm very happy to see someone trying to take a whack at the
    computational side. (Although I ended up mostly dropping the ball, that
    was the original impetus for the Decimal project, to encourage more
    computational threading.)
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Typing is cheap. Thinking is expensive." --Roy Smith, c.l.py
    Aahz, Jul 6, 2004
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ohaya
    Replies:
    0
    Views:
    326
    Ohaya
    Feb 22, 2004
  2. Replies:
    1
    Views:
    375
  3. Replies:
    1
    Views:
    402
  4. Gardner Pomper
    Replies:
    0
    Views:
    491
    Gardner Pomper
    Nov 12, 2003
  5. catsup

    SMP, GIL and Threads

    catsup, Dec 16, 2005, in forum: Python
    Replies:
    11
    Views:
    521
Loading...

Share This Page