numarray and SMP

[Christopher T King]

In a quest to speed up numarray computations, I tried writing a 'threaded
array' class for use on SMP systems that would distribute its workload
across the processors. I hit a snag when I found out that since the Python
interpreter is not reentrant, this effectively disables parallel
processing in Python. I've come up with two solutions to this problem,
both involving numarray's C functions that perform the actual vector
operations:

1) Surround the C vector operations with Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS, thus allowing the vector operations (which don't
access Python structures) to run in parallel with the interpreter.
Python glue code would take care of threading and locking.

2) Move the parallelization into the C vector functions themselves. This
would likely get poorer performance (a chain of vector operations
couldn't be combined into one threaded operation).

I'd much rather do #1, but will playing around with the interpreter state
like that cause any problems?

 

[Fernando Perez]

In a quest to speed up numarray computations, I tried writing a 'threaded
array' class for use on SMP systems that would distribute its workload
across the processors. I hit a snag when I found out that since the Python
interpreter is not reentrant, this effectively disables parallel
processing in Python. I've come up with two solutions to this problem,
both involving numarray's C functions that perform the actual vector
operations:

[...]

I suggest you repost this to the numpy list as well. Not only are the
developers there, but this issue interests many of us, so you'd get an eager
audience and more discussion. Not that I don't think c.l.py is a good forum,
quite the contrary: many threading experts live here and not in numpy. I
meant posting to both places, since the combined expertise of 'generic
threading' experts from c.l.py and numeric/numarray users/developers will
likely be a good thing.

Best,

f

 

[Christopher T King]

I suggest you repost this to the numpy list as well. Not only are the
developers there, but this issue interests many of us, so you'd get an eager
audience and more discussion. Not that I don't think c.l.py is a good forum,
quite the contrary: many threading experts live here and not in numpy. I
meant posting to both places, since the combined expertise of 'generic
threading' experts from c.l.py and numeric/numarray users/developers will
likely be a good thing.

Thanks, and done.

 

[Fernando Perez]

I suggest you repost this to the numpy list as well. Not only are the

[...]

Thanks, and done.

Just saw it. I don't really have an answer for you, but I'm curious about what
others on that list may say. We'll see.

Cheers,

f

 

[Aahz]

1) Surround the C vector operations with Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS, thus allowing the vector operations (which don't
access Python structures) to run in parallel with the interpreter.
Python glue code would take care of threading and locking.

2) Move the parallelization into the C vector functions themselves. This
would likely get poorer performance (a chain of vector operations
couldn't be combined into one threaded operation).

I'd much rather do #1, but will playing around with the interpreter state
like that cause any problems?

Not at all -- that's precisely what they're there for. All you have to
do is make sure you're not calling back into Python before doing
Py_END_ALLOW_THREADS. Traditionally, Python has only done this for I/O
operations; I'm very happy to see someone trying to take a whack at the
computational side. (Although I ended up mostly dropping the ball, that
was the original impetus for the Decimal project, to encourage more
computational threading.)