parallel NumPy: PyMPI, myMPI or something else?

[smithken04]

Hi!

I am trying to find my way to a running parallel NumPy installation.
Can you help me getting started wihtout losing my way several times?

PyMPI and myMPI are apperently not too actively developed. Is this
true? Are they mature enough so that they don't need any more updates?
Have they been replaced by more recent modules?

Then, if I have NumPy running with PyMPI or whatever, is NumPy already
optimized for a MPI environment so that for example matrix
multiplication, iterative solvers etc. are automatically distributed?
Or do I have to split each problem myself to make use of the
parallelism?

Thank you
Ken

 

[Robert Kern]

Hi!

I am trying to find my way to a running parallel NumPy installation.
Can you help me getting started wihtout losing my way several times?

PyMPI and myMPI are apperently not too actively developed. Is this
true? Are they mature enough so that they don't need any more updates?
Have they been replaced by more recent modules?

The people who I know who use MPI and Python have been using mpi4py. I haven't
used it myself, though.

http://mpi4py.scipy.org/

Then, if I have NumPy running with PyMPI or whatever, is NumPy already
optimized for a MPI environment so that for example matrix
multiplication, iterative solvers etc. are automatically distributed?
No.

Or do I have to split each problem myself to make use of the
parallelism?

Pretty much.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[smithken04]

Robert,
the answer could not have been shorter, but I got exactly the
information I was looking for
Thank you so much!

Just a single followup question:

Pretty much.

Does "pretty much" imply that actually *some* parts of NumPy do make
use of the parallel environment without any MPI handling from the
user? I didn't find an indication for this during my short skimming
over the NumPy documentation.

Ken

 

[Robert Kern]

Robert,
the answer could not have been shorter, but I got exactly the
information I was looking for
Thank you so much!

Just a single followup question:
Does "pretty much" imply that actually *some* parts of NumPy do make
use of the parallel environment without any MPI handling from the
user? I didn't find an indication for this during my short skimming
over the NumPy documentation.

Sorry, that's really just American teenager idiom that's followed me into
adulthood; it means, roughly, "Yes, this is entirely, unequivocally true," much
like "Literally!" usually means "Figuratively!"

No part of numpy proper takes advantage of parallelism. Eric Jones has been
toying with using threads to split up some operations on large arrays between
cores on an SMP system, but it is not ready to merge into the trunk. The
performance numbers haven't been that good. The cost of thread locks is high.
But then, only one approach has been tried; if you are interested in trying
others, we'd be more than happy to help you along on the numpy-discussion
mailing list.

http://svn.scipy.org/svn/numpy/branches/multicore/
http://www.scipy.org/Mailing_Lists

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco