Python with MPI enable C-module

[dfj225]

Hello All,

A software project that I am working on currently has a C++ library
that runs on a Beowulf cluster for parallel computation. The library
code uses MPI for its implementation. We are currently developing
python wrappers for the library. Our current design uses one Python
process as a front end with processes created from the library code on
the back end (it is processes of this type that run on all the nodes of
the cluster).

What I'm considering is delving into the possibility of having Python
processes on all of the nodes of the cluster. These processes would
wrap our C++ library. What I'm wondering is:

1) Would setting up an environment like this require modifying the
Python interpreter or the C++ module that is being wrapped? What I'm
hoping is that the C++ module can go on happily doing MPI operations
despite the fact that it is actually being called by the Python
interpreter.

2) Would it be possible to spawn these Python processes using mpiexec
(or something similar), or would I need to use some of the MPI-2
features to dynamically set up the MPI environment?

3) Has anyone accomplished something like this already? I know there
are extensions and modules that add MPI functionality to Python, but
I'm hoping they could be avoided, since the Python code itself should
never really have to be aware of MPI, only the C++ module that has
already been written.

Let me apologize up front if this has already been discussed before.
I've done a few searches in this group and it seems that most of the
discussion on this front took place many years ago, so the relevance is
quite low.

Thank you in advance for your attention.

~doug

 

[Mitko Haralanov]

1) Would setting up an environment like this require modifying the
Python interpreter or the C++ module that is being wrapped? What I'm
hoping is that the C++ module can go on happily doing MPI operations
despite the fact that it is actually being called by the Python
interpreter.

If I am understanding you correctly, you have a Python module which
calls into the C++ library to do the actual MPI operations. If that is
the case, then there should be any problems just calling the Python
wrappers. The MPI library should be unaffected. It would be the same as
having your node programs call into the C++ library, now you just have
one more layer of abstraction.

2) Would it be possible to spawn these Python processes using mpiexec
(or something similar), or would I need to use some of the MPI-2
features to dynamically set up the MPI environment?

If you are writing a Python wrapper to the MPI library, then the
wrapper will be just like a module and not a self sustained program. If
that is the case, mpiexec/mpirun/mpd would not be able to do anything
since they expect to start a program that has main()/__main__.

3) Has anyone accomplished something like this already? I know there
are extensions and modules that add MPI functionality to Python, but
I'm hoping they could be avoided, since the Python code itself should
never really have to be aware of MPI, only the C++ module that has
already been written.

Haven't done it but from what I know about MPI and Python, it is
possible.

 

[dfj225]

Mitko,

Thank you for your response.

You understand the problem correctly. The Python module uses MPI calls
in its implementation. The idea of the project is to create an
abstraction for a distributed memory computer. In doing so, the user is
mostly isolated from a need to understand and use MPI calls.

After my post, I was able to find some more information on the subject.
I think you are right in that mpiexec (or equivalents) would not work
without some extra work. However it seems that using an additional
module (such as <a
href="http://datamining.anu.edu.au/~ole/pypar/">Pypar</a>) would allow
me to use Python with mpiexec.

Thank you,
~doug