Mixing Python and C threads

[Guest]

I am currently developing a software called OpenFilters (available at
www.polymtl.ca/larfis). Most of the software is written in Python, but the
classes that do the hard work are written in C++ modules. Since, nowadays, most
new computers have multiple cores and that I want to speed up the software, I am
currently trying to add some multithreading to the C++ modules using openmp.

If I call the methods in the C++ modules from the Python main thread, I don't
have any problem. However, they are usually called from another Python thread
(using the threading module) to keep the GUI responsive and, then, Python
crashes.

The C++ multithreaded part does not call any Python function. It crashes whether
it is surrounded by Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS or not. I am
using Python 2.5 on Windows XP. The modules are compiled with gcc 4.2 (MinGW
unofficial release available at
http://www.esnips.com/doc/9dba8ac7-70c7-4f98-a0fa-8ea315267073/gcc-4.2.0mingw-
release-patched-SSE).

Has anybody had a similar problem? Do you know how to solve it?

Thank you for your help,

Stéphane Larouche

 

[Aahz]

If I call the methods in the C++ modules from the Python main thread, I
don't have any problem. However, they are usually called from another
Python thread (using the threading module) to keep the GUI responsive
and, then, Python crashes.

The question is whether your C++ code is designed to be called from
multiple threads. That is, if you call your C++ code from a *single*
non-main thread, does it work?

 

[Guest]

The question is whether your C++ code is designed to be called from
multiple threads. That is, if you call your C++ code from a *single*
non-main thread, does it work?

Thank you for the quick answer.

It does work if I call it from a single non-main thread. But everything must be
called from that thread, including the import of the module.

I still do not understand what is the problem. The C++ code is not thread safe,
but I call it from 1 Python thread at a time. Anyway, I have always called it
from multiple Python threads without any problem. Having put a few printf in the
module, I can say that it crashes exactly when it tries to create C threads.

Am I missing something?

 

[Aahz]

It does work if I call it from a single non-main thread. But everything
must be called from that thread, including the import of the module.

I still do not understand what is the problem. The C++ code is not
thread safe, but I call it from 1 Python thread at a time. Anyway,
I have always called it from multiple Python threads without any
problem. Having put a few printf in the module, I can say that it
crashes exactly when it tries to create C threads.

Can you reproduce your problem with stub code that only creates threads?
If yes, that indicates that you're messing with a thread structure
somewhere. Note that because you're using gcc, it's possible there's a
thread bug on Windows with your tool chain. You might want to check for
bug reports.

 

[Guest]

Can you reproduce your problem with stub code that only creates threads?
If yes, that indicates that you're messing with a thread structure
somewhere. Note that because you're using gcc, it's possible there's a
thread bug on Windows with your tool chain. You might want to check for
bug reports.

Here's the simplest piece of code I could think of to reproduce the problem:

---------- primes.cpp ----------
#include <python.h>
#include <omp.h>

using namespace std;

int is_prime(const long n)
{
long i;

for (i = 2; i < n; i++)
if (n % i == 0) return 0;

return 1;
}

long nb_primes(const long n)
{
long i, nb = 0;

#pragma omp parallel for private(i) reduction(+ : nb) // It crashes here.
for (i = 2; i <= n; i++)
if (is_prime(i)) nb += 1;

return nb;
}

static PyObject * nb_primes_wrapper(PyObject *self, PyObject *args)
{
long n, nb;

if (!PyArg_ParseTuple(args, "l", &n)) return NULL;

nb = nb_primes(n);

return Py_BuildValue("l", nb);
}

static PyMethodDef primes_methods[] =
{
{"nb_primes", nb_primes_wrapper, METH_VARARGS},
{NULL}
};

PyMODINIT_FUNC initprimes()
{
PyObject *module;

module = Py_InitModule("primes", primes_methods);
}
---------- end of primes.cpp ----------

I compile it using:

gcc -c -fopenmp -IC:/Python25/include primes.cpp
gcc -LC:/Python25/libs -LC:/mingw/lib -mdll -o primes.pyd primes.o -lpython25
-lgomp -lpthreadGC2

In Python:

import primes
primes.nb_primes(1000)

works fine, but

import primes
import threading
t = threading.Thread(target = primes.nb_primes, args = (1000,))
t.start()

crashes.

Would anybody be so kind and try to compile it with Microsoft or Intel compiler
and tell me if it also crashes (I tried Visual C++ 2005 Express Edition, but it
does not implement OpenMP). If the bug is with gcc, I will submit the problem to
the appropriate people.

Thank you,

Stéphane Larouche

 

[Aahz]

Here's the simplest piece of code I could think of to reproduce the problem:

Good work. This is about as far as my expertise can take you; if nobody
responds, you might try the new C/API mailing list (capi-sig).

 

[Guest]

Good work. This is about as far as my expertise can take you; if nobody
responds, you might try the new C/API mailing list (capi-sig).

If anybody followed this thread, you'll be pleased to know that the mingw gcc
4.2.1 "technology preview" release available from the official mingw website and
Python threads interact perfectly well together. You also have to pass the
-mthread option to the linker.

Sincerely,

Stéphane