Threads, GIL and re.match() performance

[Mirko Dziadzka]

Hi all

I understand that the C implementation of Python use a global interpreter
lock to avoid problems, so doing CPU bound tasks in multiple threads
will not result in better performance on multi-CPU systems.

However, I assumed that calls to (thread safe) C Library functions
release the global interpreter lock.

Today I checked the performance of some slow re.match() calls and found,
that the do not run in parallel on a multi-CPU system.

1) Is there a reason for this?
2) Is the regex library not thread-safe?
3) Is it possible, to release the GIL in re.match() to
get more performance?

I'm using Python 2.5

Thanks for your help

Mirko

 

[Aahz]

I understand that the C implementation of Python use a global interpreter
lock to avoid problems, so doing CPU bound tasks in multiple threads
will not result in better performance on multi-CPU systems.

However, I assumed that calls to (thread safe) C Library functions
release the global interpreter lock.

Generally speaking that only applies to I/O calls.

Today I checked the performance of some slow re.match() calls and found,
that the do not run in parallel on a multi-CPU system.

1) Is there a reason for this?
2) Is the regex library not thread-safe?
3) Is it possible, to release the GIL in re.match() to
get more performance?

Theoretically possible, but the usual rule applies: patches welcome

 

[Benjamin]

1) Is there a reason for this?

I think it is because the Python re library uses the Python C-API
which is not threadsafe.

 

[Matthieu Brucher]

Hi,

The C-API uses references counts as well, so it is not threadsafe.

Matthieu

But Python C-API[1] it's the main base for extent python with C/c++, and
this is not not threadsafe.? I dont understand

[1] http://docs.python.org/api/api.html

I think it is because the Python re library uses the Python C-API
which is not threadsafe.

 

[Jeff]

However, I assumed that calls to (thread safe) C Library functions

release the global interpreter lock.

This is mainly applicable to external C libraries. The interface to
them may not be thread-safe; anything that uses the Python API to
create/manage Python objects will require use of the GIL. So the
actual regex search may release the GIL, but the storing of results
(and possibly intermediate results) would not.

 

[Sebastian \lunar\ Wiesner]

(and possibly intermediate results)

These could be stored purely in C space, without refcounting needed.