GIL, threads and scheduling - performance cost

[adsheehan]

Hi all,

Wondering if a GIL lock/unlock causes a re-schedule/contect swap when
embedding Python in a multi-threaded C/C++ app on Unix ?

If so, do I have any control or influence on this re-scheduling ?

The app suffers from serious performance degradation (compared to pure
c/C++) and high context switches that I suspect the GIL unlocking may
be aggravating ?

Thanks for any help.

Alan

 

[Pierre Barbier de Reuille]

(e-mail address removed) a écrit :

Hi all,

Wondering if a GIL lock/unlock causes a re-schedule/contect swap when
embedding Python in a multi-threaded C/C++ app on Unix ?

If so, do I have any control or influence on this re-scheduling ?

The app suffers from serious performance degradation (compared to pure
c/C++) and high context switches that I suspect the GIL unlocking may
be aggravating ?

Well, where do you observe this degradation ? When replacing part of the
C++ code by Python's code ? Or on C++ code running parallel to your
Python code ?

Because in the first case, well, this is just something natural ! Python
runtime overhead is much greater than C++ because of its dynamic nature
(it has to resolve all the symbols at runtime ...). And given the policy
for locking/releasing the GIL, I doubt it has serious performance issues
compared to the Python interpreter itself. If current performance is an
issue, consider implementing more in C/C++ ! This will be mainly true if
you currently have some heavy looping in Python. Python is very neat to
put together processor-intensive functions written in other languages,
but not to implement them. (as an exemple looh at that:
http://www.python.org/doc/essays/list2str.html )

Thanks for any help.

Alan

Pierre

 

[adsheehan]

Merci Pierre,

Yes I agree and plan to move more to C/C++ and releasing the GIL when
entering C/C++.

I also need to understand my original question re GIL and rescheduling.
I fear that lock/unlock too often is also causing delays due to context
switching.

BTW do you have any hints/comments on SWIG/BOOST etc to glue PY and
C/C++ ?


Alan

 

[Pierre Barbier de Reuille]

(e-mail address removed) a écrit :

Merci Pierre,

Yes I agree and plan to move more to C/C++ and releasing the GIL when
entering C/C++.

I also need to understand my original question re GIL and rescheduling.
I fear that lock/unlock too often is also causing delays due to context
switching.

Well, concerning GIL and rescheduling, releasing a lock is very likely
to induce rescheduling and context switch, but it won't if the system
considers it worthless. But this is true that lock/unlock has a cost and
so, doing it too much will slow down your app.

BTW do you have any hints/comments on SWIG/BOOST etc to glue PY and
C/C++ ?

Well, for C I would recommand SWIG. For C++, I personnaly use
Boost.Python. However, you must know that SWIG did not support C++ well
when I had to choose between the two, so there was no question for me.
Then, somehow it depends on how you feel with the tools: SWIG depends on
a specific language and has its own tools while Boost is entirely
written in C++ and uses heavily templates. For performances, Boost
generate better code that SWIG, mainly because SWIG relies on Python
code that encapsulate the actual C functions while Boost does everything
in C++. Also, at least with G++, compiling Boost extension is very time-
and memory-consuming due to its heavy use of templates. As I don't use
SWIG, I don't know about the community, but at least for Boost.Python it
is quite active and questions on the mailing list are answered quickly
enough. Well, this is very quick but if you need more detailed
information, I recommend you to visit both websites.

Alan

Pierre