Scaling on multi-processor machine

[Atul Kshirsagar]

Hello,
GIL prevents my C++ application embedding and extending python to scale even
though I spawn multiple C++ threads. I read lot of references on internet
about using multiple processes rather than threads.

In that direction I was searching for some examples/resources/tools which
can help me create multi-process implementation. My application does need to
pass userdefined class objects to python interpreter. So I guess my
implementation is going to need a protocol to marshal, de-marshal python
objects between processes or use shared memory.

Does anyone here know of any examples/tools to implement the multi-processes
approach with the object/data transfer between the processes ?

 

[Aahz]

GIL prevents my C++ application embedding and extending python to scale even
though I spawn multiple C++ threads. I read lot of references on internet
about using multiple processes rather than threads.

You're not getting any scaling, or you're not getting enough scaling?

 

[Dieter Maurer]

GIL prevents my C++ application embedding and extending python to scale even
though I spawn multiple C++ threads. I read lot of references on internet
about using multiple processes rather than threads.

While your C++ threads are not touching Python objects, they
can release the GIL (but must acquire it again before they touch Python
objects, e.g. call into Python).

Thus, you can have your C++ threads work in parallel as long as
they do not touch Python objects.

Dieter