using python interpreters per thread in C++ program

[sturlamolden]

My application is a TCP server having multiple client connectons. C++
PTHREADS are for each connected socket and the message received on the
socket is evaluated by python functions.
If I use only one process level python interpreter, then every thread
has to lock the GIL & so blocking the other threads from executing the
python code even if it is not the same python function that locking
thread is calling.

Usually, TCP-servers are I/O bound. You can safely use a single Python
process for this. A function evaluating a request will hold the GIL
for a while (but not until it's done). But most other threads will be
blocked waiting for I/O. Thus, there will be little contention for the
GIL anyway, and it should not affect scalability much. Only when
multiple requests are processed simultaneously will threre be
contention for the GIL. You can create high-performance TCP servers in
plain Python using e.g. Twisted. If you are in the strange situation
that a TCP server is compute-bound, consider using multiple processes
(os.fork or multiprocessing).

I think there is no way that we can achieve this because of the GIL
being a process level state. Least I can do is have one python
interpreter initialized in main thread and lock the GIL in every
thread for python calls.

I think you will find out your server is indeed I/O bound, like 99.9%
or all other TCP servers on this planet. Try to embed a single
interpreter first. Use the simplified GIL API I showed you. Most
likely you will find that it suffice.

If you need something more scalable, associate each pthread with a
separate Python process - e.g. using a named pipe on Windows or Unix
domain socket on Linux.

 

[ganesh]

Do you have to use threads? If you use a process per connection, rather
than a thread, each process will have its own GIL.

No, i cannot change from threads to processes for handling
connections. This will change the complete design of our application
which is not feasilbe for python evaluation of the strings.

 

[sturlamolden]

Do you have to use threads? If you use a process per connection, rather
than a thread, each process will have its own GIL.

If ganesh is using Linux or Unix (which pthreads indicate), fork() is
just as efficient as threads.

On Windows one would need to keep a farm of prespawned Python
processes, connected with pipes to the main server.

 

[sturlamolden]

No, i cannot change from threads to processes for handling
connections. This will change the complete design of our application
which is not feasilbe for python evaluation of the strings.

So the problem is actually bad design?

 

[Graham Dumpleton]

 I was referring to the
'multiple interpreters in one process' feature of Python which is
largely deprecated, ...

Can you please point to where in the documentation for Python it says
that support for multiple interpreters in one process is 'largely
deprecated'.

I know that various people would like the feature to go away, but I
don't believe I have ever seen an official statement from Guido or
other person in a position to make one, state that the official view
was that the API was deprecated.

Even in Python 3.1 the documentation for the APIs seems to merely
state some of the limitations and that it is a hard problem, even
still saying that problem would be addressed in future versions.

Graham