Python, multithreading & GIL

[Carl Banks]

I don't intend to guess at what his application is about,
but that's the only case I ever hear about where it even
theoretically matters. An application with a trivial
computational aspect will run more or less concurrently.


Because it would still be slow.

I'm not arguing that the GIL is a feature, though there may
be a weak case for that (I've had pretty good luck with my
Python programs in a multithreaded system that is supposed
to be a big headache for C++ application programmers, and
I've wondered if the modest amount of extra serialization
Python imposes is actually helping me out there. But I haven't
worked that idea out, because - it doesn't matter, this issue
isn't going anywhere regardless.)

I'm not arguing that no one cares at all, or that it's
unreasonable to wish for it. I'm saying that the need for
free threading doesn't add up to enough motivation for anyone
to take on the very hairy task of a implementing it.

Well, that doesn't SOUND like what you were saying.

The assertion that having a GIL isn't a significant problem, or that
it's a bad idea to write SMP programs in pure Python, is much too
strong a statement, IMFO. That it's merely not significant enough of
a problem to justify doing the work to get rid of it is arguable (and,
so far, rather well borne out, although all bets off if Python ever
gets rid of reference counting).

 

[Lothar Scholz]

I agree
But now, looking at some other scripting languages, I don't see any that
claim to be able to do what we're discussing here. Does anybody know of a
scripting language good at "string crunching" that can exploit SMP with
threading?

Look at TCL and then mix the TCL model with pythons GIL.
TCL has multiple interpreters that work like different processes, this
means no shared memory at the script level. So if we get one GIL per
interpreter then we would get the best.

I think that we can now see that multi-core CPU's will be available in
the near future, even in desktop computers. So the Python limitation
will get more an more worse. But changing this requires a complete
change of the C API. So maybe an internal API to start and stop python
processes could be usefull, avoiding the need for multiple processors.

A script language is IMHO not the right tool to handle low level
thread synchronisation.

 

[Steve Holden]

Look at TCL and then mix the TCL model with pythons GIL.
TCL has multiple interpreters that work like different processes, this
means no shared memory at the script level. So if we get one GIL per
interpreter then we would get the best.

I think that we can now see that multi-core CPU's will be available in
the near future, even in desktop computers. So the Python limitation
will get more an more worse. But changing this requires a complete
change of the C API. So maybe an internal API to start and stop python
processes could be usefull, avoiding the need for multiple processors.

A script language is IMHO not the right tool to handle low level
thread synchronisation.

In the medium-term, perhaps the work being done on pypy will remove the
global interpreter lock as a practical requirement? Maybe someone active
on the project could comment.

Note also that it is perfectly possible to run separate multiple
interpreters inside a process, each with their own GIL, in much the way
you describe TCL as doing. One application using this approach is
mod_python.

The GIL is, though, in there for very sound practical reasons, and the
surgery required to remove it from the existing CPython implementation
might well prove fatal to the patient

regards
Steve

 

[Donn Cave]

Quoth Steve Holden <[email protected]>:
....
| In the medium-term, perhaps the work being done on pypy will remove the
| global interpreter lock as a practical requirement? Maybe someone active
| on the project could comment.

That would be interesting, if part of it is reducing the amount of
shared state that needs to be protected. I don't know what that means,
maybe some distinction in the storage model akin to the static/automatic
C model, where objects that never enter into any shared scope can be
handled without interlocks. But if it's just replacing the lock with
a lot of little locks for everything, then the ordinary single threaded
program will pay the cost and get none of the already dubious advantage.

Donn