GIL in the new glossary

[Luis P Caamano]

global interpreter lock

The lock used by Python threads to assure that only one thread
can be run at a time. This simplifies Python by assuring that no
two processes can access the same memory at the same time. Locking
the entire interpreter makes it easier for the interpreter to be
multi-threaded, at the expense of some parallelism on
multi-processor machines.

Some parallelism??? Wouldn't it be more accurate to say
"at the expense of parallelism?" The GIL doesn't eliminate
"some" paralellism, it completely eliminates any chance of
parallelism within the same interpreter.

Efforts have been made in the past to create a "free-threaded"
interpreter (one which locks shared data at a much finer
granularity), but performance suffered in the common
single-processor case.

This is true for kernels too, which is why you see at least
two versions of the linux kernel in red hat distributions,
one with SMP enabled and one without it.

Why not have a python_smp interpreter that allows parallelism
(and scalability) on SMP machines and another for uniprocessor
machines where paralellism is not possible?

Yeah, yeah, I know what you're going to say ...

"Please submit a patch."

Sigh, ... if only I had the time ... :-(

 

[Skip Montanaro]

Luis> Some parallelism??? Wouldn't it be more accurate to say "at the
Luis> expense of parallelism?" The GIL doesn't eliminate "some"
Luis> paralellism, it completely eliminates any chance of parallelism
Luis> within the same interpreter.

The global interpreter lock can explicitly be released from C code. Lots of
I/O code does this already. Only one thread can be executing Python byte
code at a time, but multiple threads can execute non-interpreter C code.
(I'm sure someone more knowledgeable about threads will correct me if I'm
off-base here.)

Luis> This is true for kernels too, which is why you see at least
Luis> two versions of the linux kernel in red hat distributions,
Luis> one with SMP enabled and one without it.

Luis> Why not have a python_smp interpreter that allows parallelism
Luis> (and scalability) on SMP machines and another for uniprocessor
Luis> machines where paralellism is not possible?

Luis> Yeah, yeah, I know what you're going to say ...

Luis> "Please submit a patch."

Luis> Sigh, ... if only I had the time ... :-(

I think you'd need a lot more time than you think (hint: don't think in
terms of "a patch"; think more in terms of "code fork"). ;-)

Greg Stein (an extremely bright guy who doesn't seem to hang out in the
Python niches much anymore - our loss) put in a fair amount of effort on
this when 1.4 was the current Python version, so you know this is something
people have been asking about for quite some time. Here's a relevant
message from Greg regarding the performance he saw:

http://mail.python.org/pipermail/python-dev/2001-August/017099.html

Based upon his numbers, I'm not sure it would be worth the effort, even if
you could get it right. Going from 1x to 1.2x performance when adding a
second processor hardly seems worth it.

To make matters worse, Python has only gotten more complex in the time since
Greg did his work. In addition, a number of things added to the language or
the library since then have quietly assumed the presence of the global
interpreter lock - "quietly", as in there's no big old "XXX" comment in the
code alerting would-be free threaders. Taken together, this means there are
many more bits of code which would need attention. It's probably a lot
easier to wait for the next fastest uniprocessor machine or overclock the
one you have.

Skip

 

[Alan Kennedy]

[Python Glossary]
[Luis P Caamano wrote]

Some parallelism??? Wouldn't it be more accurate to say
"at the expense of parallelism?" The GIL doesn't eliminate
"some" paralellism, it completely eliminates any chance of
parallelism within the same interpreter.

I just wanted to point out a relation that I have noticed has held
true for almost all GIL discussion threads that have come up in this
forum recently. That relation is:-

The amount of loud complaining and misinformation that originates from
people who have just found out about the GIL is inversely proportional
to the amount of factual knowledge that they have about the GIL.

Or:- The less they understand the subject, the more loudly they
complain.

dont-complain-before-finding-out-the-facts-ly y'rs,

 

[Andrew Dalke]

Alan Kennedy:

The amount of loud complaining and misinformation that originates from
people who have just found out about the GIL is inversely proportional
to the amount of factual knowledge that they have about the GIL.

Consider the statement

The amount of loud complaining and misinformation ... about X is
inversely proportional to the amount of factual knowledge ... about X

Isn't that generally true, even when X != GIL? I know I've
contributed my own share of complaining and misinformation
when I'm just starting to learn an API.

Andrew
(e-mail address removed)

 

[Harri Pesonen]

Some parallelism??? Wouldn't it be more accurate to say
"at the expense of parallelism?" The GIL doesn't eliminate
"some" paralellism, it completely eliminates any chance of
parallelism within the same interpreter.

I agree, GIL sucks.

This is true for kernels too, which is why you see at least
two versions of the linux kernel in red hat distributions,
one with SMP enabled and one without it.

Why not have a python_smp interpreter that allows parallelism
(and scalability) on SMP machines and another for uniprocessor
machines where paralellism is not possible?

Yeah, yeah, I know what you're going to say ...

"Please submit a patch."

Sigh, ... if only I had the time ... :-(

Exactly. I have began work on this. I call it Viper, because we don't
have Pythons in Finland. It takes the latest Python C API source code
and makes it GIL free. It is a Python script.

I have just began the work, currently it just removes the deallocator
from static objects like Py_None so that it does not matter what the
reference count is. Static objects will be never deallocated, so they
are thread-safe.

Next step is to add the pointer to thread state to every C API function,
so that there is no global state anymore. This will remove GIL, or
actually convert it to LIL, local interpreter lock. LIL is still needed
if you want to create threads in Python code. This steps makes it
possible to create several free-threading interpreters in the same process.

The final step is to remove LIL, but this breaks the existing Python
threading code, so something substituting should be developed. I had an
idea of shared interpreter state for shared memory access. Or separate
functions for inter-thread communications.

Harri

 

[John J. Lee]

Why not have a python_smp interpreter that allows parallelism
(and scalability) on SMP machines and another for uniprocessor
machines where paralellism is not possible?

[...]

Just noticed this

http://www.python.org/doc/faq/library.html#can-t-we-get-rid-of-the-global-interpreter-lock


John

 

[Jp Calderone]

Why not have a python_smp interpreter that allows parallelism
(and scalability) on SMP machines and another for uniprocessor
machines where paralellism is not possible?

[...]

Just noticed this

http://www.python.org/doc/faq/library.html#can-t-we-get-rid-of-the-global-interpreter-lock

Well thats easy to fix. Just don't have *any* locks in the SMP version...

Jp