I just killed GIL!!!

[sturlamolden]

Hello Guys...

I just had one moment of exceptional clarity, during which realized
how I could get the GIL out of my way... It's so simple, I cannot help
wondering why nobody has thought of it before. Duh! Now I am going to
sit and and marvel at my creation for a while, and then go to bed
(it's past 2:30 a.m.) Tomorrow I will contemplate whether to sell this
little secret for big bucks, give it away for free, or just keep it to
myself...

Now you are probably thinking I reinvented the gunpowder, and are
running multiple processes. Not so. I am not running parallel
processes, like parallel python or the processing module in cheese
shop. I am running multiple THREADS. In fact, I am just using
threading.Thread. The source code is pure Python, so there is no C
magic, and I only used the stuff that's already there in the standard
library. So, I just made CPython do what everyone claim to be
impossible. One single process of CPython is using all the cpu power
of my dual-core laptop.

 

[Daniel Fetchinson]

Hello Guys...

I just had one moment of exceptional clarity, during which realized
how I could get the GIL out of my way... It's so simple, I cannot help
wondering why nobody has thought of it before. Duh! Now I am going to
sit and and marvel at my creation for a while, and then go to bed
(it's past 2:30 a.m.) Tomorrow I will contemplate whether to sell this
little secret for big bucks, give it away for free, or just keep it to
myself...

Now you are probably thinking I reinvented the gunpowder, and are
running multiple processes. Not so. I am not running parallel
processes, like parallel python or the processing module in cheese
shop. I am running multiple THREADS. In fact, I am just using
threading.Thread. The source code is pure Python, so there is no C
magic, and I only used the stuff that's already there in the standard
library. So, I just made CPython do what everyone claim to be
impossible. One single process of CPython is using all the cpu power
of my dual-core laptop.

If I were you I would keep it a secret until a Hollywood producer
offers big bucks for the film rights.

 

[Tim Daneliuk]

If I were you I would keep it a secret until a Hollywood producer
offers big bucks for the film rights.

Who would play Guido, I wonder?

 

[Matias Surdi]

It's april 1st again???


sturlamolden escribió:

 

[Jonathan Gardner]

One single process of CPython is using all the cpu power
of my dual-core laptop.

Are they stuck in a while loop, waiting for their resource to become
available?

Using 100% of the CPU is a bug, not a feature. If you can't rewrite
your algorithm to be disk or network bound, next optimization step is
C.

 

[Steve Holden]

Hello Guys...

I just had one moment of exceptional clarity, during which realized
how I could get the GIL out of my way... It's so simple, I cannot help
wondering why nobody has thought of it before. Duh! Now I am going to
sit and and marvel at my creation for a while, and then go to bed
(it's past 2:30 a.m.) Tomorrow I will contemplate whether to sell this
little secret for big bucks, give it away for free, or just keep it to
myself...

Now you are probably thinking I reinvented the gunpowder, and are
running multiple processes. Not so. I am not running parallel
processes, like parallel python or the processing module in cheese
shop. I am running multiple THREADS. In fact, I am just using
threading.Thread. The source code is pure Python, so there is no C
magic, and I only used the stuff that's already there in the standard
library. So, I just made CPython do what everyone claim to be
impossible. One single process of CPython is using all the cpu power
of my dual-core laptop.

Quick, write it down before the drugs wear off.

Sorry. I'm definitely a skeptic on this story. Put up or shut up.

regards
Steve

 

[Carl Banks]

Using 100% of the CPU is a bug, not a feature.

No it isn't. That idea is borne of the narrowmindedness of people who
write server-like network apps. What's true for web servers isn't
true for every application.

If you can't rewrite
your algorithm to be disk or network bound, next optimization step is
C.

I'm sorry, but I don't like being told to use C. Perhaps I would like
the expressiveness of Python, am willing to accept the cost in
performance, but would also like to take advantage of technology to
get performance gains when I can? What's so unreasonable about that?

Look, the GIL is not a good thing. It's a trade off. Probably a good
one, too, considering the history of Python's development and the way
it simplifies writing extensions. But it's not, by itself, a good
thing.


For the record, I am not complaining about that GIL. As I said, I
understand and approve of why it's there. I am, however, complaining
about attitude that if you want to be free of the GIL you're doing
something wrong.


Carl Banks

 

[Martin v. Löwis]

For the record, I am not complaining about that GIL. As I said, I

understand and approve of why it's there. I am, however, complaining
about attitude that if you want to be free of the GIL you're doing
something wrong.

If you _want_ to be free of the GIL, you are not _doing_ anything, and
that may or may not be wrong. If you are complaining about the GIL,
I think you are doing something wrong, because complaining doesn't
help progress at all. I think neither was the case in this thread -
the guy claimed that he actually did something about the GIL, and
now we are all waiting for him to also tell us what it is that he
did. I think it is somewhat wrong to not tell in the first place,
but this is free software, and choosing not to contribute isn't
inherently wrong. Maybe the guy is making fun of us; whether that
is wrong or not depends on your notion of humor.

Regards,
Martin

 

[Marco Mariani]

Ralf Möller. No other.

And the GIL killer?

Clive Owen, Matt Damon, Mark Wahlberg?

 

[Chris]

And the GIL killer?

Clive Owen, Matt Damon, Mark Wahlberg?

Marky Mark ftw, he could even rap about it his method.

 

[Martin P. Hellwig]

sturlamolden wrote:

You killed the GIL, you bastard!

Hello Guys...

I just had one moment of exceptional clarity, during which realized
how I could get the GIL out of my way... It's so simple, I cannot help
wondering why nobody has thought of it before. Duh! Now I am going to
sit and and marvel at my creation for a while, and then go to bed
(it's past 2:30 a.m.) Tomorrow I will contemplate whether to sell this
little secret for big bucks, give it away for free, or just keep it to
myself...

Sounds good, though I have a question.
Is it transparent, meaning I can just write my code the way I used too
and it makes the decision for itself to spread the tasks on multiple
CPU's without me to worry about shared memory, spinlocks, deadlocks and
out of order inter-depending processing of subtask without heaving
having so much overhead that it has no performance advantage?

If not, what is the advantage above already present solutions?

 

[Steve Holden]

If you _want_ to be free of the GIL, you are not _doing_ anything, and
that may or may not be wrong. If you are complaining about the GIL,
I think you are doing something wrong, because complaining doesn't
help progress at all. I think neither was the case in this thread -
the guy claimed that he actually did something about the GIL, and
now we are all waiting for him to also tell us what it is that he
did. I think it is somewhat wrong to not tell in the first place,
but this is free software, and choosing not to contribute isn't
inherently wrong. Maybe the guy is making fun of us; whether that
is wrong or not depends on your notion of humor.

I suspect rather that he is merely deluded. Time alone will tell - the
OP is a fertile source of ideas, not all of them fully baked, shall we
say. I agree that the coyness in not immediately revealing the technique
is less than helpful, and increases my assessment that the assertion of
having "killed the GIL" will turn out to be wrong.

I'd love to be wrong about that, but the GIL *has* been the subject of
extensive efforts to kill it over the last five years, and it has
survived despite the best efforts of the developers.

regards
Steve

 

[sturlamolden]

If not, what is the advantage above already present solutions?

Well... I like the processing module. Except that Wintendo toy OS has
no fork() availabe for the Win32 subsystem, which makes it a bit
limited on that platform (slow at starting up, dependent on object
serialization, difficulties handling global variables, etc). The
advantage of my solution is that I can use threads and one process,
which makes it more friendly to Windows. On the other hand, the
processing module has the advantage that it cna be used on distributed
memory systems (e.g. clusters).

 

[sturlamolden]

help progress at all. I think neither was the case in this thread -
the guy claimed that he actually did something about the GIL, and
now we are all waiting for him to also tell us what it is that he
did.

Ok, I did not remove the GIL, but I found a way to remove its
notorious side effect on SMPs. So I am going to reveal it now. Forgive
my strange sence of humor at 3 o'clock in the morning.

First, think about this:

(1) What is the GIL?
(2) Where does it live?
(3) How can it be manually released?

The answer to this is:

(1) A critical section (a lock/mutex object)
(2) As a global object in Python25.dll on my computer
(3) Using Python's C API or calling methods in a ctypes.CDLL object

The Python C API has the ability to embed Python interpreters. You do
this by importing Python25.dll into the process. ctypes has the
ability to call functions in a DLL. So is it possible to embed Python
in Python? And what would be consequence be?

First, if I try to load a DLL more than once, Windows will detect this
and just give me a handle to the currently imported library. So by
importing Python25.dll with ctypes, I can just make my Python
interpreter talk to itself through its own CAPI. Yes I can create sub
interpreters using PyNew_Interpreter, but they all share the same GIL,
so it's not useful here.

So here is what I suddendly realized, and please don't laugh, its so
simple its almost silly:

I make some copies of Python25.dll, and call them Python25-1.dll,
Python25-2.dll, Python25-3.dll, Python25-4.dll, etc. Then I load them
all into my current Python process as ctypes.CDLL objects. Tada! I now
have a pool of independent Python interpreters, not sharing the GIL,
but still living in the same process.

I can make the different interpreters talk to each other, including
manipulating each other's GIL, using the using ctypes and Python's C
API for embedding.

So why does this circumvent the GIL? Because ctypes releases it before
calling functions form a CDLL object.

If I use my main interpreter to delegate a task to one of its embedded
'children', its GIL will be released while it is waiting for the
answer. Associating each embedded interpreter with a threading.Thread
is all that remains. The GIL is released while the thread operating
the child interpreter is blocked.

An there you have the answer. It's really very simple



Regards,

Sturla Molden

 

[sturlamolden]

It's april 1st again???

Not according to my calendar. This was not meant as a joke. I think I
may have solved the GIL issue. See my answer to Martin v. Löwis for a
full explanation.

 

[sturlamolden]

Quick, write it down before the drugs wear off.

Hehe, I don't take drugs, apart from NSAIDs for arthritis. Read my
answer to Martin v. Löwis.

 

[MRAB]

Hallöchen!

Daniel Fetchinson wrote:

[...]
I just had one moment of exceptional clarity, during which
realized how I could get the GIL out of my way... It's so
simple, I cannot help wondering why nobody has thought of it
before. [...]
If I were you I would keep it a secret until a Hollywood producer
offers big bucks for the film rights.

Who would play Guido, I wonder?

Ralf Möller. No other.

If you're not careful Hollywood might re-invent Guido as an all-
American hero, Guy Ross!

 

[Hrvoje Niksic]

If I use my main interpreter to delegate a task to one of its
embedded 'children', its GIL will be released while it is waiting
for the answer. Associating each embedded interpreter with a
threading.Thread is all that remains. The GIL is released while the
thread operating the child interpreter is blocked.

Have you tackled the communication problem? The way I see it, one
interpreter cannot "see" objects created in the other because they
have separate pools of ... everything. They can communicate by
passing serialized objects through ctypes, but that sounds like the
solutions that use processes.

 

[Tim Daneliuk]

Hallöchen!

Daniel Fetchinson wrote:
[...]
I just had one moment of exceptional clarity, during which
realized how I could get the GIL out of my way... It's so
simple, I cannot help wondering why nobody has thought of it
before. [...]
If I were you I would keep it a secret until a Hollywood producer
offers big bucks for the film rights.
Who would play Guido, I wonder?

Ralf Möller. No other.

If you're not careful Hollywood might re-invent Guido as an all-
American hero, Guy Ross!

"DIE HARD WITH NO GIL - He's Back And He's Pissed..."

 

[Rhamphoryncus]

Ok, I did not remove the GIL, but I found a way to remove its
notorious side effect on SMPs. So I am going to reveal it now. Forgive
my strange sence of humor at 3 o'clock in the morning.

First, think about this:

(1) What is the GIL?
(2) Where does it live?
(3) How can it be manually released?

The answer to this is:

(1) A critical section (a lock/mutex object)
(2) As a global object in Python25.dll on my computer
(3) Using Python's C API or calling methods in a ctypes.CDLL object

The Python C API has the ability to embed Python interpreters. You do
this by importing Python25.dll into the process. ctypes has the
ability to call functions in a DLL. So is it possible to embed Python
in Python? And what would be consequence be?

First, if I try to load a DLL more than once, Windows will detect this
and just give me a handle to the currently imported library. So by
importing Python25.dll with ctypes, I can just make my Python
interpreter talk to itself through its own CAPI. Yes I can create sub
interpreters using PyNew_Interpreter, but they all share the same GIL,
so it's not useful here.

So here is what I suddendly realized, and please don't laugh, its so
simple its almost silly:

I make some copies of Python25.dll, and call them Python25-1.dll,
Python25-2.dll, Python25-3.dll, Python25-4.dll, etc. Then I load them
all into my current Python process as ctypes.CDLL objects. Tada! I now
have a pool of independent Python interpreters, not sharing the GIL,
but still living in the same process.

I can make the different interpreters talk to each other, including
manipulating each other's GIL, using the using ctypes and Python's C
API for embedding.

So why does this circumvent the GIL? Because ctypes releases it before
calling functions form a CDLL object.

If I use my main interpreter to delegate a task to one of its embedded
'children', its GIL will be released while it is waiting for the
answer. Associating each embedded interpreter with a threading.Thread
is all that remains. The GIL is released while the thread operating
the child interpreter is blocked.

An there you have the answer. It's really very simple

Interesting. Windows specific, but there's other ways to do the same
thing more portably.

The bigger issue is that you can't share any objects. This
effectively gives you a multiprocess model - a bit cheaper than that,
but not enough to really supply GIL-free threading.