Will Python 3.0 remove the global interpreter lock (GIL)

Discussion in 'Python' started by llothar, Sep 3, 2007.

  1. llothar

    llothar Guest

    I'm afraid that the GIL is killing the usefullness of python for some
    types of applications now where 4,8 oder 64 threads on a chip are here
    or comming soon.

    What is the status about that for the future of python?

    I know that at the moment allmost nobody in the scripting world has
    solved this problem, but it bites and it bites hard. Only groovy as a
    Java Plugin has support but i never tried it. Writing an interpreter
    that does MT this seems to be extremely difficult to do it right, with
    lots of advanced stuff like CAS and lock free programming.

    Even Smalltalk and Common Lisp didn't get it until know (with the
    exception of certain experiments).
     
    llothar, Sep 3, 2007
    #1
    1. Advertising

  2. On 9/2/07, llothar <> wrote:
    > I'm afraid that the GIL is killing the usefullness of python for some
    > types of applications now where 4,8 oder 64 threads on a chip are here
    > or comming soon.
    >
    > What is the status about that for the future of python?
    >
    > I know that at the moment allmost nobody in the scripting world has
    > solved this problem, but it bites and it bites hard. Only groovy as a
    > Java Plugin has support but i never tried it. Writing an interpreter
    > that does MT this seems to be extremely difficult to do it right, with
    > lots of advanced stuff like CAS and lock free programming.
    >
    > Even Smalltalk and Common Lisp didn't get it until know (with the
    > exception of certain experiments).



    No. http://www.artima.com/weblogs/viewpost.jsp?thread=211430

    --
    http://www.advogato.org/person/eopadoan/
    Bookmarks: http://del.icio.us/edcrypt
     
    Eduardo O. Padoan, Sep 3, 2007
    #2
    1. Advertising

  3. llothar

    llothar Guest

    llothar, Sep 3, 2007
    #4
  4. On Sep 2, 11:16 pm, llothar <> wrote:
    > On 3 Sep., 07:38, "Eduardo O. Padoan" <>
    > wrote:
    >
    > > > No.http://www.artima.com/weblogs/viewpost.jsp?thread=211430

    >
    > > Ops, I meant:http://www.artima.com/forums/threaded.jsp?forum=106&thread=211200

    >
    > Thanks. I whish there would be a project for rewritting the C
    > interpreter
    > to make it better and more useable for threading use.
    >
    > But the CPU infrastructure is also not perfect enough so maybe it's
    > good to
    > wait with this a few more years until Intel and AMD know what they are
    > doing.



    I read somewhere that PYPY won't have the interpreter lock (I may be
    wrong though).
    Check it out: http://codespeak.net/pypy
     
    =?iso-8859-1?q?Luis_M=2E_Gonz=E1lez?=, Sep 3, 2007
    #5
  5. llothar

    Evan Klitzke Guest

    On Sun, 2007-09-02 at 17:21 -0700, llothar wrote:
    > I'm afraid that the GIL is killing the usefullness of python for some
    > types of applications now where 4,8 oder 64 threads on a chip are here
    > or comming soon.
    >
    > What is the status about that for the future of python?


    The GIL is an implementation specific issue with CPython. It will not be
    removed in CPython for the forseeable future, but you can already get a
    GIL-free interpreter with Jython and IronPython. AFAIK there are plans
    to remove the GIL in PyPy.

    According to the last PyPy release announcement, they're running at
    about half the speed of CPython, and have a preliminary JIT that can
    translate certain integer operations into assembly, and will be expanded
    upon in future releases. If you're looking for a progressive alternative
    to CPython, I'd keep an eye on that project ;-)

    --
    Evan Klitzke <>
     
    Evan Klitzke, Sep 3, 2007
    #6
  6. On Sep 3, 2:21 am, llothar <> wrote:
    > I'm afraid that the GIL is killing the usefullness of python for some
    > types of applications now where 4,8 oder 64 threads on a chip are here
    > or comming soon.
    >
    > What is the status about that for the future of python?


    This is FAQ. You will find thousands of discussion on the net about
    that.
    My personal opinion (and I am not the only one in the Python
    community) is that
    if you want to scale the way to go is to use processes, not threads,
    so removing the GIL would be a waste of effort anyway.
    Look at the 'processing' module in PyPI.

    Michele Simionato
     
    Michele Simionato, Sep 3, 2007
    #7
  7. llothar

    king kikapu Guest

    On Sep 3, 9:15 am, Michele Simionato <>
    wrote:
    > On Sep 3, 2:21 am, llothar <> wrote:
    >


    > My personal opinion (and I am not the only one in the Python
    > community) is that
    > if you want to scale the way to go is to use processes, not threads,
    > so removing the GIL would be a waste of effort anyway.
    > Look at the 'processing' module in PyPI.
    >
    > Michele Simionato



    I second that. You may also look here,
    http://www.parallelpython.com/

    I tested it and work as expected. You can see all your processing-
    cores work nicely and balanced.
     
    king kikapu, Sep 3, 2007
    #8
  8. llothar

    Ben Finney Guest

    Michele Simionato <> writes:

    > On Sep 3, 2:21 am, llothar <> wrote:
    > > I'm afraid that the GIL is killing the usefullness of python for
    > > some types of applications now where 4,8 oder 64 threads on a chip
    > > are here or comming soon.

    >
    > This is FAQ. You will find thousands of discussion on the net about
    > that.


    > My personal opinion (and I am not the only one in the Python
    > community) is that if you want to scale the way to go is to use
    > processes, not threads, so removing the GIL would be a waste of
    > effort anyway.


    Yes. Processes are cheap on well-designed operating systems, and using
    processes to subdivide your processor usage encourages simple, modular
    interfaces between parts of a program. Threads, while also cheap, are
    much more difficult and fiddly to program correctly, hence are rarely
    an overall win.

    One common response to that is "Processes are expensive on Win32". My
    response to that is that if you're programming on Win32 and expecting
    the application to scale well, you already have problems that must
    first be addressed that are far more fundamental than the GIL.

    --
    \ "Pinky, are you pondering what I'm pondering?" "Umm, I think |
    `\ so, Brain, but three men in a tub? Ooh, that's unsanitary!" -- |
    _o__) _Pinky and The Brain_ |
    Ben Finney
     
    Ben Finney, Sep 3, 2007
    #9
  9. llothar

    king kikapu Guest

    I was wondering (and maybe i still do) about this GIL "problem". I am
    relatively new to Python (less than a year) and when i started to
    think about it i said: "Oh, this IS a problem". But when i dig a
    little more, i found that "Ah, maybe it isn't".
    I strongly believe that the best usage of multiple cores processor
    will be achieved if programming languages are modified to support this
    on their "hearts". Code blocks that would be identified by the
    compiler and run in parallel and such things. Laboratories are working
    on these stuff but i do not expect something in the very-near future.

    So, as i mentioned above, there are solutions for that right now
    ("parallel python" and others) that enabled us with little effort to
    spawn a new python interpreter, thus allowing the OS to schedule it on
    a different core and do the job this way relatively cheap.
    I wouldn't recommend going to IronPython despite the fact that the CLR
    better utilize MP. The reason for this is that i would NEVER give up
    the freedom that CPython gives me by exchange "better" usage of the MP
    and platform lock-in.
     
    king kikapu, Sep 3, 2007
    #10
  10. llothar

    king kikapu Guest

    I was wondering (and maybe i still do) about this GIL "problem". I am
    relatively new to Python (less than a year) and when i started to
    think about it i said: "Oh, this IS a problem". But when i dig a
    little more, i found that "Ah, maybe it isn't".
    I strongly believe that the best usage of multiple cores processor
    will be achieved if programming languages are modified to support this
    on their "hearts". Code blocks that would be identified by the
    compiler and run in parallel and such things. Laboratories are working
    on these stuff but i do not expect something in the very-near future.

    So, as i mentioned above, there are solutions for that right now
    ("parallel python" and others) that enabled us with little effort to
    spawn a new python interpreter, thus allowing the OS to schedule it on
    a different core and do the job this way relatively cheap.
    I wouldn't recommend going to IronPython despite the fact that the CLR
    better utilize MP. The reason for this is that i would NEVER give up
    the freedom that CPython gives me by exchange "better" usage of the MP
    and platform lock-in.
     
    king kikapu, Sep 3, 2007
    #11
  11. llothar

    king kikapu Guest

    I was wondering (and maybe i still do) about this GIL "problem". I am
    relatively new to Python (less than a year) and when i started to
    think about it i said: "Oh, this IS a problem". But when i dig a
    little more, i found that "Ah, maybe it isn't".
    I strongly believe that the best usage of multiple cores processor
    will be achieved if programming languages are modified to support this
    on their "hearts". Code blocks that would be identified by the
    compiler and run in parallel and such things. Laboratories are working
    on these stuff but i do not expect something in the very-near future.

    So, as i mentioned above, there are solutions for that right now
    ("parallel python" and others) that enabled us with little effort to
    spawn a new python interpreter, thus allowing the OS to schedule it on
    a different core and do the job this way relatively cheap.
    I wouldn't recommend going to IronPython despite the fact that the CLR
    better utilize MP. The reason for this is that i would NEVER give up
    the freedom that CPython gives me by exchange "better" usage of the MP
    and platform lock-in.
     
    king kikapu, Sep 3, 2007
    #12
  12. On Sep 2, 5:38 pm, "Eduardo O. Padoan" <>
    wrote:
    > > No.http://www.artima.com/weblogs/viewpost.jsp?thread=211430

    >
    > Ops, I meant:http://www.artima.com/forums/threaded.jsp?forum=106&thread=211200
    >
    > --http://www.advogato.org/person/eopadoan/
    > Bookmarks:http://del.icio.us/edcrypt


    "No. We're not changing the CPython implementation much. Getting rid
    of the GIL would be a massive rewrite of the interpreter because all
    the internal data structures (and the reference counting operations)
    would have to be made thread-safe. This was tried once before (in the
    late '90s by Greg Stein) and the resulting interpreter ran twice as
    slow."

    How much faster/slower would Greg Stein's code be on today's
    processors versus CPython running on the processors of the late
    1990's? And if you decide to answer, please add a true/false response
    to this statement - "CPython in the late 1990's ran too slow".
     
    TheFlyingDutchman, Sep 19, 2007
    #13
  13. llothar

    Terry Reedy Guest

    "TheFlyingDutchman" <> wrote in message
    news:...
    | On Sep 2, 5:38 pm, "Eduardo O. Padoan" <>
    | wrote:
    | > > No.http://www.artima.com/weblogs/viewpost.jsp?thread=211430
    | >
    | > Ops, I
    meant:http://www.artima.com/forums/threaded.jsp?forum=106&thread=211200
    | >
    | > --http://www.advogato.org/person/eopadoan/
    | > Bookmarks:http://del.icio.us/edcrypt
    |
    | "No. We're not changing the CPython implementation much. Getting rid
    | of the GIL would be a massive rewrite of the interpreter because all
    | the internal data structures (and the reference counting operations)
    | would have to be made thread-safe. This was tried once before (in the
    | late '90s by Greg Stein) and the resulting interpreter ran twice as
    | slow."

    Since Guido wrote that, there have been put forth more ideas and interest
    and promises of efforts to remove or revise the GIL or do other things to
    make using multiple cores easier. (The later being the point of the
    concern over GIL.)

    | How much faster/slower would Greg Stein's code be on today's
    | processors versus CPython running on the processors of the late
    | 1990's?

    Perhaps a bit faster, though processor speeds have not increased so much
    the last couple of years.

    |And if you decide to answer, please add a true/false response
    | to this statement - "CPython in the late 1990's ran too slow".

    False by late 1990's standards, True by today's standards ;-).

    Most people are not currently bothered by the GIL and would not want its
    speed halved.

    In any case, any of the anti-GIL people are free to update Stein's code and
    distribute a GIl-less version of CPython. (Or to use Jython or
    IronPython.)

    tjr
     
    Terry Reedy, Sep 19, 2007
    #14
  14. On Tue, 18 Sep 2007 18:09:26 -0700, TheFlyingDutchman wrote:

    > How much faster/slower would Greg Stein's code be on today's processors
    > versus CPython running on the processors of the late 1990's?


    I think a better question is, how much faster/slower would Stein's code
    be on today's processors, versus CPython being hand-simulated in a giant
    virtual machine made of clockwork?



    --
    Steven.
     
    Steven D'Aprano, Sep 19, 2007
    #15
  15. On 2007-09-19, Steven D'Aprano <> wrote:
    > On Tue, 18 Sep 2007 18:09:26 -0700, TheFlyingDutchman wrote:
    >
    >> How much faster/slower would Greg Stein's code be on today's
    >> processors versus CPython running on the processors of the
    >> late 1990's?

    >
    > I think a better question is, how much faster/slower would
    > Stein's code be on today's processors, versus CPython being
    > hand-simulated in a giant virtual machine made of clockwork?


    That depends on whether you have the steam-driven model or the
    water-wheel driven model. The steam-drive one is quite a bit
    faster once you get it going, but the water-wheel model has a
    much shorter start-up time (though it is more weather
    dependent).

    --
    Grant Edwards grante Yow! Hey, waiter! I want
    at a NEW SHIRT and a PONY TAIL
    visi.com with lemon sauce!
     
    Grant Edwards, Sep 19, 2007
    #16
  16. On Sep 19, 8:51 am, Steven D'Aprano <st...@REMOVE-THIS-
    cybersource.com.au> wrote:
    > On Tue, 18 Sep 2007 18:09:26 -0700, TheFlyingDutchman wrote:
    > > How much faster/slower would Greg Stein's code be on today's processors
    > > versus CPython running on the processors of the late 1990's?

    >
    > I think a better question is, how much faster/slower would Stein's code
    > be on today's processors, versus CPython being hand-simulated in a giant
    > virtual machine made of clockwork?
    >
    > --
    > Steven.


    Steven, You forgot this part:

    "And if you decide to answer, please add a true/false response
    to this statement - "CPython in the late 1990's ran too slow"'.
     
    TheFlyingDutchman, Sep 19, 2007
    #17
  17. llothar

    Paul Boddie Guest

    On 19 Sep, 03:09, TheFlyingDutchman <> wrote:
    >
    > How much faster/slower would Greg Stein's code be on today's
    > processors versus CPython running on the processors of the late
    > 1990's? And if you decide to answer, please add a true/false response
    > to this statement - "CPython in the late 1990's ran too slow".


    Too slow for what? And what's the fixation with CPython, anyway? Other
    implementations of Python 2.x don't have the GIL. Contrary to popular
    folklore, Jython has been quite a reasonable implementation of Python
    for about half as long as CPython has been around, if you don't mind
    the JVM. I'm sure people have lots of complaints about Jython like
    they do about CPython and the GIL, thinking that complaining about it
    is going to make the situation better, or that they're imparting some
    kind of "wisdom" to which the people who actually wrote the code must
    be oblivious, but nobody is withholding the code from anyone who wants
    to actually improve it.

    And there we learn something: that plenty of people are willing to
    prod others into providing them with something that will make their
    lives better, their jobs easier, and their profits greater, but not so
    many are interested in contributing back to the cause and taking on
    very much of the work themselves. Anyway, the response to your
    statement is "false". Now you'll have to provide us with the insight
    we're all missing. Don't disappoint!

    Paul
     
    Paul Boddie, Sep 19, 2007
    #18
  18. On Sep 19, 3:41 pm, Paul Boddie <> wrote:
    > On 19 Sep, 03:09, TheFlyingDutchman <> wrote:
    >
    >
    >
    > > How much faster/slower would Greg Stein's code be on today's
    > > processors versus CPython running on the processors of the late
    > > 1990's? And if you decide to answer, please add a true/false response
    > > to this statement - "CPython in the late 1990's ran too slow".

    >
    > Too slow for what? And what's the fixation with CPython, anyway? Other
    > implementations of Python 2.x don't have the GIL. Contrary to popular
    > folklore, Jython has been quite a reasonable implementation of Python
    > for about half as long as CPython has been around, if you don't mind
    > the JVM. I'm sure people have lots of complaints about Jython like
    > they do about CPython and the GIL, thinking that complaining about it
    > is going to make the situation better, or that they're imparting some
    > kind of "wisdom" to which the people who actually wrote the code must
    > be oblivious, but nobody is withholding the code from anyone who wants
    > to actually improve it.


    >
    > And there we learn something: that plenty of people are willing to
    > prod others into providing them with something that will make their
    > lives better, their jobs easier, and their profits greater, but not so
    > many are interested in contributing back to the cause and taking on
    > very much of the work themselves. Anyway, the response to your
    > statement is "false". Now you'll have to provide us with the insight
    > we're all missing. Don't disappoint!
    >
    > Paul


    Paul it's a pleasure to see that you are not entirely against
    complaints.

    The very fastest Intel processor of the last 1990's that I found came
    out in October 1999 and had a speed around 783Mhz. Current fastest
    processors are something like 3.74 Ghz, with larger caches. Memory is
    also faster and larger. It appears that someone running a non-GIL
    implementation of CPython today would have significantly faster
    performance than a GIL CPython implementation of the late 1990's.
    Correct me if I am wrong, but it seems that saying non-GIL CPython is
    too slow, while once valid, has become invalid due to the increase in
    computing power that has taken place.
     
    TheFlyingDutchman, Sep 19, 2007
    #19
  19. llothar

    Terry Reedy Guest

    "Terry Reedy" <> wrote in message
    news:fcq352$u0a$...
    |
    | "TheFlyingDutchman" <> wrote in message
    | news:...

    | Since Guido wrote that, there have been put forth more ideas and interest
    | and promises of efforts to remove or revise the GIL or do other things to
    | make using multiple cores easier. (The later being the point of the
    | concern over GIL.)

    A few days ago, an undergraduate posted on the dev list that he just
    started an independent study project on removing the GIL. Maybe we'll get
    a report early next year.

    Guido also said that he is willing to make changes to the CPython internals
    to aid multiproccessor usage [as long, presumably, as it does not cut speed
    in half].

    || How much faster/slower would Greg Stein's code be on today's
    || processors versus CPython running on the processors of the late
    || 1990's?
    |
    | Perhaps a bit faster, though processor speeds have not increased so much
    | the last couple of years.

    This assumes that comparing versions of 1.5 is still relevant. As far as I
    know, his patch has not been maintained to apply against current Python.
    This tells me that no one to date really wants to dump the GIL at the cost
    of half Python's speed. Of course not. The point of dumping the GIL is to
    use multiprocessors to get more speed! So with two cores and extra
    overhead, Stein-patched 1.5 would not even break even.

    Quad (and more) cores are a different matter. Hence, I think, the
    resurgence of interest.

    ||And if you decide to answer, please add a true/false response
    || to this statement - "CPython in the late 1990's ran too slow".
    |
    | False by late 1990's standards, True by today's standards ;-).

    So now this question for you: "CPython 2.5 runs too slow in 2007: true or
    false?"

    If you answer false, then there is no need for GIL removal.
    If you answer true, then cutting its speed for 90+% of people is bad.

    | Most people are not currently bothered by the GIL and would not want its
    | speed halved.

    And another question: why should such people spend time they do not have to
    make Python worse for themselves?

    Terry Jan Reedy
     
    Terry Reedy, Sep 20, 2007
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Aahz
    Replies:
    0
    Views:
    955
  2. Fuzzyman
    Replies:
    3
    Views:
    500
    Andrew MacIntyre
    Dec 5, 2003
  3. Tomas Christiansen

    Global Interpreter Lock

    Tomas Christiansen, Sep 24, 2004, in forum: Python
    Replies:
    3
    Views:
    309
    Michael Hoffman
    Sep 24, 2004
  4. Paul Rubin

    global interpreter lock

    Paul Rubin, Aug 19, 2005, in forum: Python
    Replies:
    59
    Views:
    1,307
    Michael Sparks
    Sep 15, 2005
  5. Replies:
    2
    Views:
    286
Loading...

Share This Page