Will multithreading make python less popular?

Discussion in 'Python' started by rushenaly@gmail.com, Feb 16, 2009.

  1. Guest

    Hi everybody,
    I am an engineer. I am trying to improve my software development
    abilities. I have started programming with ruby. I like it very much
    but i want to add something more. According to my previous research i
    have designed a learning path for myself. It's like something below.
    1. Ruby (Mastering as much as possible)
    2. Python (Mastering as much as possible)
    3. Basic C++ or Basic Java
    And the story begins here. As i search on the net, I have found that
    because of the natural characteristics of python such as GIL, we are
    not able to write multi threaded programs. Oooops, in a kind of time
    with lots of cpu cores and we are not able to write multi threaded
    programs. That is out of fashion. How a such powerful language doesn't
    support multi threading. That is a big minus for python. But there is
    something interesting, something like multi processing. But is it a
    real alternative for multi threading. As i searched it is not, it
    requires heavy hardware requirements (lots of memory, lots of cpu
    power). Also it is not easy to implement, too much extra code...

    After all of that, i start to think about omiting python from my
    carrier path and directly choosing c++ or java. But i know google or
    youtube uses python very much. How can they choose a language which
    will be killed by multi threading a time in near future. I like python
    and its syntax, its flexibility.

    What do you think about multi threading and its effect on python. Why
    does python have such a break and what is the fix. Is it worth to make
    investment of time and money to a language it can not take advantage
    of multi cores?

    Thank you...
    Rushen
    , Feb 16, 2009
    #1
    1. Advertising

  2. On 16 Feb., 10:34, wrote:
    > Hi everybody,
    > I am an engineer. I am trying to improve my software development
    > abilities. I have started programming with ruby. I like it very much
    > but i want to add something more. According to my previous research i
    > have designed a learning path for myself. It's like something below.
    >       1. Ruby (Mastering as much as possible)
    >       2. Python (Mastering as much as possible)
    >       3. Basic C++ or Basic Java
    > And the story begins here. As i search on the net,  I have found that
    > because of the natural characteristics of python such as GIL, we are
    > not able to write multi threaded programs. Oooops, in a kind of time
    > with lots of cpu cores and we are not able to write multi threaded
    > programs. That is out of fashion. How a such powerful language doesn't
    > support multi threading. That is a big minus for python.


    On comp.lang.ruby David Masover wrote this at 29 Jul. 2008, 07:55:
    -----
    Right now, Ruby shares a problem with Python called the GIL -- the
    Global (or
    Giant) Interpreter Lock. What this means is that only one Ruby
    instruction
    may execute at a time. So even though they're using separate OS
    threads, and
    even though different Ruby threads might run on different cores, the
    speed of
    your program (at least the Ruby part) is limited to the speed of a
    single
    core.
    -----

    Please don't mix threads and parallel processing on more then one CPU
    core.

    Andreas
    Andreas Kaiser, Feb 16, 2009
    #2
    1. Advertising

  3. Hi, Rushen!

    I'm also new to using Python but from what I've found, GIL is very
    intentional decision. It is one of the features of Python which make
    it so powerful. I believe that if it didn't have GIL, Python wouldn't
    be half near where it is now (regarding it as a language, community,
    platform support, popularity, ...).

    The most important question is do you really need multi-threading for
    what you do? There is lot of software which doesn't require mt at all,
    or could be written without mt. Also, if you haven't learnt C++ or
    Java yet, mt is not something you should be worried about in the near
    future - there are lot of other, more important things, you need to
    learn before opening door mt hell :)

    Best,
    Aleksa


    On Mon, Feb 16, 2009 at 10:34, <> wrote:
    > Hi everybody,
    > I am an engineer. I am trying to improve my software development
    > abilities. I have started programming with ruby. I like it very much
    > but i want to add something more. According to my previous research i
    > have designed a learning path for myself. It's like something below.
    > 1. Ruby (Mastering as much as possible)
    > 2. Python (Mastering as much as possible)
    > 3. Basic C++ or Basic Java
    > And the story begins here. As i search on the net, I have found that
    > because of the natural characteristics of python such as GIL, we are
    > not able to write multi threaded programs. Oooops, in a kind of time
    > with lots of cpu cores and we are not able to write multi threaded
    > programs. That is out of fashion. How a such powerful language doesn't
    > support multi threading. That is a big minus for python. But there is
    > something interesting, something like multi processing. But is it a
    > real alternative for multi threading. As i searched it is not, it
    > requires heavy hardware requirements (lots of memory, lots of cpu
    > power). Also it is not easy to implement, too much extra code...
    >
    > After all of that, i start to think about omiting python from my
    > carrier path and directly choosing c++ or java. But i know google or
    > youtube uses python very much. How can they choose a language which
    > will be killed by multi threading a time in near future. I like python
    > and its syntax, its flexibility.
    >
    > What do you think about multi threading and its effect on python. Why
    > does python have such a break and what is the fix. Is it worth to make
    > investment of time and money to a language it can not take advantage
    > of multi cores?
    >
    > Thank you...
    > Rushen
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >




    --
    Aleksa Todorovic - Lead Programmer
    Eipix Entertainment
    http://www.eipix.com/
    Aleksa Todorovic, Feb 16, 2009
    #3
  4. On Feb 16, 10:34 am, wrote:
    > Hi everybody,
    > I am an engineer. I am trying to improve my software development
    > abilities. I have started programming with ruby. I like it very much
    > but i want to add something more. According to my previous research i
    > have designed a learning path for myself. It's like something below.
    >       1. Ruby (Mastering as much as possible)
    >       2. Python (Mastering as much as possible)
    >       3. Basic C++ or Basic Java
    > And the story begins here. As i search on the net,  I have found that
    > because of the natural characteristics of python such as GIL, we are
    > not able to write multi threaded programs. Oooops, in a kind of time
    > with lots of cpu cores and we are not able to write multi threaded
    > programs. That is out of fashion. How a such powerful language doesn't
    > support multi threading. That is a big minus for python. But there is
    > something interesting, something like multi processing. But is it a
    > real alternative for multi threading. As i searched it is not, it
    > requires heavy hardware requirements (lots of memory, lots of cpu
    > power). Also it is not easy to implement, too much extra code...


    multiprocessing is already implemented for you in the standard
    library.
    Of course it does not require heavy hardware requirements.

    > After all of that, i start to think about omiting python from my
    > carrier path and directly choosing c++ or java. But i know google or
    > youtube uses python very much. How can they choose a language which
    > will be killed by multi threading a time in near future. I like python
    > and its syntax, its flexibility.
    >
    > What do you think about multi threading and its effect on python. Why
    > does python have such a break and what is the fix. Is it worth to make
    > investment of time and money to a language it can not take advantage
    > of multi cores?


    You can take advantage of multi cores, just not with threads but with
    processes,
    which BTW is the right way to go in most situations. So (assuming you
    are not
    a troll) you are just mistaken in thinking that the only way to
    use multicores is via multithreading.

    Michele Simionato
    Michele Simionato, Feb 16, 2009
    #4
  5. Tim Rowe Guest

    2009/2/16 <>:
    > Hi everybody,
    > I am an engineer. I am trying to improve my software development
    > abilities. I have started programming with ruby. I like it very much
    > but i want to add something more. According to my previous research i
    > have designed a learning path for myself. It's like something below.
    > 1. Ruby (Mastering as much as possible)
    > 2. Python (Mastering as much as possible)
    > 3. Basic C++ or Basic Java
    > And the story begins here. As i search on the net, I have found that
    > because of the natural characteristics of python such as GIL, we are
    > not able to write multi threaded programs. Oooops, in a kind of time
    > with lots of cpu cores and we are not able to write multi threaded
    > programs. That is out of fashion. How a such powerful language doesn't
    > support multi threading. That is a big minus for python.


    In a way, you've answered your own question. Why are you learning
    three languages? Perhaps you've already realised that being
    Turing-complete isn't the last word in language choice; that different
    languages make different compromises, so the best language for one
    task may not be the best language for a different task. Ok, Python
    doesn't cope well with threading. It doesn't cope well with hard
    real-time, either. So what? A deep saucepan isn't much use for making
    an omlette (it can be done, but it's inefficient), but it's ideal for
    making soup. Just because multiple cores are available doesn't mean
    they will help your program significantly. Most real-world programs
    work just fine in a single core, and it usually just isn't worth all
    of the extra design effort, coding effort and debugging effort of
    threading to cut the user response time down from a tenth of a second
    to a twentieth. For *most* applications the single-thread Python
    programmer will have something written, shipped and doing the job
    whilst the multi-thread programmer is still trying to debug an
    intermittent livelock that goes away whenever instrumentation is
    added. For those few cases where threading is a genuine advantage,
    Python is not ideal. But in the real world I doubt they're enough to
    make a significant dent in Python's popularity.

    --
    Tim Rowe
    Tim Rowe, Feb 16, 2009
    #5
  6. Guest

    Hi again,

    Dear Andreas

    I know about GIL in ruby interpreter, they are trying to solve
    problems because of GIL but it is not so important for me because i
    like ruby because of its esthetic and it helps me to grasp some
    programming concepts. As i know it is not so powerful language like
    java. (Powerful language : rich libraries, wide community, very
    variety of usage). why i want to learn python because it has syntax
    like ruby but it is also powerful language, not of course as much as
    java or c++

    And also, i am trying to find an answer to the question mark in my
    head. Is there a way to use multi cores in python as much effective
    and easier as multi threading.

    Dear Aleksa

    Of course i have a long way to ago even if i am too old for these
    things. However, i think that being able to use both cores or more is
    a very big plus for execution time, speed like these. I believe many
    programs will be rewritten to be able to use multiple cores.

    Dear Michele

    As i know every process has own hardware sources like memory and cpu
    so i think it requires more source than multi threading.

    Dear Tim

    I want to learn python + c++ or java because of the desire of having
    python's felxibility and easiness and c++ or java's stability and
    speed and power together.

    Thank you
    Rushen
    , Feb 16, 2009
    #6
  7. Tim Rowe Guest

    2009/2/16 <>:

    > I want to learn python + c++ or java because of the desire of having
    > python's felxibility and easiness and c++ or java's stability and
    > speed and power together.


    Yes, that's what I mean by different tradeoffs. Python is much easier
    to program in than C++ or Java (in my experience, at least), but C++
    and Java scale better and at least have the potential to be faster.
    I'm not convinced that library support is significantly better for C++
    or Java -- Python's libraries seem pretty rich to me. And that extra
    speed might not be needed as often as you think. My postgrad
    dissertation involved heavy number-crunching on large data sets, and
    in my proposal I said I'd switch from Python to C++ when Python got
    too slow. In fact, Python never did get too slow (I didn't even have
    to switch to numpy), and plugging together ad-hoc modules, defined in
    an XML script, was a dream in Python when I'd probably still be coding
    it today in C++. Horses for courses. It's almost always wrong to say
    that language A is better than language B; the most you can say is
    that language A is better than language B for some specific task.


    --
    Tim Rowe
    Tim Rowe, Feb 16, 2009
    #7
  8. Guest

    Dear Andrew,

    I think reading "beating the averages" by paul graham before some
    experience is not a very good decision.
    :)

    Thank you
    Andrew
    , Feb 16, 2009
    #8
  9. Or the other way around :)

    Little off-topic, but...

    After several months of fighting with Java threads, dead locks, live
    locks, race conditions, I've rewritten my game server synchronization
    so that threads are executed in concurrent way (with only exceptions
    being socket sending and recieving threads), and all synchronization
    is implemented on top of Java blocking structures (little blasphemy:
    the way Stackless does it). After that change, the game suddenly
    started working like a charm!


    On Mon, Feb 16, 2009 at 13:24, <> wrote:
    > Of course i have a long way to ago even if i am too old for these
    > things. However, i think that being able to use both cores or more is
    > a very big plus for execution time, speed like these. I believe many
    > programs will be rewritten to be able to use multiple cores.




    --
    Aleksa Todorovic - Lead Programmer
    Eipix Entertainment
    http://www.eipix.com/
    Aleksa Todorovic, Feb 16, 2009
    #9
  10. Guest

    Dear Aleksa,

    As you mentioned, using multi cores makes programs more fast and more
    popular. But what about stackless python? Does it interpret same set
    of python libraries with Cpython or Does it have a special sub set?

    Thank you
    Rusen
    , Feb 16, 2009
    #10
  11. "andrew cooke" <andrew@a....org> wrote:


    >The GIL is an implementation detail. I suspect that it could be largely
    >removed if there was sufficient need. But that still wouldn't make Python
    >a good language for programming on multiple cores. That's not as big a
    >deal as you think, because we currently DON'T KNOW what would make a good
    >language for programming on multiple cores - it's an open topic in the
    >wider community.


    Those who do not study history are doomed to repeat it.

    Occam was the language that should have won the marketing
    prize, but didn't.

    - Hendrik
    Hendrik van Rooyen, Feb 16, 2009
    #11
  12. Hi,

    > But there is
    > something interesting, something like multi processing. But is it a
    > real alternative for multi threading. As i searched it is not, it
    > requires heavy hardware requirements (lots of memory, lots of cpu
    > power).


    Not necessarily.

    For the memory, modern operating Systems can ustilize a copy on write
    semantics in their Applications virtual memory space. That means that after
    starting a new process with fork(), the new Process shares all physical
    Memory Pages with the original one as long a none of them writes to wome
    Memory. The first write is cought by the OS. Then the Page will be copied
    and the writing process will in future use the new copy of that page while
    the other one (or eventually many) stay with the original. Multiprocessing
    needs more Memory than well optimized Multithreading, but it is not as bad
    as it is often said.

    For the CPU: There are two things that make the difference here.

    One ist Context Switching. When the OS switches between two threads of the
    same Process, it does not need to change the memory lookup table. When
    switching between Processes that is always necessary. Switching from a
    Thread of Process A to a Thread of Process B is just like switching from A
    to B.

    Using Threads just increases the probability that the OS can do a faster
    Context Switch. So Threads are faster here, but it is not a too big issue as
    well.

    The Second Thing is Communication. With Threds you can simply use variables
    to communicate between Threads. To Prevent conflicting access to those
    Variables you will use e.g. mutexes. That can be done with Shared Memory as
    well. Shared Memory is a bit more difficult to handle than simply using the
    heap, but again the impact is not really so big.

    Google uses Processes for the Tabs in Chrome, because that way they get
    around many Memory Management Problems they would have with Threads or with
    a singlethreaded reactor. Using Processes is not per se a bad Idea. You pay
    a bit with Memory and CPU but in many situations you get a much simpler
    programming model.

    Christof
    Christof Donat, Feb 16, 2009
    #12
  13. Guest

    hi there,

    [snip]
    > Google uses Processes for the Tabs in Chrome, because that way they get
    > around many Memory Management Problems they would have with Threads or with
    > a singlethreaded reactor. Using Processes is not per se a bad Idea. You pay
    > a bit with Memory and CPU but in many situations you get a much simpler
    > programming model.


    note also that one can bet on things like KSM [1] to further improve
    the memory situation.

    For example, we are investigating its use in the large (read heavy)
    application frameworks at CERN where the typical vmem footprint is
    ~2Gb.
    Running KSM together with a multiprocessing-based event loop manager,
    we
    managed to overcommit ~300% of the memory (ie: run 3 instances of our
    application on a 2Gb-per-core machine)

    cheers,
    sebastien

    [1] http://lwn.net/Articles/306642
    , Feb 16, 2009
    #13
  14. Guest

    Hi again

    OpenERP and ERP5 was written in python as i know. I really wonder how
    they do this without threads. I want to see a real time graph at the
    same time while i am working on the same screen. What is the secret?

    Thanks
    Rushen
    , Feb 16, 2009
    #14
  15. Aahz Guest

    In article <>,
    Hendrik van Rooyen <> wrote:
    >
    >Occam was the language that should have won the marketing prize, but
    >didn't.


    It wasn't simple enough.
    --
    Aahz () <*> http://www.pythoncraft.com/

    Weinberg's Second Law: If builders built buildings the way programmers wrote
    programs, then the first woodpecker that came along would destroy civilization.
    Aahz, Feb 17, 2009
    #15
  16. On Feb 16, 10:20 pm, wrote:
    > Hi again
    >
    > OpenERP and ERP5 was written in python as i know. I really wonder how
    > they do this without threads. I want to see a real time graph at the
    > same time while i am working on the same screen. What is the secret?
    >
    > Thanks
    > Rushen


    Here is an example of using multiprocessing (which is included
    in Python 2.6 and easy_installable in older Python versions)
    to print a spin bar while a computation is running:

    import sys, time
    import multiprocessing

    DELAY = 0.1
    DISPLAY = [ '|', '/', '-', '\\' ]

    def spinner_func(before='', after=''):
    write, flush = sys.stdout.write, sys.stdout.flush
    pos = -1
    while True:
    pos = (pos + 1) % len(DISPLAY)
    msg = before + DISPLAY[pos] + after
    write(msg); flush()
    write('\x08' * len(msg))
    time.sleep(DELAY)

    def long_computation():
    # emulate a long computation
    time.sleep(3)

    if __name__ == '__main__':
    spinner = multiprocessing.Process(
    None, spinner_func, args=('Please wait ... ', ''))
    spinner.start()
    try:
    long_computation()
    print 'Computation done'
    finally:
    spinner.terminate()
    Michele Simionato, Feb 17, 2009
    #16
  17. Paddy Guest

    On Feb 16, 9:34 am, wrote:
    > Hi everybody,
    > I am an engineer. I am trying to improve my software development
    > abilities. I have started programming with ruby. I like it very much
    > but i want to add something more. According to my previous research i
    > have designed a learning path for myself. It's like something below.
    >       1. Ruby (Mastering as much as possible)
    >       2. Python (Mastering as much as possible)
    >       3. Basic C++ or Basic Java
    > And the story begins here. As i search on the net,  I have found that
    > because of the natural characteristics of python such as GIL, we are
    > not able to write multi threaded programs.


    You are likely to find a lot of 'tick-list' type of comparison data on
    the web that either needs a lot of knowledge to interpret, or is
    misleading/wrong or all three!

    Their are several versions of Python out their such as Ironpython,
    Stackless Python, Jython as well as CPython - the main Python release.
    They have different threading capabilities, but compilers of feature
    comparison tick-lists tend to just stick to what CPython can do.

    As an aside; if you were thinking of using threading for performance
    reasons, then its best to first think of improving your general
    ability to explore different algorithms. A change to an algorithm
    often has the most impact on the performance of code. A better single
    threaded, single process algorithm can offer better performaance than
    throwing threadds or multiple processes alone when using a poor
    underlying algorithm.

    I was just exploring different ways of solving a problem on my blog:
    http://paddy3118.blogspot.com/2009/02/comparison-of-python-solutions-to.html
    (But no parallel solutions were attempted).

    Have fun programming!

    - Paddy.
    Paddy, Feb 17, 2009
    #17
  18. "Aahz" <aahz@py....aft.com> wrote:


    > In article <>,
    > Hendrik van Rooyen <ma....orp.co.za> wrote:
    > >
    > >Occam was the language that should have won the marketing prize, but
    > >didn't.

    >
    > It wasn't simple enough.


    I thought (at the time) that it was quite good at hiding some
    horrible complexities of communication between different
    processes on the same, and different processors.

    All done by the compiler, automagically.

    I think now that a hard look at the underlying techniques
    used then could add something to the debate referred to
    earlier - but there may be a barrier because the dataflow
    or systolic array type programming model is not one
    that is currently fashionable.

    - Hendrik
    Hendrik van Rooyen, Feb 17, 2009
    #18
  19. "Ben Finney" <> wrote:

    (Aahz) writes:

    > In article <>,
    > Hendrik van Rooyen <> wrote:
    > >Occam was the language that should have won the marketing prize,
    > >but didn't.

    >
    > It wasn't simple enough.


    *bdom-tsssh* <URL:http://en.wikipedia.org/wiki/Occam's_razor>

    If Aahz was trolling, then he got me. I know about William of Occam,
    after whom the language was named, and his razor, but did not make
    the association, and answered seriously.

    If you play with razor blades, you get cut.
    :)

    - Hendrik
    Hendrik van Rooyen, Feb 17, 2009
    #19
  20. a écrit :
    (snip)
    > And the story begins here. As i search on the net, I have found that
    > because of the natural characteristics of python such as GIL, we are
    > not able to write multi threaded programs.


    I'm surprised no one here corrected that point yet, so here we go: yes,
    Python does support multithreading. The fact that threads won't be
    executed concurrently on a multicore machine (due to the GIL) is a
    different problem (cf Andrew Cooke's answer on this - and his advice to
    study Erlang if you want to do concurrent programming).
    Bruno Desthuilliers, Feb 17, 2009
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Brett C.
    Replies:
    11
    Views:
    1,123
    Greg Ewing
    Aug 13, 2004
  2. nitesh
    Replies:
    6
    Views:
    338
    DocuMaker
    Dec 10, 2007
  3. jiajia wu
    Replies:
    0
    Views:
    355
    jiajia wu
    Oct 1, 2009
  4. 6668
    Replies:
    0
    Views:
    147
  5. lllll
    Replies:
    0
    Views:
    130
    lllll
    Jun 8, 2009
Loading...

Share This Page