Will multithreading make python less popular?

[rushenaly]

Hi everybody,
I am an engineer. I am trying to improve my software development
abilities. I have started programming with ruby. I like it very much
but i want to add something more. According to my previous research i
have designed a learning path for myself. It's like something below.
1. Ruby (Mastering as much as possible)
2. Python (Mastering as much as possible)
3. Basic C++ or Basic Java
And the story begins here. As i search on the net, I have found that
because of the natural characteristics of python such as GIL, we are
not able to write multi threaded programs. Oooops, in a kind of time
with lots of cpu cores and we are not able to write multi threaded
programs. That is out of fashion. How a such powerful language doesn't
support multi threading. That is a big minus for python. But there is
something interesting, something like multi processing. But is it a
real alternative for multi threading. As i searched it is not, it
requires heavy hardware requirements (lots of memory, lots of cpu
power). Also it is not easy to implement, too much extra code...

After all of that, i start to think about omiting python from my
carrier path and directly choosing c++ or java. But i know google or
youtube uses python very much. How can they choose a language which
will be killed by multi threading a time in near future. I like python
and its syntax, its flexibility.

What do you think about multi threading and its effect on python. Why
does python have such a break and what is the fix. Is it worth to make
investment of time and money to a language it can not take advantage
of multi cores?

Thank you...
Rushen

 

[Andreas Kaiser]

Hi everybody,
I am an engineer. I am trying to improve my software development
abilities. I have started programming with ruby. I like it very much
but i want to add something more. According to my previous research i
have designed a learning path for myself. It's like something below.
      1. Ruby (Mastering as much as possible)
      2. Python (Mastering as much as possible)
      3. Basic C++ or Basic Java
And the story begins here. As i search on the net,  I have found that
because of the natural characteristics of python such as GIL, we are
not able to write multi threaded programs. Oooops, in a kind of time
with lots of cpu cores and we are not able to write multi threaded
programs. That is out of fashion. How a such powerful language doesn't
support multi threading. That is a big minus for python.

On comp.lang.ruby David Masover wrote this at 29 Jul. 2008, 07:55:
-----
Right now, Ruby shares a problem with Python called the GIL -- the
Global (or
Giant) Interpreter Lock. What this means is that only one Ruby
instruction
may execute at a time. So even though they're using separate OS
threads, and
even though different Ruby threads might run on different cores, the
speed of
your program (at least the Ruby part) is limited to the speed of a
single
core.
-----

Please don't mix threads and parallel processing on more then one CPU
core.

Andreas

 

[Aleksa Todorovic]

Hi, Rushen!

I'm also new to using Python but from what I've found, GIL is very
intentional decision. It is one of the features of Python which make
it so powerful. I believe that if it didn't have GIL, Python wouldn't
be half near where it is now (regarding it as a language, community,
platform support, popularity, ...).

The most important question is do you really need multi-threading for
what you do? There is lot of software which doesn't require mt at all,
or could be written without mt. Also, if you haven't learnt C++ or
Java yet, mt is not something you should be worried about in the near
future - there are lot of other, more important things, you need to
learn before opening door mt hell

Best,
Aleksa

 

[Michele Simionato]

Hi everybody,
I am an engineer. I am trying to improve my software development
abilities. I have started programming with ruby. I like it very much
but i want to add something more. According to my previous research i
have designed a learning path for myself. It's like something below.
      1. Ruby (Mastering as much as possible)
      2. Python (Mastering as much as possible)
      3. Basic C++ or Basic Java
And the story begins here. As i search on the net,  I have found that
because of the natural characteristics of python such as GIL, we are
not able to write multi threaded programs. Oooops, in a kind of time
with lots of cpu cores and we are not able to write multi threaded
programs. That is out of fashion. How a such powerful language doesn't
support multi threading. That is a big minus for python. But there is
something interesting, something like multi processing. But is it a
real alternative for multi threading. As i searched it is not, it
requires heavy hardware requirements (lots of memory, lots of cpu
power). Also it is not easy to implement, too much extra code...

multiprocessing is already implemented for you in the standard
library.
Of course it does not require heavy hardware requirements.

After all of that, i start to think about omiting python from my
carrier path and directly choosing c++ or java. But i know google or
youtube uses python very much. How can they choose a language which
will be killed by multi threading a time in near future. I like python
and its syntax, its flexibility.

What do you think about multi threading and its effect on python. Why
does python have such a break and what is the fix. Is it worth to make
investment of time and money to a language it can not take advantage
of multi cores?

You can take advantage of multi cores, just not with threads but with
processes,
which BTW is the right way to go in most situations. So (assuming you
are not
a troll) you are just mistaken in thinking that the only way to
use multicores is via multithreading.

Michele Simionato

 

[Tim Rowe]

Hi everybody,
I am an engineer. I am trying to improve my software development
abilities. I have started programming with ruby. I like it very much
but i want to add something more. According to my previous research i
have designed a learning path for myself. It's like something below.
1. Ruby (Mastering as much as possible)
2. Python (Mastering as much as possible)
3. Basic C++ or Basic Java
And the story begins here. As i search on the net, I have found that
because of the natural characteristics of python such as GIL, we are
not able to write multi threaded programs. Oooops, in a kind of time
with lots of cpu cores and we are not able to write multi threaded
programs. That is out of fashion. How a such powerful language doesn't
support multi threading. That is a big minus for python.

In a way, you've answered your own question. Why are you learning
three languages? Perhaps you've already realised that being
Turing-complete isn't the last word in language choice; that different
languages make different compromises, so the best language for one
task may not be the best language for a different task. Ok, Python
doesn't cope well with threading. It doesn't cope well with hard
real-time, either. So what? A deep saucepan isn't much use for making
an omlette (it can be done, but it's inefficient), but it's ideal for
making soup. Just because multiple cores are available doesn't mean
they will help your program significantly. Most real-world programs
work just fine in a single core, and it usually just isn't worth all
of the extra design effort, coding effort and debugging effort of
threading to cut the user response time down from a tenth of a second
to a twentieth. For *most* applications the single-thread Python
programmer will have something written, shipped and doing the job
whilst the multi-thread programmer is still trying to debug an
intermittent livelock that goes away whenever instrumentation is
added. For those few cases where threading is a genuine advantage,
Python is not ideal. But in the real world I doubt they're enough to
make a significant dent in Python's popularity.

 

[rushenaly]

Hi again,

Dear Andreas

I know about GIL in ruby interpreter, they are trying to solve
problems because of GIL but it is not so important for me because i
like ruby because of its esthetic and it helps me to grasp some
programming concepts. As i know it is not so powerful language like
java. (Powerful language : rich libraries, wide community, very
variety of usage). why i want to learn python because it has syntax
like ruby but it is also powerful language, not of course as much as
java or c++

And also, i am trying to find an answer to the question mark in my
head. Is there a way to use multi cores in python as much effective
and easier as multi threading.

Dear Aleksa

Of course i have a long way to ago even if i am too old for these
things. However, i think that being able to use both cores or more is
a very big plus for execution time, speed like these. I believe many
programs will be rewritten to be able to use multiple cores.

Dear Michele

As i know every process has own hardware sources like memory and cpu
so i think it requires more source than multi threading.

Dear Tim

I want to learn python + c++ or java because of the desire of having
python's felxibility and easiness and c++ or java's stability and
speed and power together.

Thank you
Rushen

 

[Tim Rowe]

I want to learn python + c++ or java because of the desire of having
python's felxibility and easiness and c++ or java's stability and
speed and power together.

Yes, that's what I mean by different tradeoffs. Python is much easier
to program in than C++ or Java (in my experience, at least), but C++
and Java scale better and at least have the potential to be faster.
I'm not convinced that library support is significantly better for C++
or Java -- Python's libraries seem pretty rich to me. And that extra
speed might not be needed as often as you think. My postgrad
dissertation involved heavy number-crunching on large data sets, and
in my proposal I said I'd switch from Python to C++ when Python got
too slow. In fact, Python never did get too slow (I didn't even have
to switch to numpy), and plugging together ad-hoc modules, defined in
an XML script, was a dream in Python when I'd probably still be coding
it today in C++. Horses for courses. It's almost always wrong to say
that language A is better than language B; the most you can say is
that language A is better than language B for some specific task.

 

[rushenaly]

Dear Andrew,

I think reading "beating the averages" by paul graham before some
experience is not a very good decision.


Thank you
Andrew

 

[Aleksa Todorovic]

Or the other way around

Little off-topic, but...

After several months of fighting with Java threads, dead locks, live
locks, race conditions, I've rewritten my game server synchronization
so that threads are executed in concurrent way (with only exceptions
being socket sending and recieving threads), and all synchronization
is implemented on top of Java blocking structures (little blasphemy:
the way Stackless does it). After that change, the game suddenly
started working like a charm!

 

[rushenaly]

Dear Aleksa,

As you mentioned, using multi cores makes programs more fast and more
popular. But what about stackless python? Does it interpret same set
of python libraries with Cpython or Does it have a special sub set?

Thank you
Rusen

 

[Hendrik van Rooyen]

The GIL is an implementation detail. I suspect that it could be largely
removed if there was sufficient need. But that still wouldn't make Python
a good language for programming on multiple cores. That's not as big a
deal as you think, because we currently DON'T KNOW what would make a good
language for programming on multiple cores - it's an open topic in the
wider community.

Those who do not study history are doomed to repeat it.

Occam was the language that should have won the marketing
prize, but didn't.

- Hendrik

 

[Christof Donat]

Hi,

But there is
something interesting, something like multi processing. But is it a
real alternative for multi threading. As i searched it is not, it
requires heavy hardware requirements (lots of memory, lots of cpu
power).

Not necessarily.

For the memory, modern operating Systems can ustilize a copy on write
semantics in their Applications virtual memory space. That means that after
starting a new process with fork(), the new Process shares all physical
Memory Pages with the original one as long a none of them writes to wome
Memory. The first write is cought by the OS. Then the Page will be copied
and the writing process will in future use the new copy of that page while
the other one (or eventually many) stay with the original. Multiprocessing
needs more Memory than well optimized Multithreading, but it is not as bad
as it is often said.

For the CPU: There are two things that make the difference here.

One ist Context Switching. When the OS switches between two threads of the
same Process, it does not need to change the memory lookup table. When
switching between Processes that is always necessary. Switching from a
Thread of Process A to a Thread of Process B is just like switching from A
to B.

Using Threads just increases the probability that the OS can do a faster
Context Switch. So Threads are faster here, but it is not a too big issue as
well.

The Second Thing is Communication. With Threds you can simply use variables
to communicate between Threads. To Prevent conflicting access to those
Variables you will use e.g. mutexes. That can be done with Shared Memory as
well. Shared Memory is a bit more difficult to handle than simply using the
heap, but again the impact is not really so big.

Google uses Processes for the Tabs in Chrome, because that way they get
around many Memory Management Problems they would have with Threads or with
a singlethreaded reactor. Using Processes is not per se a bad Idea. You pay
a bit with Memory and CPU but in many situations you get a much simpler
programming model.

Christof

 

[seb.binet]

hi there,

[snip]

Google uses Processes for the Tabs in Chrome, because that way they get
around many Memory Management Problems they would have with Threads or with
a singlethreaded reactor. Using Processes is not per se a bad Idea. You pay
a bit with Memory and CPU but in many situations you get a much simpler
programming model.

note also that one can bet on things like KSM [1] to further improve
the memory situation.

For example, we are investigating its use in the large (read heavy)
application frameworks at CERN where the typical vmem footprint is
~2Gb.
Running KSM together with a multiprocessing-based event loop manager,
we
managed to overcommit ~300% of the memory (ie: run 3 instances of our
application on a 2Gb-per-core machine)

cheers,
sebastien

[1] http://lwn.net/Articles/306642

 

[rushenaly]

Hi again

OpenERP and ERP5 was written in python as i know. I really wonder how
they do this without threads. I want to see a real time graph at the
same time while i am working on the same screen. What is the secret?

Thanks
Rushen

 

[Aahz]

Occam was the language that should have won the marketing prize, but
didn't.

It wasn't simple enough.

 

[Michele Simionato]

Hi again

OpenERP and ERP5 was written in python as i know. I really wonder how
they do this without threads. I want to see a real time graph at the
same time while i am working on the same screen. What is the secret?

Thanks
Rushen

Here is an example of using multiprocessing (which is included
in Python 2.6 and easy_installable in older Python versions)
to print a spin bar while a computation is running:

import sys, time
import multiprocessing

DELAY = 0.1
DISPLAY = [ '|', '/', '-', '\\' ]

def spinner_func(before='', after=''):
write, flush = sys.stdout.write, sys.stdout.flush
pos = -1
while True:
pos = (pos + 1) % len(DISPLAY)
msg = before + DISPLAY[pos] + after
write(msg); flush()
write('\x08' * len(msg))
time.sleep(DELAY)

def long_computation():
# emulate a long computation
time.sleep(3)

if __name__ == '__main__':
spinner = multiprocessing.Process(
None, spinner_func, args=('Please wait ... ', ''))
spinner.start()
try:
long_computation()
print 'Computation done'
finally:
spinner.terminate()

 

[Paddy]

Hi everybody,
I am an engineer. I am trying to improve my software development
abilities. I have started programming with ruby. I like it very much
but i want to add something more. According to my previous research i
have designed a learning path for myself. It's like something below.
      1. Ruby (Mastering as much as possible)
      2. Python (Mastering as much as possible)
      3. Basic C++ or Basic Java
And the story begins here. As i search on the net,  I have found that
because of the natural characteristics of python such as GIL, we are
not able to write multi threaded programs.

You are likely to find a lot of 'tick-list' type of comparison data on
the web that either needs a lot of knowledge to interpret, or is
misleading/wrong or all three!

Their are several versions of Python out their such as Ironpython,
Stackless Python, Jython as well as CPython - the main Python release.
They have different threading capabilities, but compilers of feature
comparison tick-lists tend to just stick to what CPython can do.

As an aside; if you were thinking of using threading for performance
reasons, then its best to first think of improving your general
ability to explore different algorithms. A change to an algorithm
often has the most impact on the performance of code. A better single
threaded, single process algorithm can offer better performaance than
throwing threadds or multiple processes alone when using a poor
underlying algorithm.

I was just exploring different ways of solving a problem on my blog:
http://paddy3118.blogspot.com/2009/02/comparison-of-python-solutions-to.html
(But no parallel solutions were attempted).

Have fun programming!

- Paddy.

 

[Hendrik van Rooyen]

It wasn't simple enough.

I thought (at the time) that it was quite good at hiding some
horrible complexities of communication between different
processes on the same, and different processors.

All done by the compiler, automagically.

I think now that a hard look at the underlying techniques
used then could add something to the debate referred to
earlier - but there may be a barrier because the dataflow
or systolic array type programming model is not one
that is currently fashionable.

- Hendrik

 

[Hendrik van Rooyen]

It wasn't simple enough.

*bdom-tsssh* <URL:http://en.wikipedia.org/wiki/Occam's_razor>

If Aahz was trolling, then he got me. I know about William of Occam,
after whom the language was named, and his razor, but did not make
the association, and answered seriously.

If you play with razor blades, you get cut.


- Hendrik

 

[Bruno Desthuilliers]

(e-mail address removed) a écrit :
(snip)

And the story begins here. As i search on the net, I have found that
because of the natural characteristics of python such as GIL, we are
not able to write multi threaded programs.

I'm surprised no one here corrected that point yet, so here we go: yes,
Python does support multithreading. The fact that threads won't be
executed concurrently on a multicore machine (due to the GIL) is a
different problem (cf Andrew Cooke's answer on this - and his advice to
study Erlang if you want to do concurrent programming).