Python is slow

[David Cournapeau]

It has been mentioned in this thread the pypy project (isn't it enough
for you??)

Since pypy can't be used today for most production use (most python
packages can't work on it), I don't see how it could be enough for
anyone interested in solving problems today. I want faster function
calls to use with numpy: do you know of any solution ? Pypy certainly
isn't, at least today.

cheers,

David

 

[sturlamolden]

I want faster function
calls to use with numpy: do you know of any solution ? Pypy certainly
isn't, at least today.

An interesting thing for numpy would be to use CUDA. If we can move
floating point ops to the GPU, a common desktop computer could yield
teraflops. A subclass of ndarray could be written for the nvidia GPU.

Using OpenMP within NumPy would also be interesting. There are desktop
computers available today with two quadcore processors.

There is multiprocessing, which works nicely with numpy. You can even
have multiple processes working on ndarrys that point to the same
shared memory. Just allocate a multiprocessing.Array and use its
buffer to create ndarray views.

 

[Stefan Behnel]

I want faster function
calls to use with numpy: do you know of any solution ?

http://cython.org/

Stefan

 

[sturlamolden]

http://cython.org/

How is the numpy support in Cython going? It was supposed to know
about ndarrays natively. I.e. not treat them as Python objects, but
rather as known C structs. That way an operation like arr[n] would not
result in a callback to Python, but translate directly to fast pointer
arithmetics.

 

[bearophileHUGS]

sturlamolden:

On a recent benchmark Java 6 -server beats C compiled by GCC 4.2.3 And
most of that magic comes from an implementation of a dynamically typed
language (Smalltalk). [...]
http://shootout.alioth.debian.org/u32q/benchmark.php?test=all〈=all

That is indeed a nice result, JavaVM has come a long way from the
first one used for applets. That result comes mostly from the fact
that this is a test on a 4-core CPU, that is less easy to manage from
C. You can see that in the single 64-bit core tests:
http://shootout.alioth.debian.org/u64/benchmark.php?test=all&lang=all
And take a look at the memory used too, up to 34 times higher for the
JVM on the 4-core CPU.

In the next years people that use low-level languages like C may need
to invent a new language fitter for multi-core CPUs, able to be used
on GPUs too (see the OpenCL), less error-prone than C, able to use the
CPU vector instructions efficiently. (The D language is probably unfit
for this purpose, because even if it's meant to be a system language,
I don't think it can be used much to replace C everywhere it's used
now.) A C+ maybe?

I agree that CPython may quite enjoy having something built-in like
Psyco, but it's a lot of work for an open source project. Probably
with 1/3 or 1/2 of the work poured on PyPy you may create that
improvement for CPython. Maybe PyPy will someday produce some fruit,
but I think they have used the wrong strategy: instead of trying to
create something very new that someday will work, it's often better to
try to improve something that today everybody uses, AND try to be
useful from almost the very beginning.

Bye,
bearophile

 

[MRAB]

Nothing is being done, and woth Py3k it got even worse.


None of those projects addresses inefficacies in the CPython
interpreter, except for psyco - which died of an overdose PyPy.

PyPy is interesting if they ever will be able to produce something
useful. They have yet to prove that. Even if PyPy can come up with a
Python JIT, they will still be decades behind the technologies of
Strongtalk and Java. That is the problem with reinventing the wheel
all over again.

Not to forget LLVM and Parrot which also will support Python
frontends.

Python is developed and maintained by volunteers. If you'd like to have
a go at writing a JIT interpreter for it, then go ahead. No-one here
will stop you.

 

[Stefan Behnel]

How is the numpy support in Cython going? It was supposed to know
about ndarrays natively.

It does.

I.e. not treat them as Python objects, but
rather as known C structs. That way an operation like arr[n] would not
result in a callback to Python, but translate directly to fast pointer
arithmetics.

http://docs.cython.org/docs/numpy_tutorial.html

Stefan

 

[Andreas Kostyrka]

None of those projects addresses inefficacies in the CPython
interpreter, except for psyco - which died of an overdose PyPy.

Bullshit. All that discussion about performance forgets that performance is a function of the whole system, not the language.

Worse you can measure it really badly.

E.g. it's relative simple to compare CPython versus IronPython versus Jython. For a given benchmark program.

As programs do not trivially translate from language A to language B, nor does fluency in language A make you automatically fluent in
language B after learning the syntax.

PyPy is interesting if they ever will be able to produce something
useful. They have yet to prove that. Even if PyPy can come up with a
Python JIT, they will still be decades behind the technologies of
Strongtalk and Java. That is the problem with reinventing the wheel
all over again.

Well, it's reinventing the wheel. The problem that Java is a different kind of wheel
(boxed vs. unboxed objects, plus more static compile time bindings), Smalltalk is also different (e.g. multiple inheritence),
so you need to have a specific toolbox for the wheel, sorry. Keeping and enhancing the tribal wisdom
about toolbox design is what a subtribe of the Computer Scientists do.

Btw, Psyco is not a JIT like most JVMs had them, it's a specializing compiler. JVM JITs traditionally speeded up the unboxed data
type operations. Psyco does something comparable, but it has to specialize first on data types. The end effect is similiar, but the
background of what happens is quite different.

Not to forget LLVM and Parrot which also will support Python
frontends.

When they do, they'll do. There have flown quite a bit of Python version since the time that it was announced that
Parrot will have a Python frontend.

Andreas

 

[Luis M. González]

Nothing is being done, and woth Py3k it got even worse.


None of those projects addresses inefficacies in the CPython
interpreter, except for psyco - which died of an overdose PyPy.

PyPy is interesting if they ever will be able to produce something
useful. They have yet to prove that. Even if PyPy can come up with a
Python JIT, they will still be decades behind the technologies of
Strongtalk and Java. That is the problem with reinventing the wheel
all over again.

Not to forget LLVM and Parrot which also will support Python
frontends.

So, what's your conclusion?

 

[Bruno Desthuilliers]

sturlamolden a écrit :
(snip)

Creating a fast implementation of a dynamic language is almost rocket
science. But it has been done. There is Stronghold, the fastest
version of Smalltalk known to man, on which the Sun Java VM is based.
On a recent benchmark Java 6 -server beats C compiled by GCC 4.2.3

cf bearophile's comment on this point (CPU architecture and RAM)

And
most of that magic comes from an implementation of a dynamically typed
language (Smalltalk).

Err... Where is _Java_ "dynamic" actually ? A benchmark of _Smalltalk_
VM vs CPython VM would make more sense.

Second, there are other fast implementations of dynamic languages. The
CMUCL and SBCL versions of Common Lisp comes to min; you can see how
SBCL does in the same benchmark (CMUCL tends to be even faster).

Could it be that there are some type hints in the lisp versions of the
source code ?

So Python is a lot slower than it needs to be.

Please fix it, you're welcome.

 

[Isaac Gouy]

sturlamolden:

On a recent benchmark Java 6 -server beats C compiled by GCC 4.2.3 And
most of that magic comes from an implementation of a dynamically typed
language (Smalltalk). [...]
http://shootout.alioth.debian.org/u32q/benchmark.php?test=all〈=all

That is indeed a nice result, JavaVM has come a long way from the
first one used for applets. That result comes mostly from the fact
that this is a test on a 4-core CPU, that is less easy to manage from
C. You can see that in the single 64-bit core tests:http://shootout.alioth.debian.org/u64/benchmark.php?test=all〈=all

Whether or not it's less easy to manage from C is unclear, but you are
correct to point out few of those C programs have been updated to
exploit quadcore - so the reasonable comparison is with C++.


And the benchmarks game also provides x86 measurements with programs
forced onto a single core which shows GCC ahead

http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=all

 

[Isaac Gouy]

sturlamolden a écrit :
(snip)


cf bearophile's comment on this point (CPU architecture and RAM)


Err... Where is _Java_ "dynamic" actually ? A benchmark of _Smalltalk_
VM vs CPython VM would make more sense.

http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=vw&lang2=python



Could it be that there are some type hints in the lisp versions of the
source code ?


Please fix it, you're welcome.

 

[sturlamolden]

And it is not just this Python site that is slow. There are many many
Python sites which are very slow. And please don’t say that it could
be the web hosting or the server which is slow — because when so many
Python sites are slower than PHP sites, it couldn’t be the web
hosting.   Also, Zope/Plone is even slower.

Python is slow. Very slow.

By the way... I know of a very slow Python site called YouTube.com. In
fact, it is so slow that nobody ever uses it.

 

[cm_gui]

hahaha, do you know how much money they are spending on hardware to
make
youtube.com fast???

 

[James Mills]

Obviously not enough to get to the point where it's cheaper to have the
programmers write C code. And the hardware is more for handling the intense
traffic that YouTube gets, not for speeding up the site.

Seriously cm_gui, you're a fool.
Python is not slow.

--JamesMills

 

[James Mills]

Seriously cm_gui, you're a fool.
Python is not slow.

And I should clarify that by stating
that the CPython interpreter is NOT slow.

--JamesMills

 

[Andreas Kostyrka]

Am Sun, 14 Dec 2008 20:38:58 -0800 (PST)

hahaha, do you know how much money they are spending on hardware to
make
youtube.com fast???

yeah, as they do for basically all big sites, no matter what language
is used for implementation.

Next is the fact that it's rather simple with Python to meet speed
demands where external factors like Gb vs 10Gb network cards are the
limiting factor.

And last, you do realize that most "simple" websites do hinge on the
performance and scalability of the underlying SQL server. In practice
some languages like PHP do force that "LAMP" model much stronger on the
developer, which makes developing systems that scale beyond a certain
point a challenge.

So to summarize, Python is fast enough for even demanding stuff, and
when done correctly even number crunching or binary parsing huge files
or possible in competitive speeds. But you sometime need a developer
that can wield the tool with a certain experience, and not a stupid
rookie that whines that his tool does not make his O(n**n) algorithm
automatically blazing fast.

Andreas

 

[Steven D'Aprano]

hahaha, do you know how much money they are spending on hardware to make
youtube.com fast???

Oooh, I know!

ONE MILLION DOLLARS!!!!

And still cheaper and easier than re-writing YouTube's infrastructure in
another language.

 

[James Mills]

So to summarize, Python is fast enough for even demanding stuff, and
when done correctly even number crunching or binary parsing huge files
or possible in competitive speeds. But you sometime need a developer
that can wield the tool with a certain experience, and not a stupid
rookie that whines that his tool does not make his O(n**n) algorithm
automatically blazing fast.

Amen! +10

--JamesMills

 

[Luis M. González]

hahaha, do you know how much money they are spending on hardware to
make
youtube.com fast???

Buddy, just stop whining and go with c++ if it makes you happy.
By the way, what's the blazingly fast application you need to write so
desperately?
What kind of performance problem have you find in python that makes
you so unhappy?
What are you going to do with all the extra speed provided by c++ (a
Hello World! ?)...