Help me choose a C++ compiler to work with Python

[rhmd]

Just found Python and I love it. What an elegant language!
I would like to use it for various applications, but the
mathematical calculations are way too slow (a million sines 8 seconds
in Python vs. 2 seconds in Excel VBA), so was thinking of learning
enough C++ to do the number crunching in C++ and integrating the C++
routines with Python. My question:
My question:
Which compiler works best with Python?
I use Windows XP and 98. Have very little experience with C++. Am
doing statistical and other simulations that require billions of
calculations (taking 1 to 2 hours in Excel VBA). GCC is free, MS
C++.Net is affordable at <$100 but supposedly has problems until the
next version (and I could only use it on my XP machine), older
versions of MS C++ are no longer for sale. What's the most robust
solution? I would also like to make .dll files that I can plug into
MS Excel.
Sorry for the vagueness of this question, but I'm sure many of you
have experience with this and any information, opinions, and even
prejudices about the various compilers are welcome.
Thanks.

 

[Peter Hansen]

I use Windows XP and 98. Have very little experience with C++. Am
doing statistical and other simulations that require billions of
calculations

Have you considered and investigated Numeric Python or similar
products? Perhaps your assumption that you have to revert to
hand-crafted C code is premature.

-Peter

 

[Cy Edmunds]

Just found Python and I love it. What an elegant language!

Yeah, me too.

I would like to use it for various applications, but the
mathematical calculations are way too slow (a million sines 8 seconds
in Python vs. 2 seconds in Excel VBA), so was thinking of learning
enough C++ to do the number crunching in C++ and integrating the C++
routines with Python.

This one really has me squirming in my seat. C++ isn't a very good language
for learning just a little bit of. Compared to Python I would say the
learning curve is much steeper. It offers a lot of control over how the
object code is generated but that also makes it dangerous for casual users.
If you do it wrong a lot of bad things can happen, including poor
performance -- although poor performance would perhaps be the best of those
bad things!

I suggest you look at various Python libraries to do whatever numerical
processing you might need to do. They are mostly written in C anyway so
their performance is excellent. For instance, I use SciPy for Fourier
analysis and PIL for image processing, getting good performance on these
numerically intensive tasks without ever leaving the ease and comfort of
good old Python.

My question:

My question:
Which compiler works best with Python?

I wouldn't choose your C++ compiler for Python compatibility. How good is
the debugger? How is the standards conformance? C++ compilers vary widely in
these and other important criteria.

I use Windows XP and 98. Have very little experience with C++. Am
doing statistical and other simulations that require billions of
calculations (taking 1 to 2 hours in Excel VBA).

Take a look at the UVS library under my sig for some C++ code which I have
found useful for statistical simulations.

GCC is free, MS

C++.Net is affordable at <$100 but supposedly has problems until the
next version (and I could only use it on my XP machine), older
versions of MS C++ are no longer for sale. What's the most robust
solution?

The .NET compiler has a very good debugger and much better standard
compliance than the old 6.0 compiler, but I haven't used lot of different
recent vintage C++ compilers.

I would also like to make .dll files that I can plug into

MS Excel.

I generally use COM objects written in Python to connect to Excel. I am very
pleased with that technique.

http://starship.python.net/crew/mhammond/

 

[John J. Lee]

news:[email protected]... [...]
This one really has me squirming in my seat. C++ isn't a very good language
for learning just a little bit of. Compared to Python I would say the

Agreed! But Koenig & Moo's book (Accelerated C++) is the best I've
seen for doing that.

[...]

I wouldn't choose your C++ compiler for Python compatibility. How good is

I don't think that's unreasonable, as long as you're *aware* of all
the other issues. Ease of use with standard Python might come out as
most important. And on Windows, Python work with MSVC is much less of
a pain for most things than gcc (aka mingw).

[...]

The .NET compiler has a very good debugger and much better standard
compliance than the old 6.0 compiler, but I haven't used lot of different
recent vintage C++ compilers.

What *is* the current position with MSVC 7 and Python? Anyone? I
still haven't heard a clear story.


John

 

[John J. Lee]

I would like to use it for various applications, but the
mathematical calculations are way too slow (a million sines 8 seconds
in Python vs. 2 seconds in Excel VBA), so was thinking of learning
enough C++ to do the number crunching in C++ and integrating the C++
routines with Python. My question:

[...]

Do remember that frequently only small parts of your code (even the
number crunching part) are responsible for most of the running time.
Writing everything in Python then porting the 'hot spots' is often the
best strategy. Use the profiler.

Also (apart from Numeric / numarray), remember (or discover) Pyrex and
psyco. Probably Pyrex is more useful to you than psyco. Don't apply
either to your whole program. Just pick the bottlenecks.

Quite a few people here have touted O'Caml as a replacement for C++
for this kind of thing. C is worth considering, too, especially if
you find the parts you need to move down to a lower-level language are
small.

If you insist on using C++, make sure to look at Boost.Python
(including pyste).


John

 

[Scott David Daniels]

Have you considered and investigated Numeric Python or similar
products? Perhaps your assumption that you have to revert to
hand-crafted C code is premature.

-Peter

Look at the enthought Python installation (one is available for each
of 2.2 and 2.3) -- they not only install a slew of the packages you
might need for calculation heavy Python use, but (to support weave,
I believe), they also include a mingw32 compiler (minimal gnu win32)
which you can use for C++. If you get to the point where you are
optimization-bound on your compiler (which you should determine by
measuring, not intuiting), you could do worse than the Intel compiler.

-Scott David Daniels
(e-mail address removed)

 

[d2003xx]

Just found Python and I love it. What an elegant language!
I would like to use it for various applications, but the
mathematical calculations are way too slow (a million sines 8 seconds
in Python vs. 2 seconds in Excel VBA), so was thinking of learning
enough C++ to do the number crunching in C++ and integrating the C++
routines with Python. My question:
My question:
Which compiler works best with Python?
I use Windows XP and 98. Have very little experience with C++. Am
doing statistical and other simulations that require billions of
calculations (taking 1 to 2 hours in Excel VBA). GCC is free, MS
C++.Net is affordable

The standard C/C++ compiler (no optimization) is included in .Net
framework SDK, which is available for free in M$'s web-site.

at <$100 but supposedly has problems until the
next version (and I could only use it on my XP machine), older
versions of MS C++ are no longer for sale. What's the most robust
solution? I would also like to make .dll files that I can plug into
MS Excel.
Sorry for the vagueness of this question, but I'm sure many of you
have experience with this and any information, opinions, and even
prejudices about the various compilers are welcome.

I guess gcc is the only choice.. If you don't mind that it takes
forever to compile C++ source code, until version 3.4 is released (it
will supports pre-compiled header).

 

[achrist]

It is equally trivial to connect Python to Delphi using ctypes.
Delphi can easily make a DLL that looks to the outside world like
C. I have found Delphi to be maybe a little bit slower than MS C,
but a little faster than gcc. They are all pretty close on speed,
so you can't really pick a winner a priori. If you are doing VBA
now, can't you buy the entry-level version of VB for around 100
bucks and create C-callable DLL's?

Al

 

[rhmd]

Thanks to all who have responded. Your information was very helpful,
I'm very impressed with the quality and thoughtfulness of the
responses. Below find brief excerpts of some of the responses (which
I hope will be useful for others who search for info on this topic),
followed by more detail on my question.

BEGIN EXCERPTS
Have you considered and investigated Numeric Python or similar
products?
Look at the enthought Python installation (one is available for each
of 2.2 and 2.3) -- they not only install a slew of the packages you
might need for calculation heavy Python use, but (to support weave,
I believe), they also include a mingw32 compiler (minimal gnu win32)
which you can use for C++. If you get to the point where you are
optimization-bound on your compiler (which you should determine by
measuring, not intuiting), you could do worse than the Intel compiler.
I suggest you look at various Python libraries to do whatever
numerical
processing you might need to do. They are mostly written in C anyway
so
their performance is excellent. For instance, I use SciPy for Fourier
analysis and PIL for image processing, getting good performance on
these
numerically intensive tasks without ever leaving the ease and comfort
of
good old Python....The .NET compiler has a very good debugger and much
better standard compliance than the old 6.0 compiler, but I haven't
used lot of different recent vintage C++ compilers...I generally use
COM objects written in Python to connect to Excel. I am very pleased
with that technique.

Koenig & Moo's book (Accelerated C++) is the best I've
seen for [learning C++ for a few applications]..Python work with MSVC
is much less of a pain for most things than gcc (aka mingw).
Writing everything in Python then porting the 'hot spots' is often
the
best strategy. Use the profiler.
Also (apart from Numeric / numarray), remember (or discover) Pyrex
and
psyco. Probably Pyrex is more useful to you than psyco. Don't apply
either to your whole program. Just pick the bottlenecks. If you
insist on using C++, make sure to look at Boost.Python(including
pyste).
gcc ... If you don't mind that it takes
forever to compile C++ source code, until version 3.4 is released (it
will support pre-compiled header).
It is equally trivial to connect Python to Delphi using ctypes.
Delphi can easily make a DLL that looks to the outside world like
C. I have found Delphi to be maybe a little bit slower than MS C,
but a little faster than gcc. They are all pretty close on speed,
so you can't really pick a winner a priori. If you are doing VBA
now, can't you buy the entry-level version of VB for around 100
bucks and create C-callable DLL's?
END EXCERPTS

MORE DETAIL ON MY QUESTION
VB.NET was another option I considered, have heard it produces
binary that runs "almost" as fast as that produced by C++, but am not
sure if that is true. VB would be very convenient for me, but can it
produce DLLs that Python can use? The reason I was considering using
Python/C++ combination is that I use Python already (as scripting
language for the Blender animation program) and thought I could also
use it with Excel to overcome certain Excel limitations, such as
Excel's ridiculous limitations on chart markers (i.e., could make
better charts with Python-- this would not be a time bottleneck so
Python's slowness is not an issue there), etc. I also thought that,
despite Python's reputation for being a bit slow, it would at least be
faster than Excel VBA interpreter, but this may not be the case. At
any rate, there are some calculations where I need a LOT of speed,
e.g. generating billions (with a B) of random numbers with the
Mersenne Twister algorithm (which is available in C++), for
statistical problems (mulitple undefined boundary conditions which can
only be extracted with a Monte Carlo method). For this, even doubling
or quadrupling Python binary's speed would not be enough.

 

[Fernando Perez]

rhmd wrote:

MORE DETAIL ON MY QUESTION

At
any rate, there are some calculations where I need a LOT of speed,
e.g. generating billions (with a B) of random numbers with the
Mersenne Twister algorithm

Off the top of my head I don't know which rng RandomArray (part of Numeric)
uses, but its speed is not that bad:

In [20]: def rloop(n,m):
....: random = RA.random
....: size=10**m
....: for i in xrange(10**n):
....: _=random(size)
....:

In [21]: timing (1,rloop,1,5) # times are in cpu seconds
Out[21]: 0.12000000000000455

In [22]: timing (1,rloop,2,5)
Out[22]: 1.1999999999999957

In [23]: timing (1,rloop,3,5)
Out[23]: 12.469999999999999


The scaling is pretty linear, so for a billion terms you'd need about 2
minutes.

hth.

Cheers,

f

 

[Alex Martelli]

Fernando Perez wrote:
...

Off the top of my head I don't know which rng RandomArray (part of
Numeric) uses, but its speed is not that bad: ...
The scaling is pretty linear, so for a billion terms you'd need about 2
minutes.

Similar orders of magnitude for the built-in random number generator,
which IS a Mersenne Twister in Python 2.3 -- at least on my old trusty
Athlon box (no doubt today's CPU's would easily take 3 or 4 times less):

[alex@lancelot python2.3]$ python timeit.py \

-s "from random import random" \
-s "times=[None]*1000"
"for x in times: random()"

1000 loops, best of 3: 380 usec per loop

380 microseconds for 1,000 calls to random.random(), and here, too,
very linear scaling, suggests about 380 seconds for a billion calls,
i.e., about 6 or 7 minutes, on this old machine.


Alex

 

[rhmd]

Similar orders of magnitude for the built-in random number generator,
which IS a Mersenne Twister in Python 2.3 -- at least on my old trusty
Athlon box (no doubt today's CPU's would easily take 3 or 4 times less):

[alex@lancelot python2.3]$ python timeit.py \

-s "from random import random" \
-s "times=[None]*1000"
"for x in times: random()"

1000 loops, best of 3: 380 usec per loop

380 microseconds for 1,000 calls to random.random(), and here, too,
very linear scaling, suggests about 380 seconds for a billion calls,
i.e., about 6 or 7 minutes, on this old machine.

Wow! I didn't realize that 2.3 had upgraded to Mersenne Twister
(which has a period of more than 10**10000, vs. WichmannHill which has
a period of less than 10**10 and the Excel 2000 PNRG which (to my
shock and chagrin) had a period of about 10**7). This is awesome.

 

[rhmd]

There was an error in my previous message Wichmann Hill has a period
of less than 10**14 (not 10**10)

Similar orders of magnitude for the built-in random number generator,
which IS a Mersenne Twister in Python 2.3 -- at least on my old trusty
Athlon box (no doubt today's CPU's would easily take 3 or 4 times less):

[alex@lancelot python2.3]$ python timeit.py \

-s "from random import random" \
-s "times=[None]*1000"
"for x in times: random()"

1000 loops, best of 3: 380 usec per loop

380 microseconds for 1,000 calls to random.random(), and here, too,
very linear scaling, suggests about 380 seconds for a billion calls,
i.e., about 6 or 7 minutes, on this old machine.

Wow! I didn't realize that 2.3 had upgraded to Mersenne Twister
(which has a period of more than 10**10000, vs. WichmannHill which has
a period of less than 10**10 and the Excel 2000 PNRG which (to my
shock and chagrin) had a period of about 10**7). This is awesome.