Distributed App - C++ with Python for Portability?

[Roopan]

Hello!

I am looking at developing an enterprise-grade distributed data
sharing application - key requirements are productivity and platform
portability.

Will it be sensible to use C++ for performance-critical sections and
Python for all the glue logic.

Pls comment from your *experiences* how Python scales to large
projects( > 200KLOC).
I assume the C++/Python binding is fairly painless.

Regards
Elam.

 

[Paddy]

Hello!

I am looking at developing an enterprise-grade distributed data
sharing application - key requirements are productivity and platform
portability.

Will it be sensible to use C++ for performance-critical sections and
Python for all the glue logic.

Pls comment from your *experiences* how Python scales to large
projects( > 200KLOC).
I assume the C++/Python binding is fairly painless.

Regards
Elam.

You might try prototyping as much as possible in Python as soon as
possible.
Then, and only after getting something that computes the right
results,
profile your prototype to see where the real bottlenecks are.
Sometimes it
it is not evident at the beginning where the bottlenecks in the
prototype
will be.
If you write most of the prototype in Python then you will write less
lines of code, in a shorter time and so should be more inclined to
experiment with different algorithms, and do more testing. (Doctest
can
be great).
After profiling their may be other ways to remove a bottleneck, such
as
using existing highly-optimised libraries such as Numpy; Psycho, an
optimising interpreter that can approach C type speeds for Python
code;
and you could create your own C++ based libraries.

You might want to ask the Mercurial development team how they got
their
impressive speed and functionality out of using mainly Python with
critical regions in C. - Or watch this:
http://video.google.com/videoplay?docid=-7724296011317502612

- Paddy.

 

[Matt Nordhoff]

After profiling their may be other ways to remove a bottleneck, such
as
using existing highly-optimised libraries such as Numpy; Psycho, an
optimising interpreter that can approach C type speeds for Python
code;
and you could create your own C++ based libraries.

You might want to ask the Mercurial development team how they got
their
impressive speed and functionality out of using mainly Python with
critical regions in C. - Or watch this:
http://video.google.com/videoplay?docid=-7724296011317502612

For what you do decide to rewrite in C, you can also use a language like
Cython [1] (which is a fork of Pyrex [2]). It looks mostly like Python,
and is translated to C without you having to write all of the
boilerplate Python C API stuff. Of course, not quite as efficient
well-tuned raw C, but much more pleasant to write.

(FWIW, Bazaar [3], another VCS written in Python similar to Mercurial,
has rewritten parts of two modules in Pyrex. Another one is in raw C,
because that's what was contributed.)

[1] <http://www.cython.org/>
[2] <http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/>
[3] <http://bazaar-vcs.org/>
--

 

[Stefan Behnel]

For what you do decide to rewrite in C, you can also use a language like
Cython [1] (which is a fork of Pyrex [2]). It looks mostly like Python,
and is translated to C without you having to write all of the
boilerplate Python C API stuff. Of course, not quite as efficient
well-tuned raw C, but much more pleasant to write.

And if you really need the efficiency of "well-tuned raw C", it's one function
call away in your Cython code.

Stefan

 

[Matt Nordhoff]

And if you really need the efficiency of "well-tuned raw C", it's one function
call away in your Cython code.

What do you mean by that?

I know nothing about how Cython compares to C in performance, so I said
"well-tuned" because it must be possible to write C that is faster than
Cython, though it may take some effort.
--

 

[Stefan Behnel]

What do you mean by that?

I know nothing about how Cython compares to C in performance, so I said
"well-tuned" because it must be possible to write C that is faster than
Cython, though it may take some effort.

So, you write the hand-optimised function in plain C, declare it in Cython and
call it. That's what I meant. Since Cython compiles to C code, linking against
a C module is straight forward. And this still keeps you from having to write
all the Python API glue code in plain C.

Stefan

 

[castironpi]

And if you really need the efficiency of "well-tuned raw C", it's one function

So, you write the hand-optimised function in plain C, declare it in Cython and
call it. That's what I meant. Since Cython compiles to C code, linking against
a C module is straight forward. And this still keeps you from having to write
all the Python API glue code in plain C.

Python was originally intended to just link C modules, right? (ITTRW
if that's the right word?) What are Python's roots? What are its
principles? What are its fundamentals? (And what does Python 8! look
like!?) We can even get multi-threading running outside of the Global
Interpreter Lock, if the thread only acquires it to access shared
object code... make that mananged objects.

One big decision is if you have a separate Python interpreter running
on every remote location, versus say as just a relay mechanism. C on
the rest. CMIIW, correct me if I'm wrong, but the productivity
bottlenecks vs. performance bottlenecks make a trade-off.

For my part, sometimes I get __getattr__ bottlenecks, and other times
object instantiation bottlenecks. There was an iterator one in there
too. But no I didn't have competing C code to test to compare. If I
understand OP correctly, C libraries will be running at all locations,
which means they'll need separate compilations. In fact, come to
think of it, I'm having trouble with the concept of cross-platform
distributed. Do OP mean that lots of programmers are sharing data?
Will all the platforms be performing all of the tasks? Or do they
specialize? (Not necessarily / no / yes ok.) Lastly, if you can get
a boost by buying more servers, then there's a resource bottleneck
breakpoint to consider too.

 

[Diez B. Roggisch]

Hello!

I am looking at developing an enterprise-grade distributed data
sharing application - key requirements are productivity and platform
portability.

Will it be sensible to use C++ for performance-critical sections and
Python for all the glue logic.

Pls comment from your *experiences* how Python scales to large
projects( > 200KLOC).
I assume the C++/Python binding is fairly painless.

It depends. There are good wrappers out there, I personally prefer SIP.
However, a mixed language environment is always a PITA, especially for
distribution.

If you can, write everything in python. Identify bottlenecks, and if you
must, I suggest using C + ctypes for performance-critical code.

Obviously it's a matter of taste, but C++ is a beast, and getting it to work
seamless under varying compilers and OSes could be avoided using plain C.

Diez

 

[Bob Martin]

It depends. There are good wrappers out there, I personally prefer SIP.
However, a mixed language environment is always a PITA, especially for
distribution.

If you can, write everything in python. Identify bottlenecks, and if you
must, I suggest using C + ctypes for performance-critical code.

Obviously it's a matter of taste, but C++ is a beast, and getting it to work
seamless under varying compilers and OSes could be avoided using plain C.

Diez

Java is more portable than most other languages, especially if your app needs a gui.

 

[Stefan Behnel]

Java is more portable than most other languages, especially if your app needs a gui.

I don't think anything is more portable for an App GUI than HTML through a
built-in web server.

That said, does gcj compile GUI Apps by now? Or does that not count as "portable"?

Stefan

 

[Carl Banks]

Hello!

I am looking at developing an enterprise-grade distributed data
sharing application - key requirements are productivity and platform
portability.

Will it be sensible to use C++ for performance-critical sections and
Python for all the glue logic.
Pls comment from your *experiences* how Python scales to large
projects( > 200KLOC).
I assume the C++/Python binding is fairly painless.

200K with C++ lines of code or Python lines of code?

I can comment on 50K lines of Python code, and considering how I write
Python and how most people write C++ that might be about the same as a
200K C++ program. But it's not an enterprise app, so take it as you
will.

I've had no real issues. 95% of the program in Python; with only a
couple sections written in C, and a few wrappers for C/C++ libraries.
My wrappers are as thin as I can reasonably make them; where needed I
make the interface prettier with a higher-level Python wrapper.

I'm writing regular extensions, not using ctypes, Cython, SWIG,
Boost:ython or anything like that. I am using ctypes for some
related tools and it's pretty nice. The others I can't comment on,
except to say I sometimes didn't have enough memory to compile third-
party SWIG extensions.

I develop it on Linux, but every few months or so I'd copy it to
Windows and update it so that it works on both. This was a minor
hastle, which is not so bad considering it hadn't seen Windows for
months. (In fairness, I am using the same compiler, gcc, on both.
Getting mingw gcc to work on Windows was a major hassle, but a one-
time thing.)


Python's a lot better productivity-wise, and it's worked well as a
cross-platform solution for me. The interfacing is the main
drawback. It's always a bit of effort to interface things, and to get
it working on multiple platforms, but I thought it was worth it.


Carl Banks

 

[Diez B. Roggisch]

Java is more portable than most other languages, especially if your app needs a gui.

Depending on what GUI you want, python is at least as portable - Tkinter
is available for lots of platforms. And swing as a GUI plain sucks -
there is a reason for SWT. But that needs to be installed separately and
per platform.

Additionally, Java is not Python

Diez

 

[dave_mikesell]

Java is more portable than most other languages, especially if your app needs a gui.

The promise of Java portability was one of the biggest scams ever
perpetrated on the software industry. There are issues going from OS
to OS, VM to VM, DB to DB, app server to app server, etc. Certainly
no easier than porting C++ and the appropriate libraries, IMHO.

 

[Bob Martin]

The promise of Java portability was one of the biggest scams ever
perpetrated on the software industry. There are issues going from OS
to OS, VM to VM, DB to DB, app server to app server, etc. Certainly
no easier than porting C++ and the appropriate libraries, IMHO.

Quite untrue - I have a stack of large java apps which run without change on
Linux, OS/2 and Windows. Not many, if any, other languages can match that,
and definitely not C++.

 

[dave_mikesell]

Quite untrue - I have a stack of large java apps which run without change on
Linux, OS/2 and Windows. Not many, if any, other languages can match that,
and definitely not C++.

I'm happy that it's worked for you, but I haven't seen it in my
experience. My current client has code that works in 1.4.0, but not
higher versions of the JDK without code changes. And other that won't
run in later versions of the app server (JBoss) than the one for which
it was developed. And upgrading our SQL Server version required an
upgrade of DB drivers. Another headache. They have their own JVM
requirements as well, as do other packages and libraries. Getting
them all to play together hasn't been seamless or easy by any stretch.

 

[Jeff Schwab]

I assume the C++/Python binding is fairly painless.

http://www.boost.org/libs/python/doc/tutorial/doc/html/index.html

Ahhhh.