Using python as primary language

[Chris Mellon]

But if Python gets slow when you add fine-grained locks, then most
certainly it wouldn't get so slow if the locks were very fast, right?

But that's not what my question was about. It was about whether it
would make sense to, on the same python installation, select between
the GIL and fine-grained locks at startup. Because even if the locks
slows down the interpreter, if they let you utilize a 32 core CPU, it
may not be so bad anymore. Or am I missing something?

The performance overhead of a runtime swap between fine-grained locks
and the GIL would probably swamp any benefit you gained from having it
be switchable. A compile time switch is more reasonable, but it has
all the other problems of going fine-grained in the first place (it
breaks compatibility with existing extensions, large code size
increase, increased maintenance burden) and it adds the additional
problem that one or other of the modes won't be as heavily tested
(this will start out as the fine-grained mode, of course, which is the
worst case because it's the more complicated and subtly error prone
one). It's really best in the long term to pick one and stick with it.

The heavy use of dicts is one of the problems - they're all over the
place and even if you removed the GIL, a "global dict lock" would give
essentially the same effect. And a per-dict lock means that there will
be a *lot* more locking and unlocking, which means a slower
interpreter.

 

[kyosohma]

What I meant was that one should be able to "draw" a report template.
Basically a graphical user interface for RML in this case. I
personally would opt for writing RML or whatever code myself. But it's
impossible to convice my boss. The dialog usually goes like this:

"So, did you find a click-and-play editor for reports" - "Not yet, but
anyway, writing code is more flexible and easier to manage later on" -
"Hmmm... but it's code!" - "Sure, but you only write it once for one
report, you can easily re-use code-snippets, modifying the code does
not require one additional program, you just use your text-editor of
choice,..." - "Okay,.... but it's CODE!"....

and this goes on forever. My boss seems to be allergic to writing code
by hand. Which is very frustrating. I'm happy that Qt has the
"designer", although it's very easy to code the GUI's too

wxPython has multiple "designers", like Boa and SPE...but as for a way
to "draw a report", I think you're stuck with something like VBA in an
Office application or the "holy grail" of report generators, Crystal
Reports. Personally speaking, I find Crystal to be more difficult to
use than coding it myself. To each their own, I suppose.

Mike

 

[Rhamphoryncus]

The heavy use of dicts is one of the problems - they're all over the
place and even if you removed the GIL, a "global dict lock" would give
essentially the same effect. And a per-dict lock means that there will
be a *lot* more locking and unlocking, which means a slower
interpreter.

It's worse than that. Not only are they slower for a single thread,
but when using multiple threads the cores will contend over the cache
lines containing the locks, leaving you yet slower still!

I've starting putting together a site explaining my approach to
solving these problems:

http://code.google.com/p/python-safethread/

 

[Boris Borcic]

What I meant was that one should be able to "draw" a report template.
Basically a graphical user interface for RML in this case. I
personally would opt for writing RML or whatever code myself. But it's
impossible to convice my boss. The dialog usually goes like this:

"So, did you find a click-and-play editor for reports" - "Not yet, but
anyway, writing code is more flexible and easier to manage later on" -
"Hmmm... but it's code!" - "Sure, but you only write it once for one
report, you can easily re-use code-snippets, modifying the code does
not require one additional program, you just use your text-editor of
choice,..." - "Okay,.... but it's CODE!"....

and this goes on forever. My boss seems to be allergic to writing code
by hand. Which is very frustrating. I'm happy that Qt has the
"designer", although it's very easy to code the GUI's too

Probably worth pointing out that a click-and-point editor for reports can't be
both fully mimetic (as a GUI designer may be) and practical.

FWIW OpenOffice 2.3 added something of the sort. The kind of tools I'd expect to
lead me to roadblocks. Now OpenOffice *does* allow scripting with python; but
the couple times I looked into it, the specs&docs on the scripting interface
scared me completly.

Good luck, BB

 

[Terry Reedy]

| But that's not what my question was about. It was about whether it
| would make sense to, on the same python installation, select between
| the GIL and fine-grained locks at startup. Because even if the locks
| slows down the interpreter, if they let you utilize a 32 core CPU, it
| may not be so bad anymore. Or am I missing something?

1. The need for some group to develop and maintain what anounts to a forked
version of CPython.

2. If micro-locked Python ran, say, half as fast, then you can have a lot
of IPC (interprocess communition) overhead and still be faster with
multiple processes rather than multiple threads.

 

[Rhamphoryncus]

| But that's not what my question was about. It was about whether it
| would make sense to, on the same python installation, select between
| the GIL and fine-grained locks at startup. Because even if the locks
| slows down the interpreter, if they let you utilize a 32 core CPU, it
| may not be so bad anymore. Or am I missing something?

1. The need for some group to develop and maintain what anounts to a forked
version of CPython.

That's pretty much what I'm doing.

2. If micro-locked Python ran, say, half as fast, then you can have a lot
of IPC (interprocess communition) overhead and still be faster with
multiple processes rather than multiple threads.

Of course you'd be faster still if you rewrote key portions in C.
That's usually not necessary though, so long as Python gives a roughly
constant overhead compared to C, which in this case would be true so
long as Python scaled up near 100% with the number of cores/threads.

The bigger question is one of usability. We could make a usability/
performance tradeoff if we had more options, and there's a lot that
can give good performance, but at this point they all offer poor to
moderate usability, none having good usability. The crux of the
"multicore crisis" is that lack of good usability.

 

[Waldemar Osuch]

In our company we are looking for one language to be used as default
language. So far Python looks like a good choice (slacking behind
Java). A few requirements that the language should be able cope with
are:

* Database access to Sybase.
This seems to be available for python, but the windows-binaries for
the library
are not available. Self-Compiling them proved to be non-trivial (As
always
on windows).

Consider using ODBC. "mxODBC" as well as "pyodbc" and "ceODBC" could
be
good alternatives.

* Easy GUI creation.
Solved using PyQt.
* Cross Platform (Linux + Windows).
Again, PyQt, solves this
* Easy deployment.
Solved using py2exe + innosetup
* Charting (Histograms, Line charts, bar charts, pie charts, ...)
I am currently looking into PyQwt, which looks promising.
* Report generation with GUI support
reportlab + rml?

So far, nearly all point seems to be manageable. But I was not yet
able to find a solution for the report generation. What we would like
to have is a sort of GUI interface to prepare the reports without
having to "code" the positioning. I found reportlab, and it looks like
it can do all that is needed in terms of output. But you still have to
code the report. And this is a no go. In context, I found RML and saw
links to graphical RML editors. But I have not yet found a concrete
example code, or documentation. What am I missing? Is RML a reportlab
creation or is it a recognised open standard? If not, is there an open
standard, that is easily to process with python?

I still have to try it but pisa looks very promising:
http://pisa.spirito.de/content/501/pisa3.html
PDF generation using HTML+CSS for layout.
I hope your boss considers HTML acceptable "coding".

Any pointers? I would prefer coding Python to coding Java or
worse..... VB which is another contender in our roundup.

You mean VB before .Net? You know that Microsoft ended support for
it.
Thanks to that brilliant decision Python is an "official" language at
my place of work.

 

[Martin Vilcans]

Of course you'd be faster still if you rewrote key portions in C.
That's usually not necessary though, so long as Python gives a roughly
constant overhead compared to C, which in this case would be true so
long as Python scaled up near 100% with the number of cores/threads.

The bigger question is one of usability. We could make a usability/
performance tradeoff if we had more options, and there's a lot that
can give good performance, but at this point they all offer poor to
moderate usability, none having good usability. The crux of the
"multicore crisis" is that lack of good usability.

Certainly. I guess it would be possible to implement GIL-less
threading in Python quite easily if we required the programmer to
synchronize all data access (like the synchronized keyword in Java for
example), but that gets harder to use. Am I right that this is the
problem?

Actually, I would prefer to do parallell programming at a higher
level. If Python can't do efficient threading at low level (such as in
Java or C), then so be it. Perhaps multiple processes with message
passing is the way to go. It just that it seems so... primitive.

 

[Hrvoje Niksic]

But if Python gets slow when you add fine-grained locks, then most
certainly it wouldn't get so slow if the locks were very fast,
right?

Given the sheer number of increfs and decrefs happening, they should
be impossibly fast (meaning: nonexistent). Even adding an additional
test to those macros slows down Python, let alone adding a lock.

But that's not what my question was about. It was about whether it
would make sense to, on the same python installation, select between
the GIL and fine-grained locks at startup. Because even if the locks
slows down the interpreter, if they let you utilize a 32 core CPU,
it may not be so bad anymore.

Unfortunately the selection must be done at compile time, not at
run-time. Even taking into account the possibility of selection, so
far no one has come up with a set of patches that remove the GIL that
would come close to being accepted.

This is a recent discussion about GIL removal that provides good
reading:
http://mail.python.org/pipermail/python-dev/2007-September/thread.html#74545

 

[Marc 'BlackJack' Rintsch]

Actually, I would prefer to do parallell programming at a higher
level. If Python can't do efficient threading at low level (such as in
Java or C), then so be it. Perhaps multiple processes with message
passing is the way to go. It just that it seems so... primitive.

Well, to me threads seem to be more primitive. You have to be careful
with shared memory accesses and they don't scale so well if you want to go
from single machines to a cluster.

Take a look at XML-RPC or Pyro for higher level communication between
processes.

Ciao,
Marc 'BlackJack' Rintsch

 

[Rhamphoryncus]

Certainly. I guess it would be possible to implement GIL-less
threading in Python quite easily if we required the programmer to
synchronize all data access (like the synchronized keyword in Java for
example), but that gets harder to use. Am I right that this is the
problem?

Actually, I would prefer to do parallell programming at a higher
level. If Python can't do efficient threading at low level (such as in
Java or C), then so be it. Perhaps multiple processes with message
passing is the way to go. It just that it seems so... primitive.

What you've got to ask is "what is a message?" Can you define your
own class and use it as a message? Doing so seems very useful. If
you allow that you end up with something like Concurrent Pascal's
monitors (which enforce their boundary, so they lack the memory model
needed by java or C.)

Once you make message passing easy to use you end up with something
that looks very much like threads anyway, just lacking the shared-
state and the resulting memory model.

 

[Russell E. Owen]

In our company we are looking for one language to be used as default
language. So far Python looks like a good choice (slacking behind
Java). A few requirements that the language should be able cope with
are:

* Database access to Sybase.
This seems to be available for python, but the windows-binaries for
the library
are not available. Self-Compiling them proved to be non-trivial (As
always
on windows).
* Easy GUI creation.
Solved using PyQt.
* Cross Platform (Linux + Windows).
Again, PyQt, solves this
* Easy deployment.
Solved using py2exe + innosetup

How do you deploy python on linux? The only solution I know of is
pyinstaller and I never could get it to work for me on linux (I never
bothered to try on Windows since I already had py2exe doing what I
needed).

In my opinion this is one of Python's few weaknesses compared to Java.
Others that come to mind are:
- Lack of a built-in networking library that works with GUI toolkits
(use Twisted Framework and hope it continues to be supported)
- Lack of a good built-in GUI toolkit (but there are several good
alternatives including Qt)

* Charting (Histograms, Line charts, bar charts, pie charts, ...)
I am currently looking into PyQwt, which looks promising.

HippoDraw is very good. I am not familiar with PyQwt so I cannot compare
them.

-- Russell

 

[kyosohma]

How do you deploy python on linux? The only solution I know of is
pyinstaller and I never could get it to work for me on linux (I never
bothered to try on Windows since I already had py2exe doing what I
needed).

The wxPython users seem to create RPMs and Debs. (http://
www.wxpython.org/download.php)

You might find these articles interesting as well:

http://docs.python.org/dist/dist.html
http://www.linux.com/feature/118439
http://docs.python.org/dist/creating-rpms.html

In my opinion this is one of Python's few weaknesses compared to Java.
Others that come to mind are:
- Lack of a built-in networking library that works with GUI toolkits
(use Twisted Framework and hope it continues to be supported)
- Lack of a good built-in GUI toolkit (but there are several good
alternatives including Qt)

The wxPython user's group mentions charting quite a bit. I think they
use matplotlib among others. You might contact them for suggestions as
well.

Mike

 

[Diez B. Roggisch]

How do you deploy python on linux? The only solution I know of is
pyinstaller and I never could get it to work for me on linux (I never
bothered to try on Windows since I already had py2exe doing what I
needed).

Having a running python interpreter under linux is the same effort as
getting a JRE running. Actually due to licensing problems, python
usually just ships whereas java has to be explicitly installed.

From there, deploying using setuptools is easy enough IMHO.

In my opinion this is one of Python's few weaknesses compared to Java.
Others that come to mind are:
- Lack of a built-in networking library that works with GUI toolkits
(use Twisted Framework and hope it continues to be supported)
- Lack of a good built-in GUI toolkit (but there are several good
alternatives including Qt)

I'm not sure if you mean both above "compared to Java" - but I won't
call Swing/AWT "good" - eclipse doesn't come with SWT for nothing. And I
simply don't understand the networking that works with GUI toolkits. I'm
aware of some eventloop-unifications twisted does, but I never really
understood the need for that - multi-threading is a problem for GUIs,
not for networking. And queues are your friend.

Diez

 

[sturlamolden]

The wxPython user's group mentions charting quite a bit. I think they
use matplotlib among others. You might contact them for suggestions as
well.

Indeed, use NumPy/SciPy and matplotlib if you are using Python for
numerical computing and data visualization. Matplotlib has capabilites
similar to the 2D plotting/charting of Matlab, and gives you beautiful
antialiased graphs. Matplotlib works nicely with wxPython, PyGTK and
tkinter.

NumPy and matplotlib is not in the Python standard library, but I hope
that will change one day.

 

[sturlamolden]

In our company we are looking for one language to be used as default
language. So far Python looks like a good choice (slacking behind
Java). A few requirements that the language should be able cope with
are:

* Database access to Sybase.
This seems to be available for python, but the windows-binaries for
the library
are not available. Self-Compiling them proved to be non-trivial (As
always
on windows).
* Easy GUI creation.
Solved using PyQt.
* Cross Platform (Linux + Windows).
Again, PyQt, solves this
* Easy deployment.
Solved using py2exe + innosetup
* Charting (Histograms, Line charts, bar charts, pie charts, ...)
I am currently looking into PyQwt, which looks promising.
* Report generation with GUI support
reportlab + rml?

These are all library issues, and has nothing to do with the language!

Besides that, if Visual Studio and .NET is what you want, you can
always use IronPython.

 

[sturlamolden]

I'm not sure if you mean both above "compared to Java" - but I won't
call Swing/AWT "good" - eclipse doesn't come with SWT for nothing.

Swing vs. SWT is a matter of taste and religion.

The main complaint against Swing was how it looked. That has changed.
Also, Swing was sluggish ong Windows and SWT was sluggish on X11.

And I
simply don't understand the networking that works with GUI toolkits. I'm
aware of some eventloop-unifications twisted does, but I never really
understood the need for that - multi-threading is a problem for GUIs,
not for networking. And queues are your friend.

Indeed.

I think the need for these "eventloop unifications" stems from Visual
Basic. VB programmers never learned to use more than one thread, and
they are still struggling to unlearn the bad habits they aquired.