Going the PL/1 way

M

mudd

That's not to say there aren't limited problem domains where threads
are the right answer, but as a general construct they're wildly
overused. I'm frankly glad that Python doesn't encourage more of the
same.

Is there anything else that Python doesn't do well that we should appreciate?
 
A

Anthony Baxter

Is there anything else that Python doesn't do well that we should appreciate?

I wouldn't go so far as to say that we should _appreciate_ the GIL, more that
it ends up being not so much of a problem because Python gives us many,
many better ways to do things than with threads.

About the only actual issue with the GIL is that it stops a single
Python interpreter
from taking better advantage of multiple CPUs in a box. But, if you code with a
knowledge of the GIL, this is easy enough to work around.

The advantages of the GIL is that the implementation is far more robust, and far
less overhead is wasted in lots of tiny little locks. In addition, it
means that a single
python instruction is guaranteed to be thread safe - so in a threaded app I can
do something like
somedict['foo'] = 1
without having to wrap access in a lock to make sure I don't end up with a
hosed dictionary. Once you end up with lots of little locks in a threaded
application, madness is surely not far away.

Anthony
 
G

G. S. Hayes

Is there anything else that Python doesn't do well that we should appreciate?



That wasn't really the point; the point was more to argue against the
viewpoint that work on language features like list comprehensions or
decorators is somehow less important than working on removing the GIL.
Obviously in an ideal world you could implement every idea that came
along, but in the real world developers tend to work on problems that
affect them--and if you have a bunch of developers who rarely run into
problems with the GIL (because they're not often abusing threads) then
it's just not going to be a very high priority. Rightly so, IMO,
since it only affects a very small percentage of well-designed
programs, so putting work into bigger problem areas (especially any
language features that genuinely improve development speed and
readability) gets a much higher payoff.



I wouldn't be against removing the GIL if an approach was found that
didn't adversely affect the common case; naive approaches to
fine-grained locking are often really bad for general case behavior,
but there are approaches that can mitigate this. But I would be
against lobbying core language developers to take time off of other
projects to work on this one; it's definitely more of a "scratch an
itch" type problem` that can wait until someone feels motivated to do
it than a "suck it up because the community needs it" kind of thing.
 
C

Christian Tismer

Harald said:

Ooh, this is truely too much fear. Decorators are about
modification of how code is understood, but not modification.
[ If it were so, then I'd be probably much more interested :) ]
I honestly ask that someone could explain to the less enlightened like me
what this decorator thingy really is. What does it make a correct step on
the path to enlightenment?

Here a very short and very incomplete explainment of what
a decorator is. FOr a more complete explanation, please
refer to the PEP, or even wade through python-dev.
I also just give one simple example, there are lots of
other uses for decorators.
But since probably more than 95 percent of Python code will
not need decorators in the near future, just one example
should be enough to just don't let you worry about what
the Voodoo is.

Something many Python programmers even don't know about are
class methods and static methods.

A static method is simply a method which is defined in a class,
but that does not operate on self. The self variable is not
supplied. A static method is not supposed to be a method of an
instance. It is just sitting in a class for convenience, but
behaves like an ordinary function.

There is a more advanced concept: a class method. Instead of self
or nothing, it always receives the class as first parameter.
But please read the docstring of the classmethod function.
We will stick with the simple static method, here.

Until Python 2.3, it was necessary to "declare" a static method
by calling the function staticmethod on it.

Sample code:

class MyClass(object):

def func(arg1, arg2):
return arg1 + arg2

func = classmethod(func)

What can you see here?
The "def func" lacks a self parameter. You want to call it
with the two parameters arg1 and arg2, but without the
extra line
func = classmethod(func)

you would get this result:
Trying to call func via the class gives you
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
TypeError: unbound method func() must be called with MyClass instance as
first argument (got int instance instead)
Trying to call func via an instance also doesn't work:
Traceback (most recent call last):

Now, with the mentioned extra line, the function object is
modified in a way that it understands that it is an ordinary
function, and it works in both cases:
.... def func(arg1, arg2):
.... return arg1 + arg2
.... func=staticmethod(func)
....
--- now finally on the decorator issue ---

What you have see is, that staticmethod has to be called *after*
the function is defined. The modified function is assigned to
the class as a post process.
For very long functions, this can become cumbersome.
Not only that you have to type the function name three times,
you also have to scroll through the full function definition,
possibly even through the rest of the whole class definition,
to see whther the function is modified somehow.

One of the purposes of decorators is now to overcome this
problem, and to annotate the function earlier than at its end.
There are many proposals how to do this, also with extended
functionality. But I'll reduce the example to the currently
proposed syntax:

class MyClass(object):

@classmethod

def func(arg1, arg2):
return arg1 + arg2


This is the basic idea what a decorator does:
Make modifiers like classmethod into a prefix
of a function.


I know this description is incomplete, but I beg those who
know better _not_ to respond and prove their knowledge.
I wanted to give a simple idea. Corrections are welcome, of course.
Everything else is documented in PEP318.

ciao - chris

--
Christian Tismer :^) <mailto:[email protected]>
Mission Impossible 5oftware : Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/
14109 Berlin : PGP key -> http://wwwkeys.pgp.net/
work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/
 
R

Robin Becker

Christian Tismer wrote:

........
Trying to call func via an instance also doesn't work:

Traceback (most recent call last):


Now, with the mentioned extra line, the function object is
modified in a way that it understands that it is an ordinary
function, and it works in both cases:

... def func(arg1, arg2):
... return arg1 + arg2
... func=staticmethod(func)
...

Isn't robustness an issue with the @ decorator as well? The most important
symbol can be omitted without causing a syntax error, but I suppose the same can
be argued for any of the unary operations.
 
M

Miklós

Ooh, this is truely too much fear. Decorators are about
modification of how code is understood, but not modification.
[ If it were so, then I'd be probably much more interested :) ]
[..snip..]

Now, with the mentioned extra line, the function object is
modified in a way that it understands that it is an ordinary
function, and it works in both cases:
[..snip..]
What you have see is, that staticmethod has to be called *after*
the function is defined. The modified function is assigned to
the class as a post process. [..snip..]
to see whther the function is modified somehow. [..snip..]
This is the basic idea what a decorator does:
Make modifiers like classmethod into a prefix
of a function.

Great, so it's still about code modification.
With Python decorators, a method/function can be modified virtually in any
way, outside its declaration.
Sure you could override things before, too, but now this is a request for
code abuse.

Best,
Miklós
 
C

Christian Tismer

Miklós wrote:

....
Great, so it's still about code modification.

No, this is, with respect, nonsense.
With Python decorators, a method/function can be modified virtually in any
way, outside its declaration.

No. You cannot do anything that could not be done
without decorators, before (at least today).
Sure you could override things before, too, but now this is a request for
code abuse.

My explanation was *not* meant as a request for list abuse.

--
Christian Tismer :^) <mailto:[email protected]>
Mission Impossible 5oftware : Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/
14109 Berlin : PGP key -> http://wwwkeys.pgp.net/
work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/
 
M

Michael Ekstrand

It is first time I felt it's "we and them" world, divided into
"Python developers" and "Python users". They want features. We want
stability and performance.

Perhaps I am unique among Python users. But I've always welcomed the new
features Python has introduced in recent times, especially the
new-style class stuff in 2.3.

Perhaps it is because my programing paradigm is intensely theoretical,
with a lot of thought put towards purity of model. But it seems that
the features the "Python developers" are adding increase Python's
theoretical beauty, and hence make life easier for me as a user. Except
for when I'm stuck needing to code for pre-2.3 Python. Or coding with
Zope, where I feel that my hands are tied because I can't use some new
features.

-Michael
 
D

Donn Cave

That is not really surprizing, the GIL is a very simple and very
efficient way to manage
multithreading. After reading, I can't undersand that other interpreted
languages don't do the same.
The use of GIL make that interpreter work as a single thread process.

Right, and other languages do indeed do the same, for
example I believe ocaml has a global lock - even though
compiled to native code, it has the same kind of shared
global state for storage and so forth.

It is very effective, and it works fine for the kinds
of things people usually do today with threads. If the
computer of the future makes many processors available
for concurrent processing, it won't work quite so well
for that, but who knows what will happen.

I'm not sure if I followed the rest of your post, but
note that Python's thread architecture works well with
blocking I/O. It just depends on the native code module
that actually posts the I/O request to release the lock
first, and recover it afterwards, so any other threads
that may be waiting can be scheduled. On the other hand,
there are often better ways to deal with blocking I/O
than threads.

Donn Cave, (e-mail address removed)
 
J

Jarek Zgoda

G. S. Hayes said:
Java has drunk the threads Kool-Aid, and Java programmers are very
likely to be thread-crazy because of the lack of a select equivalent
(until recently) and the lack of good access to processes via fork or
similar. Luckily, it's usually pretty easy to convince people of the
benefits of NOT overusing threads once they've made that mistake once.

Try GUI programming without using threads.
 
J

Jeff Shannon

Jarek said:
G. S. Hayes <[email protected]> pisze:




Try GUI programming without using threads.

I've done this plenty of times. Threads seem to only be essential when
long-running background tasks are involved. There's a large set of GUI
programs where this is not an issue. I've also never had GIL problems
with multithreaded GUI apps. Then again, I'm not writing for
multiprocessor machines.

As I understand it, the GIL only becomes a real "problem" in the case of
a multiprocessor machine running multithreaded code, and the "problem"
is only that you get less-than-optimal efficiency. I would submit that
the vast majority of Python applications run on single-processor
machines, and that for many of the apps that *are* run on SMP it's not
essential to run at theoretical-optimum efficiency. Multithreaded,
multiprocessor applications where peak efficiency is necessary, threads
can't be replaced by processes, and the use of C extensions (which don't
suffer from GIL issues) is impractical make up a rather small section of
the Python usage-base, and it's only those applications that see
significant negative impacts from the GIL. Yes, it'd be great if the
GIL wasn't necessary, but if eliminating it makes *my* single-proc,
single-thread apps run significantly slower then the change is a net loss.

Jeff Shannon
Technician/Programmer
Credit International
 
S

Sam Holden

Try GUI programming without using threads.

Right this minute I'm writing a GUI program which creates a long lasting
tcp connection, and at times less long lasting udp "connections".

No threads.

Well I guess one thread... :)
 
G

G. S. Hayes

Try GUI programming without using threads.



For the GUI tasks that I've done (mostly data visualization stuff for
doing sophisticated views and edits on SQL databases and music
visualization software for doing realtime FFT/spectrum
analyzer/oscilloscope kinds of things that move with the music) it's
been absolutely no problem to use real processes for GUI work. Why
exactly do you not want protected memory for these things?



Lots of big, sophisticated GUIs are implemented without threads (e.g.
Netscape Navigator/Communicator). It almost always winds up being
easier on the programmer and more responsive for the user in the long
run.



Threads often seem simple at the outset, but most of the time they
wind up being far more complex to program and debug than judicious use
of state machines and processes. The major exception is libraries
(especially matrix-math kinds of things) where the internals are so
deeply entwined that it's not easy to figure out well-segmented subset
to put in shared memory. Often (but not always) the amount of work it
takes to figure out all the locking issues is a lot higher than what
it would've taken to seperate the data structures out cleanly. And
sometimes you have legitimate real-world needs that require threads
(most often libraries designed around that model or the two MAJOR
platforms, Java and Windows, without good multiprocess support, but
sometimes problem domains where you really NEED all kinds of crazy
shared memory data structures and segmenting them out doesn't buy you
much).
 
G

G. S. Hayes

Jeff Shannon said:
Threads seem to only be essential when
long-running background tasks are involved. There's a large set of GUI
programs where this is not an issue.



Threads aren't essential even there; processes are not only acceptable
but preferrable in most cases.



The rest of your post was spot-on, exactly the points I've been trying
to convey.
 
M

mudd

state machines and processes [and inter-process
communication/coordination]? It sounds like you're saying the alternative
to threads is "home brewed" threads. I don't follow.

Until recently linux implemented threads as processes. I'm really not
concerned with how the threads are implemented in the thread library. I
just don't want a language to prevent me from accessing the thread
library.
 
S

Skip Montanaro

Istvan> I think one of the latent messages of this thread was that
Istvan> adding new "gratuitous" features to the core language is a bad
Istvan> thing.

Istvan> Just because someone sits down and codes it does not mean it
Istvan> should be added to the language. Features are forever, you
Istvan> cannot just can't eliminate them in the next version.

...

Istvan> Python is said to come with the batteries included, then
Istvan> that's where new features should go, to the battery level
Istvan> not into the wiring.

The notion of decorated functions has been in Python since (at least)
staticmethod and classmethod were added as builtins. At that time it was
presumed some syntax would be developed in the future to make their use less
clunky (few people like that the "declaration" of a static method occurs at
the end of its body) if it turned out they were useful. They have been
found to be useful. We are now in the throes of the effort to define a
suitable syntax for such a construct.

Decorators are not theorhetical features with no proven use. They are used
today and will likely be used more widely once it is easier to do so.

Skip
 
S

Skip Montanaro

Anthony> As far as "fixing" the GIL - well, no-one on Python-Dev has
Anthony> obviously found it to be a problem for them.

I wouldn't state the case that strongly. There appear to be bigger fish for
the python-dev gang to fry up to this point. Perhaps if more of us were
scientific computation types the GIL would be seen as a more important
barrier to be removed. That a fairly bright guy (Greg Stein) removed it
once and found performance to be disappointing doesn't advocate strongly for
removal of the GIL either.

Skip
 
J

Jacob Hallen

Anthony> As far as "fixing" the GIL - well, no-one on Python-Dev has
Anthony> obviously found it to be a problem for them.

I wouldn't state the case that strongly. There appear to be bigger fish for
the python-dev gang to fry up to this point. Perhaps if more of us were
scientific computation types the GIL would be seen as a more important
barrier to be removed. That a fairly bright guy (Greg Stein) removed it
once and found performance to be disappointing doesn't advocate strongly for
removal of the GIL either.

Skip

So, what we really want to do is to remove the GIL, and keep it.

Ignoring trivialities like practical implementation, we can do this, since
the moment we start running our program, we will be able to find out
if we are running in an environment were we have access to multiple
processors (and if the program has the ability to use them (at least most
of the time)). If we can use multi-processing, we invoke a Python
interpreter with fine grained locking, otherwise we stick with the
GIL.

It is not really reasonable to have 2 different sets of source code
to maintain the 2 versions of the interpreter, but it might be possible
to make an abstraction of the locking that allows the source code
to compile into 2 different interpreters with different locking
strategies.

Practical implementation would probably be a very big project. Especially in
the case of the standard Python implementation.

Jacob Hallén

--
 
C

Christopher T King

So, what we really want to do is to remove the GIL, and keep it.

Ignoring trivialities like practical implementation, we can do this, since
the moment we start running our program, we will be able to find out
if we are running in an environment were we have access to multiple
processors (and if the program has the ability to use them (at least most
of the time)). If we can use multi-processing, we invoke a Python
interpreter with fine grained locking, otherwise we stick with the
GIL.

The real reason behind the GIL is that the Python interpreter is not
re-entrant; it keeps internal state in a global structure which must be
switched out (and stored somewhere) on thread changes. The real solution
to this problem is to make the interpreter stateless, thus obviating the
need for the GIL entirely. I think this task would be much easier to do
in Stackless than in CPython, but I may be wrong.
 
D

Daniel Dittmar

Christopher said:
The real reason behind the GIL is that the Python interpreter is not
re-entrant; it keeps internal state in a global structure which must be
switched out (and stored somewhere) on thread changes. The real solution
to this problem is to make the interpreter stateless, thus obviating the
need for the GIL entirely. I think this task would be much easier to do
in Stackless than in CPython, but I may be wrong.

Another reason is reference counting, which must be synchronized.

Daniel
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,772
Messages
2,569,593
Members
45,104
Latest member
LesliVqm09
Top