Lisp mentality vs. Python mentality

[Aahz]

And let's look at my recent experience with Python: I wanted to
implement a daemon process and stumbled at a simplest problem with
threading: neither Thread, nor Threading module provides thread-
killing possibility. Surely, I'm not so experienced in Python as in
Lisp (in which I'd definitely be able to solve this problem by
extending the library), but I don't see an obvious solution, which
will stay inside the language: I have to either use the shell or stick
to the limited set of provided options and skew my program design to
work with them. Any other suggestions?

The problem is that thread-killing (in the literal sense) doesn't work.
Unlike processes, there's no thread-environment encapsulation at the OS
level, which means that things don't get cleaned up properly. Even Java
has mostly given up on thread-killing. The only way to kill threads
safely is to have them terminate themselves. Your other option is to use
multiple processes.

 

[Aaron Brady]

That is a wonderful link.  Thanks for sharing.

--Scott David Daniels
(e-mail address removed)

import mentality? help( mentality ).

 

[Mikael Jansson]

Ultimately, Lisp is first and foremost academic (Scheme especially so)
while Python is first and foremost practical.  I think Paul Graham's
essays on Lisp exemplify the Lisp mentality.

Oh, you mean Blub?

http://www.paulgraham.com/avg.html

;-)

 

[Martin v. Löwis]

So if I'm reading right you are saying something in the lines:

"using too many functions is bad just because it is unreadable and
non-understandable to average (could I say mediocre?) programmers"...

No, this style is also unreadable to advanced programmers, in
particular when you fail to include the functions you use, and
leave the reader with guesswork.

Unfortunately I thought that delegating responsibilities to other
functions, and thus writing small chunks of code, is what good
software engineering is... Well my bad...

You misunderstood indeed. Splitting a large algorithm into smaller
parts is good if the original algorithm cannot be easily understood
in a single piece. By splitting it, you introduce new abstractions
that can help to simplify the original algorithm. OTOH, adding
these abstractions also requires the reader to familiarize himself
with the abstractions. There is a tradeoff, and the optimum is
somewhere in the middle.

Splitting code into tiny functions certainly is *not* good software
engineering.

Although you have a point -- that of being hard to comprehend by
average programmers -- but this doesn't mean it is a wrong (as in
ugly) solution... Also, with respects, but the "pythonic" solution
involving generators (or iterators) and "zip" or "all" function --
although I appreciate it as it comes close to FP -- is not what I
would call readable and understandable by non-guru programmers...

Please go back to my formulation - it doesn't use any of this, and
I agree that the formulations with zip, generators, etc are indeed
more difficult to read (perhaps *precisely* because they come
closer to FP).

Regards,
Martin

 

[Vsevolod]

The problem is that thread-killing (in the literal sense) doesn't work.
Unlike processes, there's no thread-environment encapsulation at the OS
level, which means that things don't get cleaned up properly. Even Java
has mostly given up on thread-killing. The only way to kill threads
safely is to have them terminate themselves. Your other option is to use
multiple processes.

Well, somehow, in Lisp it's not a problem.

Cheers,
Vsevolod

 

[Aahz]

I don't remember who, but something famously said, in effect:
Debugging is hard, maybe twice as hard as writing the code in
the first place. Unless you are one of those nonexistent few
who always write correct programs from the word go, you will
have to debug your own code before it works fully correctly.
Therefore, you had better write code so simple that you can
know what is going wrong with it. If your code is too hard
to understand for the average programmer, you are either four
times as brilliant as those "average" programmers or you are
in big trouble.

Fom my .sig database:

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it." --Brian W. Kernighan

 

[Carl Banks]

I think you're exaggerating. Go ask this question in c.l.l and the
first answer you'll get is mismatch.

What could have made you think I was exaggerating. Could it be the
line where I said "Carl, Banks who mighe exaggerating"??

But, from the other point of view your exaggeration makes sense: Lisp
unlike Python, IMO, is the language, where it's pleasant to program
not only applications, but the internals as well. So some people may
find interest in reprogramming what's already there. In lisp
programmer's mentality it's good to know, that you have that ability.

And yet, the impression I get from Lisp programmers is that they
prefer to program internals. They will say it's an advantage of Lisp
that their list type is handled with low-level primitives like car,
cdr, and cons, rather than high-level operations like indexing,
appending, etc. Why would that be an advantage unless they are doing
that kind of stuff all the time?

And let's look at my recent experience with Python: I wanted to
implement a daemon process and stumbled at a simplest problem with
threading: neither Thread, nor Threading module provides thread-
killing possibility. Surely, I'm not so experienced in Python as in
Lisp (in which I'd definitely be able to solve this problem by
extending the library), but I don't see an obvious solution, which
will stay inside the language: I have to either use the shell or stick
to the limited set of provided options and skew my program design to
work with them. Any other suggestions?

Say you are running a thread and you want the power to be able to kill
it at any time. The thread is either communicating with the rest of
the program periodically, or it isn't. If it is, then there are ample
opportunities to tell the thread to terminate itself. If it isn't,
then you might as well use a separate process which you can kill.
Next question.


Carl Banks

 

[Paul Rubin]

Say you are running a thread and you want the power to be able to kill
it at any time. The thread is either communicating with the rest of
the program periodically, or it isn't. If it is, then there are ample
opportunities to tell the thread to terminate itself. If it isn't,
then you might as well use a separate process which you can kill.

That's not so satisfying. If you use a separate process, it can't
share Python objects with the main process, isn't under the same
memory management, etc. With the standard modules that comes with
Python, you can't share memory at all (except with mmap, which gives
no synchronization mechanisms). You can't pass open sockets from one
process to another with the standard library, making it harder to
implement typical multi-threaded servers. You do get better
scalability, but at the expense of having to serialize all IPC data
and use heavyweight communication mechanisms. Threads exist because
they are useful.

 

[Carl Banks]

That's not so satisfying.  If you use a separate process, it can't
share Python objects with the main process,

Which is "communicating with the rest of the program periodically".

Presumably you have to protect objects to share them? There you go:
anytime you try to acquire a lock have the thread check to see whether
to abort.


Carl Banks

 

[Paul Rubin]

Which is "communicating with the rest of the program periodically".

Presumably you have to protect objects to share them? There you go:
anytime you try to acquire a lock have the thread check to see whether
to abort.

Normally, acquiring a lock doesn't require running code in other
threads, at least in the uncontended case. If you have to switch
threads twice in order to acquire a lock, your implementation could
use some improvement.

 

[Paul Rubin]

But precisely because of that sharing the thread may be in the middle
something that "must complete" -- no with-statement locking will get
unlocked, no "finally:" clauses in code in the standard library, no
... -- there is just too much that goes wrong when a thread is
infinitely starved (which, in effect, is what would happen if you
could kill it).

Right, it's better to recognize this sort of problem than to brush it
off. There was a discussion in sourceforge a long time ago about
adding a way to raise exceptions in threads, and I think there is some
way to do it through the C API.

 

[Dan Sommers]

So if I'm reading right you are saying something in the lines:
"using too many functions is bad just because it is unreadable and
non-understandable to average (could I say mediocre?) programmers"...
Unfortunately I thought that delegating responsibilities to other
functions, and thus writing small chunks of code, is what good software
engineering is... Well my bad...

Also from the Zen: flat is better than nested. One of the aspects of
flatter call trees and object hierarchies is that I hit the bottom
(language features or the standard library) sooner, and I "should"
already have the language and its standard library in my brain. That
said, I also tend to break my programs into layers, but I do try to make
each layer as thin as possible (but no thinner).

Although you have a point -- that of being hard to comprehend by
average programmers -- but this doesn't mean it is a wrong (as in ugly)
solution... Also, with respects, but the "pythonic" solution involving
generators (or iterators) and "zip" or "all" function -- although I
appreciate it as it comes close to FP -- is not what I would call
readable and understandable by non-guru programmers...

I finally got it through my thick head that I've been *doing* functional
programming with [*nix] shells and pipes for years, well back into my non-
guru days.

Dan

 

[Steven D'Aprano]

Paul Rubin:

That works and is nice and readable:


import operator
from itertools import imap

def equal_sequences(a, b, comp=operator.eq):

[snip]

Sorry, this is worse than unreadable. It is *misleading*. It doesn't test
for equal sequences except as a special case. What it does is perform a
generic element-by-element comparison, not necessarily an equality test.

What was wrong with the name compare() used in previous posts in this
thread?

 

[Aahz]

Well, somehow, in Lisp it's not a problem.

Does Lisp even have OS-level threads? What Lisp are you using, on what
OS? Are they perhaps Erlang-style cooperative threads instead?

 

[Carl Banks]

Normally, acquiring a lock doesn't require running code in other
threads, at least in the uncontended case.  If you have to switch
threads twice in order to acquire a lock, your implementation could
use some improvement.

Come on, you're just making stuff up. How the *hell* do you get
switching threads twice out of that? I'm saying that if threads have
to synchronize data, then it's no big deal to have the thread check a
flag over whether to terminate when synchronizing. Take a look at
this recipe for an example:

http://code.activestate.com/recipes/576461/

That concept, I say, can be applied to any thread that shares data
with the main thread, with any sort of regularity, even if it's
doesn't use queues specifically.

The only time it's any trouble to notify a thread to terminate is when
that thread's running code that's not aware it's in a threaded app
(which implies that it's not sharing any data, aside from read-only).
But since it's not communicating with another threads you might as
well run it in another process; communication overhead is not going to
be a big deal in that case.


Carl banks

 

[Ciprian Dorin, Craciun]

Also from the Zen:  flat is better than nested.  One of the aspects of
flatter call trees and object hierarchies is that I hit the bottom
(language features or the standard library) sooner, and I "should"
already have the language and its standard library in my brain.  That
said, I also tend to break my programs into layers, but I do try to make
each layer as thin as possible (but no thinner).

I agree with your opinion about keeping the abstraction layers
shallow, but in my view high-order and helper functions do not
comprise a new abstraction layer. For example in Lisp, using map,
reduce (fold), or any other high-order function is just like using
for, or while in a normal imperative language.

Another example is the simple Observer pattern. How many think of
it as a new layer? Instead anytime we see a method that is named
add_listener we already have an idea of what it does... So we could
say that in FP world there are some patterns that involve "delegating
control" to them.

And also I would like to point out that "hitting the language
sooner", means to know every function in the standard Python library
(which is by far uncomprehensible, its huge) and most of the times you
also need the documentation. And if we go this path, when debugging we
could use a smart IDE, which should show as tool-tip-text for function
names their documentation, and in this way all we have to do to
understand a particular function is just to point it out.

Ciprian.

    Although you have a point -- that of being hard to comprehend by
average programmers -- but this doesn't mean it is a wrong (as in ugly)
solution... Also, with respects, but the "pythonic" solution involving
generators (or iterators) and "zip" or "all" function -- although I
appreciate it as it comes close to FP -- is not what I would call
readable and understandable by non-guru programmers...

I finally got it through my thick head that I've been *doing* functional
programming with [*nix] shells and pipes for years, well back into my non-
guru days.

Dan

 

[Vsevolod]

Does Lisp even have OS-level threads? What Lisp are you using, on what
OS? Are they perhaps Erlang-style cooperative threads instead?
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"If you think it's expensive to hire a professional to do the job, wait
until you hire an amateur." --Red Adair

Different Lisp implementations provide different solutions. SBCL
provides OS-level threads (on Linux), which I personally use, while
CMUCL offers green threads. Allegro, LispWorks, Clozure CL, Sceineer
CL and ECL as well have threading, but I don't use them, so won't
speak, which implementation of threading they have. There's a common
unification library -- bordeaux-threads -- that abstracts away
implementation specifics. It's API includes the function destroy-
thread.

As well I'd like to outline, that, IMO, your answer exhibits the
common attitude among pythonistas: everything should be done in one
true way, which is the best option (and that is how it's implemented
in the current version of the language). As of PEP-20: "There should
be one-- and preferably only one --obvious way to do it. Although that
way may not be obvious at first unless you're Dutch." And if someone
disagrees -- he just doesn't understand...

Cheers,
Vsevolod

 

[Richard Brodie]

There's a common unification library -- bordeaux-threads --
that abstracts away implementation specifics. It's API includes
the function destroy-thread.

Which is deprecated, like the Java one. It's not hard to provide
a kill thread call, if you don't mind it having undefined semantics.

 

[Vsevolod]

Which is deprecated, like the Java one. It's not hard to provide
a kill thread call, if you don't mind it having undefined semantics.

"This should be used with caution: it is implementation-defined
whether the thread runs cleanup forms or releases its locks first."
This doesn't mean deprecated. It means: implementation-dependent. For
example in SBCL: "Terminate the thread identified by thread, by
causing it to run sb-ext:quit - the usual cleanup forms will be
evaluated". And it works fine.

Best regards,
Vsevolod

 

[Marco Mariani]

I don't remember who, but something famously said, in effect:
Debugging is hard, maybe twice as hard as writing the code in
the first place. Unless you are one of those nonexistent few

He would be the K in K&R.