Python's "only one way to do it" philosophy isn't good?

[Douglas Alan]

But then it should be slow for both generators and iterators.

Python *is* slow for both generators and iterators. It's slow for
*everything*, except for cases when you can have most of the work done
within C-coded functions or operations that perform a lot of work
within a single call. (Or, of course, cases where you are i/o
limited, or whatever.)

Or if the generators were built into the language and directly
supported by the compiler. In some cases implementing a feature is
*not* a simple case of writing a macro, even in Lisp. Generators may
well be one such case.

You can't implement generators in Lisp (with or without macros)
without support for generators within the Lisp implementation. This
support was provided as "stack groups" on Lisp Machines and as
continuations in Scheme. Both stack groups and continuations are
slow. I strongly suspect that if they had provided direct support for
generators, rather than indirectly via stack groups and continuations,
that that support would have been slow as well.

|>oug

 

[Douglas Alan]

"Python" doesn't *have* any refcounting semantics.

I'm not convinced that Python has *any* semantics at all outside of
specific implementations. It has never been standardized to the rigor
of your typical barely-readable language standards document.

If you rely on the behavior of CPython's memory allocation and
garbage collection you run the risk of producing programs that won't
port tp Jython, or IronPython, or PyPy, or ...

This is a trade-off that many users *are* willing to make.

Yes, I have no interest at the moment in trying to make my code
portable between every possible implementation of Python, since I have
no idea what features such implementations may or may not support.
When I code in Python, I'm coding for CPython. In the future, I may
do some stuff in Jython, but I wouldn't call it "Python" -- it'd call
it "Jython". When I do code for Jython, I'd be using it to get to
Java libraries that would make my code non-portable to CPython, so
portability here seems to be a red herring.

|>oug

 

[Lenard Lindstrom]

n Python, you can be 100% sure that your files
will be closed in a timely manner without explicitly closing them, as
long as you are safe in making certain assumptions about how your code
will be used. Such assumptions are called "preconditions", which are
an understood notion in software engineering and by me when I write
software.

So documenting an assumption is more effective than removing the
assumption using a with statement?

 

[Douglas Alan]

Douglas Alan wrote:

n Python, you can be 100% sure that your files
will be closed in a timely manner without explicitly closing them, as
long as you are safe in making certain assumptions about how your code
will be used. Such assumptions are called "preconditions", which are
an understood notion in software engineering and by me when I write
software.

So documenting an assumption is more effective than removing the
assumption using a with statement?

Once again I state that I have nothing against "with" statements. I
used it all the time ages ago in Lisp.

But (1) try/finally blocks were not to my liking for this sort of
thing because they are verbose and I think error-prone for code
maintenance. I and many others prefer relying on the refcounter for
file closing over the try/finally solution. Consequently, using the
refcounter for such things is a well-entrenched and succinct idiom.
"with" statements are a big improvement over try/finally, but for
things like file closing, it's six of one, half dozen of the other
compared against just relying on the refcounter.

(2) "with" statements do not work in all situations because often you
need to have an open file (or what have you) survive the scope in
which it was opened. You may need to have multiple objects be able to
read and/or write to the file. And yet, the file may not want to be
kept open for the entire life of the program. If you have to decide
when to explicitly close the file, then you end up with the same sort
of modularity issues as when you have to free memory explicitly. The
refcounter handles these sorts of situations with aplomb.

(3) Any code that is saving tracebacks should assume that it is likely
to cause trouble, unless it is using code that is explicitly
documented to be robust in the face of this, just as any code that
wants to share objects between multiple threads should assume that
this is likely to cause trouble, unless it is using code that is
explicitly documented to be robust in the face of this.

(4) Any code that catches exceptions should either return soon or
clear the exception. If it doesn't, the problem is not with the
callee, but with the caller.

(5) You don't necessarily want a function that raises an exception to
deallocate all of its resources before raising the exception, since
you may want access to these resources for debugging, or what have you.

|>oug

 

[Lenard Lindstrom]

Douglas Alan wrote:

n Python, you can be 100% sure that your files
will be closed in a timely manner without explicitly closing them, as
long as you are safe in making certain assumptions about how your code
will be used. Such assumptions are called "preconditions", which are
an understood notion in software engineering and by me when I write
software.

So documenting an assumption is more effective than removing the
assumption using a with statement?

Once again I state that I have nothing against "with" statements. I
used it all the time ages ago in Lisp.

Sorry if I implied that. I assumed it would be clear I was only
referring to the specific case of implicitly closing files using
reference counting.

But (1) try/finally blocks were not to my liking for this sort of
thing because they are verbose and I think error-prone for code
maintenance. I and many others prefer relying on the refcounter for
file closing over the try/finally solution. Consequently, using the
refcounter for such things is a well-entrenched and succinct idiom.
"with" statements are a big improvement over try/finally, but for
things like file closing, it's six of one, half dozen of the other
compared against just relying on the refcounter.

I agree that try/finally is not a good way to handle resources.

(2) "with" statements do not work in all situations because often you
need to have an open file (or what have you) survive the scope in
which it was opened. You may need to have multiple objects be able to
read and/or write to the file. And yet, the file may not want to be
kept open for the entire life of the program. If you have to decide
when to explicitly close the file, then you end up with the same sort
of modularity issues as when you have to free memory explicitly. The
refcounter handles these sorts of situations with aplomb.

Hmm. I come from a C background so normally don't think of a file object
as leading a nomadic life. I automatically associate a file with a home
scope that is responsible for opening and closing it. That scope could
be defined by a function or a module. But I'm not a theorist so can't
make any general claims. I can see, though, how ref count could close a
file sooner than if one waits until returning to some ultimate enclosing
scope.

(3) Any code that is saving tracebacks should assume that it is likely
to cause trouble, unless it is using code that is explicitly
documented to be robust in the face of this, just as any code that
wants to share objects between multiple threads should assume that
this is likely to cause trouble, unless it is using code that is
explicitly documented to be robust in the face of this.

Luckily there is not much need to save tracebacks.

(4) Any code that catches exceptions should either return soon or
clear the exception. If it doesn't, the problem is not with the
callee, but with the caller.

Explicitly clear the exception? With sys.exc_clear?

(5) You don't necessarily want a function that raises an exception to
deallocate all of its resources before raising the exception, since
you may want access to these resources for debugging, or what have you.

No problem:
def __exit__(self, exc_type, exc_val, exc_tb):
if exc_type is None:
file.__exit__(self, None, None, None)
return False


Traceback (most recent call last):
File "<pyshell#36>", line 1, in <module>
del f
NameError: name 'f' is not defined with MyFile("something", "w") as f:
raise StandardError("")
except StandardError:
print "Caught"


CaughtFalse


But that is not very imaginative:
def __exit__(self, exc_type, exc_val, exc_tb):
if exc_type is not None:
self.my_last_posn = self.tell()
return file.__exit__(self, exc_type, exc_val, exc_tb)


Traceback (most recent call last):
File "<pyshell#44>", line 1, in <module>
del f
NameError: name 'f' is not defined with MyFile("something", "w") as f:
f.write("A line of text\n")
raise StandardError("")
except StandardError:
print "Caught"


Caught16L

 

[Paul Rubin]

But that's a library issue, not a language issue. The technology
exists completely within Lisp to accomplish these things, and most
Lisp programmers even know how to do this, as application frameworks
in Lisp often do this kind. The problem is getting anything put into
the standard. Standardizing committees just suck.

Lisp is just moribund, is all. Haskell has a standardizing committee
and yet there are lots of implementations taking the language in new
and interesting directions all the time. The most useful extensions
become de facto standards and then they make it into the real
standard.

 

[Michele Simionato]

No I haven't been lucky -- I just know what I'm doing.








Right. So? I understand this issue completely and I code
accordingly.

What does it mean you 'code accordingly'? IMO the only clean way out
of this issue
is to NOT rely on the garbage collector and to manage resource
deallocation
explicitely, not implicitely. Actually I wrote a recipe to help with
this
a couple of months ago and this discussion prompted me to publish it:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/523007
But how would you solve the issue using destructors only? I am just
curious,
I would be happy if there was a simple and *reliable* solution, but I
sort
of doubt it. Hoping to be proven wrong,


Michele Simionato

 

[Douglas Alan]

Lisp is just moribund, is all. Haskell has a standardizing committee
and yet there are lots of implementations taking the language in new
and interesting directions all the time. The most useful extensions
become de facto standards and then they make it into the real
standard.

You only say this because you are not aware of all the cool dialetcs
of Lisp that are invented. The problem is that they rarely leave the
tiny community that uses them, because each community comes up with
it's own different cool dialect of Lisp. So, clearly the issue is not
one of any lack of motivation or people working on Lisp innovations --
it's getting them to sit down together and agree on a standard.

This, of course is a serious problem. One that is very similar to the
problem with Python vs. Ruby on Rails. It's not the problem that you are
ascribing to Lisp, however.

|>oug

P.S. Besides Haskell is basically a refinement of ML, which is a
dialect of Lisp.

P.P.S. I doubt that any day soon any purely (or even mostly)
functional language is going to gain any sort of popularity outside of
academia. Maybe 20 years from now, they will, but I wouldn't bet on
it.

 

[Paul Rubin]

P.S. Besides Haskell is basically a refinement of ML, which is a
dialect of Lisp.

I'd say Haskell and ML are descended from Lisp, just like mammals are
descended from fish.

 

[Douglas Alan]

What does it mean you 'code accordingly'? IMO the only clean way out
of this issue is to NOT rely on the garbage collector and to manage
resource deallocation explicitely, not implicitely.

(1) I don't rely on the refcounter for resources that ABSOLUTELY,
POSITIVELY must be freed before the scope is left. In the code that
I've worked on, only a small fraction of resources would fall into
this category. Open files, for instance, rarely do. For open files,
in fact, I actually want access to them in the traceback for debugging
purposes, so closing them using "with" would be the opposite of what I
want.

(2) I don't squirrel away references to tracebacks.

(3) If a procedure catches an exception but isn't going to return
quickly, I clear the exception.

|>oug

 

[Douglas Alan]

I'd say Haskell and ML are descended from Lisp, just like mammals are
descended from fish.

Hardly -- they all want to share the elegance of lambda calculus,
n'est-ce pas? Also, ML was originally implemented in Lisp, and IIRC
correctly, at least in early versions, shared much of Lisp's syntax.

Also, Scheme has a purely functional core (few people stick to it, of
course), and there are purely functional dialects of Lisp.

|>oug

 

[Paul Rubin]

Hardly -- they all want to share the elegance of lambda calculus,
n'est-ce pas? Also, ML was originally implemented in Lisp, and IIRC
correctly, at least in early versions, shared much of Lisp's syntax.

Haskell and ML are both evaluate typed lambda calculus unlike Lisp
which is based on untyped lambda calculus. Certainly the most
familiar features of Lisp (dynamic typing, S-expression syntax,
programs as data (Lisp's macro system results from this)) are absent
from Haskell and ML. Haskell's type system lets it do stuff that
Lisp can't approach. I'm reserving judgement about whether Haskell is
really practical for application development, but it can do stuff that
no traditional Lisp can (e.g. its concurrency and parallelism stuff,
with correctness enforced by the type system). It makes it pretty
clear that Lisp has become Blub.

ML's original implementation language is completely irrelevant; after
all Python is still implemented in C.

Also, Scheme has a purely functional core (few people stick to it, of
course), and there are purely functional dialects of Lisp.

Scheme has never been purely functional. It has had mutation since
the beginning.

Hedgehog Lisp (purely functional, doesn't have setq etc.) is really
cute. I almost used it in an embedded project but that got cancelled
too early. It seems to me more like a poor man's Erlang though, than
anything resemblant to ML.

 

[Douglas Alan]

Haskell and ML are both evaluate typed lambda calculus unlike Lisp
which is based on untyped lambda calculus. Certainly the most
familiar features of Lisp (dynamic typing, S-expression syntax,
programs as data (Lisp's macro system results from this)) are absent
from Haskell and ML.

And that is supposed to make them better and more flexible??? The
ideal language of the future will have *optional* manifest typing
along with type-inference, and will have some sort of pramgma to turn
on warnings when variables are forced to become dynamic due to there
not being enough type information to infer the type. But it will
still allow programming with dynamic typing when that is necessary.

The last time I looked at Haskell, it was still in the stage of being
a language that only an academic could love. Though, it was certainly
interesting.

Haskell's type system lets it do stuff that Lisp can't approach.

What kind of stuff? Compile-time polymorphism is cool for efficiency
and type safety, but doesn't actually provide you with any extra
functionality that I'm aware of.

I'm reserving judgement about whether Haskell is really practical
for application development, but it can do stuff that no traditional
Lisp can (e.g. its concurrency and parallelism stuff, with
correctness enforced by the type system). It makes it pretty clear
that Lisp has become Blub.

Where do you get this idea that the Lisp world does not get such
things as parallelism? StarLisp was designed for the Connection
Machine by Thinking Machines themselves. The Connection Machine was
one of the most parallel machines ever designed. Alas, it was ahead of
it's time.

Also, I know a research scientist at CSAIL at MIT who has designed and
implemented a version of Lisp for doing audio and video art. It was
designed from the ground-up to deal with realtime audio and video
streams as first class objects. It's actually pretty incredible -- in
just a few lines of code, you can set up a program that displays the
same video multiplied and tiled into a large grid of little videos
tiles, but where a different filter or delay is applied to each tile.
This allows for some stunningly strange and interesting video output.
Similar things can be done in the language with music (though if you
did that particular experiment it would probably just sound
cacophonous).

Does that sound like an understanding of concurrency to you? Yes, I
thought so.

Also, Dylan has optional manifests types and type inference, so the
Lisp community understands some of the benefits of static typing.
(Even MacLisp had optional manifest types, but they weren't there for
safety, but rather for performance. Using them, you could get Fortran
level of performance out of Lisp, which was quite a feat at the time.)

ML's original implementation language is completely irrelevant;
after all Python is still implemented in C.

Except that in the case of ML, it was mostly just a thin veneer on
Lisp that added a typing system and type inference.

Scheme has never been purely functional. It has had mutation since
the beginning.

I never said that was purely functional -- I said that it has a purely
functional core. I.e., all the functions that have side effects have
and "!" on their ends (or at least they did when I learned the
language), and there are styles of programming in Scheme that
discourage using any of those functions.

|>oug

P.S. The last time I took a language class (about five or six years
ago), the most interesting languages I thought were descended from
Self, not any functional language. (And Self, of course is descended
from Smalltalk, which is descended from Lisp.)

 

[Douglas Alan]

Explicitly clear the exception? With sys.exc_clear?

Yes. Is there a problem with that?

|>oug

 

[Douglas Alan]

P.S. The last time I took a language class (about five or six years
ago), the most interesting languages I thought were descended from
Self, not any functional language. (And Self, of course is descended
from Smalltalk, which is descended from Lisp.)

I think that Cecil is the particular language that I was most thinking
of:

http://en.wikipedia.org/wiki/Cecil_programming_language

|>oug

 

[Paul Rubin]

And that is supposed to make them better and more flexible???

Well no, by itself the absence of those Lisp characteristics mainly
means it's a pretty far stretch to say that Haskell and ML are Lisp
dialects.

The ideal language of the future will have *optional* manifest
typing along with type-inference, and will have some sort of pramgma
to turn on warnings when variables are forced to become dynamic due
to there not being enough type information to infer the type. But
it will still allow programming with dynamic typing when that is
necessary.

If I understand correctly, in Haskell these are called existential types:

http://haskell.org/hawiki/ExistentialTypes

The last time I looked at Haskell, it was still in the stage of being
a language that only an academic could love.

I used to hear the same thing said about Lisp.

What kind of stuff? Compile-time polymorphism is cool for efficiency
and type safety, but doesn't actually provide you with any extra
functionality that I'm aware of.

For example, it can guarantee referential transparency of functions
that don't live in certain monads. E.g. if a function takes an
integer arg and returns an integer (f :: Integer -> Integer), the type
system guarantees that computing f has no side effects (it doesn't
mutate arrays, doesn't open network sockets, doesn't print messages,
etc). That is very helpful for concurrency, see the paper "Composable
Memory Transactions" linked from here:

http://research.microsoft.com/Users/simonpj/papers/stm/index.htm

other stuff there is interesting too.

Where do you get this idea that the Lisp world does not get such
things as parallelism? StarLisp was designed for the Connection
Machine...

Many parallel programs have been written in Lisp and *Lisp, and
similarly in C, C++, Java, and Python, through careful use of manually
placed synchronization primitives, just as many programs using dynamic
memory allocation have been written in C with manual use of malloc and
free. This presentation shows some stuff happening in Haskell that
sounds almost as cool as bringing garbage collection to the
malloc/free era:

http://research.microsoft.com/~simonpj/papers/ndp/NdpSlides.pdf

As for where languages are going, I think I already mentioned Tim
Sweeney's presentation on "The Next Mainstream Programming Language":

http://www.st.cs.uni-sb.de/edu/seminare/2005/advanced-fp/docs/sweeny.pdf

It's not Haskell, but its type system is even more advanced than Haskell's.

 

[Lenard Lindstrom]

Yes. Is there a problem with that?

As long as nothing tries to re-raise the exception I doubt it breaks
anything:
raise StandardError("Hello")
except StandardError:
sys.exc_clear()
raise


Traceback (most recent call last):
File "<pyshell#6>", line 5, in <module>
raise
TypeError: exceptions must be classes, instances, or strings
(deprecated), not NoneType


But it is like calling the garbage collector. You are tuning the program
to ensure some resource isn't exhausted. It relies on implementation
specific behavior to be provably reliable*. If this is indeed the most
obvious way to do things in your particular use case then Python, and
many other languages, is missing something. If the particular problem is
isolated, formalized, and general solution found, then a PEP can be
submitted. If accepted, this would ensure future and cross-platform
compatibility.


* reference counting is an integral part of the CPython C api so cannot
be changed without breaking a lot of extension modules. It will remain
as long as CPython is implemented in C.

 

[Douglas Alan]

As long as nothing tries to re-raise the exception I doubt it breaks
anything:

raise StandardError("Hello")
except StandardError:
sys.exc_clear()
raise


Traceback (most recent call last):
File "<pyshell#6>", line 5, in <module>
raise
TypeError: exceptions must be classes, instances, or strings
(deprecated), not NoneType

I guess I don't really see that as a problem. Exceptions should
normally only be re-raised where they are caught. If a piece of code
has decided to handle an exception, and considers it dealt with, there
is no reason for it not to clear the exception, and good reason for it
to do so. Also, any caught exception is automatically cleared when
the catching procedure returns anyway, so it's not like Python has
ever considered a caught exception to be precious information that
ought to be preserved long past the point where it is handled.

But it is like calling the garbage collector. You are tuning the
program to ensure some resource isn't exhausted.

I'm not sure I see the analogy: Calling the GC can be expensive,
clearing an exception is not. The exception is going to be cleared
anyway when the procedure returns, the GC wouldn't likely be.

It's much more like explicitly assigning None to a variable that
contains a large data structure when you no longer need the contents
of the variable. Doing this sort of thing can be a wise thing to do
in certain situations.

It relies on implementation specific behavior to be provably
reliable*.

As Python is not a formally standardized language, and one typically
relies on the fact that CPython itself is ported to just about every
platform known to Man, I don't find this to be a particular worry.

If this is indeed the most obvious way to do things in your
particular use case then Python, and many other languages, is
missing something. If the particular problem is isolated,
formalized, and general solution found, then a PEP can be
submitted. If accepted, this would ensure future and cross-platform
compatibility.

Well, I think that the refcounting semantics of CPython are useful,
and allow one to often write simpler, easier-to-read and maintain
code. I think that Jython and IronPython, etc., should adopt these
semantics, but I imagine they might not for performance reasons. I
don't generally use Python for it's speediness, however, but rather
for it's pleasant syntax and semantics and large, effective library.

|>oug

 

[Douglas Alan]

No problem:

[...]
def __exit__(self, exc_type, exc_val, exc_tb):
if exc_type is not None:
self.my_last_posn = self.tell()
return file.__exit__(self, exc_type, exc_val, exc_tb)

I'm not sure I understand you here. You're saying that I should have
the foresight to wrap all my file opens is a special class to
facilitate debugging?

If so, (1) I don't have that much foresight and don't want to have
to. (2) I debug code that other people have written, and they often
have less foresight than me. (3) It would make my code less clear to
ever file open wrapped in some special class.

Or are you suggesting that early in __main__.main(), when I wish to
debug something, I do something like:

__builtins__.open = __builtins__.file = MyFile

?

I suppose that would work. I'd still prefer to clear exceptions,
though, in those few cases in which a function has caught an exception
and isn't going to be returning soon and have the resources generally
kept alive in the traceback. To me, that's the more elegant and
general solution.

|>oug

 

[Lenard Lindstrom]

I guess I don't really see that as a problem. Exceptions should
normally only be re-raised where they are caught. If a piece of code
has decided to handle an exception, and considers it dealt with, there
is no reason for it not to clear the exception, and good reason for it
to do so.

It is only a problem if refactoring the code could mean the exception is
re-raised instead of handled at that point. Should the call to exc_clear
be overlooked then the newly added raise will not work.

Also, any caught exception is automatically cleared when
the catching procedure returns anyway, so it's not like Python has
ever considered a caught exception to be precious information that
ought to be preserved long past the point where it is handled.

That's the point. Python takes care of clearing the traceback. Calls to
exc_clear are rarely seen. If they are simply a performance tweak then
it's not an issue *. I was just concerned that the calls were necessary
to keep resources from being exhausted.

I'm not sure I see the analogy: Calling the GC can be expensive,
clearing an exception is not. The exception is going to be cleared
anyway when the procedure returns, the GC wouldn't likely be.

The intent of a high level language is to free the programmer from such
concerns as memory management. So a call to the GC is out-of-place in a
production program. Anyone encountering such a call would wonder what is
so critical about that particular point in the execution. So
encountering an exc_clear would make me wonder why it is so important to
free that traceback. I would hope the comments would explain it.

It's much more like explicitly assigning None to a variable that
contains a large data structure when you no longer need the contents
of the variable. Doing this sort of thing can be a wise thing to do
in certain situations.

I just delete the name myself. But this is different. Removing a name
from the namespace, or setting it to None, prevents an accidental access
later. A caught traceback is invisible.

As Python is not a formally standardized language, and one typically
relies on the fact that CPython itself is ported to just about every
platform known to Man, I don't find this to be a particular worry.

But some things will make it into ISO Python. Registered exit handlers
will be called at program termination. A context manager's __exit__
method will be called when leaving a with statement. But garbage
collection will be "implementation-defined" **.

Well, I think that the refcounting semantics of CPython are useful,
and allow one to often write simpler, easier-to-read and maintain
code.

Just as long as you have weighed the benefits against a future move to a
JIT-accelerated, continuation supporting PyPy interpreter that might not
use reference counting.

I think that Jython and IronPython, etc., should adopt these
semantics, but I imagine they might not for performance reasons. I
don't generally use Python for it's speediness, however, but rather
for it's pleasant syntax and semantics and large, effective library.

Yet improved performance appeared to be a priority in Python 2.4
development, and Python's speed continues to be a concern.


* I see in section 26.1 of the Python 2.5 /Python Library Reference/ as
regards exc_clear: "This function can also be used to try to free
resources and trigger object finalization, though no guarantee is made
as to what objects will be freed, if any." So using exc_clear is not so
much frowned upon as questioned.

** A term that crops up a lot in the C standard /ISO/IEC 9899:1999 (E)/.