Article of interest: Python pros/cons for the enterprise

[Paul Rubin]

def mkadder(n):
return lambda x: x + n

I have gotten the impression that this was somehow inferior in Python
though, at least in terms of performance. Every time somebody uses
lambda here, they seem to get a bunch "why are you using lambda?"
responses. If I am grossly mistake, please just enlighten me.

There are some Python users who object to lambda on stylistic
grounds, and would rather that you say

def mkdadder(n):
def adder(x, n):
return x + n
return adder

Those same users tend to object to higher-order functions (preferring
listcomps or genexps). It seems to me though that Python is moving
more towards HOF-oriented styles because of the utility of HOF's over
iterators (look at itertools). That might make its learning curve
steeper for beginners; I don't have a good sense of this.

I note from your other posts that you seem to have a strong Lisp
bent. Lisp programmer and Smalltalk programmers stand out in the
crowd. I first noted this when somebody found a while-loop offensive,
on the grounds that recursion was somehow a more natural way to
implement looping. It can take a while to convince somebody like that
they different idioms work best in different languages.

Well, the saying is "to iterate is human; to recurse, divine" ;-).
Scheme has while-loops but they are just syntax sugar for recursion.

 

[Paul Rubin]

Mercy, a whole book?
http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?EAN=9780201179286


On what grounds? "I don't approve of the programming language you're
using. Cease and desist. Nyah!"

Well, "outlawed" of course is a figure of speech; however use of those
languages should be a red flag in a design review, and a factor in
determining product liability if a program misbehaves. Think of all
the buffer overflow exploits against Microsoft Windows programs that
couldn't have happened if Microsoft had used safer languages. There
was clearly a failure to use best engineering practices.

 

[Jeff Schwab]

Well, "outlawed" of course is a figure of speech; however use of those
languages should be a red flag in a design review, and a factor in
determining product liability if a program misbehaves.

Your product gets the benefit of the doubt if it was written in Python,
but not if written in C? What if the Python interpreter screws up?

Think of all
the buffer overflow exploits against Microsoft Windows programs that
couldn't have happened if Microsoft had used safer languages. There
was clearly a failure to use best engineering practices.

In the first place, you're not distinguishing C from C++. I know you
think they're the same. They're not. Python's array bounds checking is
done in C, as is the case for traditional OS kernels. C++, like Python,
offers you container types that "smell" like arrays, but do safety
checking under the hood; the C++ bounds checks, unlike the Python
checks, are done in-language. If C++ were marketed like Java, people
might advertise that the C++ standard containers are "pure C++," or that
"C++ is written in C++".

Secondly, you're going to have a hard time writing an operating system
kernel without fixed-size buffers. You have to read data in chunks of
some size. It's fine to say that buffer access should be checked to
make sure indexes should be in range, but somebody actually has to write
the code to do the checking, and then programmers have to use it.
Buffer overflows are the result of developers deliberately choosing not
to use bounds-checking.

In Python, somebody could still define a monolithic array, use different
parts of it to represent different things, and then "oops" lose track of
their index, thereby overwriting one part of the array with data that
should have stayed in the other part. If you're thinking "Yeah, they
*could*, but only a dodo would do that, and it's not Python's fault that
the programmer did something bone-headed," then you're beginning to get
the picture. If you somehow wrote an OS kernel in Python, and the
above-mentioned bone-headedness took place, then there would be no
"lower level" to save your OS from having a dangerous security hole.
Even if the hardware supported checked access for large quantities of
arbitrarily sized arrays, somebody would have to enable that access.
Even then you're trusting the hardware developers to check for invalid
indexes (bus errors and seg faults). Should we outlaw computer
hardware, too?

Thirdly, you mentioned a "failure to use best engineering practices."
If you mean there was a failure to design and implement a bug-free
operating system, then by Jiminy, you're right. "Best practices,"
though, is usually just what managers say when they mean "what everybody
else is doing." The idea is that if you're a manager who sticks to
"best practices," then if everything goes to hell, the problem can't be
linked specifically to you.

Server-side Python is now, at this very moment, becoming acceptable as
"best practice." It's no better or worse for this; it's just that a lot
more uninspired, uncreative programmers are about to start using it. Do
not underestimate their ability to make any programming language look
bad. If you make it fool-proof, they'll build a better fool. The only
way to keep them from writing code that will blow up at run-time is to
impose formal verification of their program, every time they change it;
that's what the C++ static type system lets you do, in a limited and
easily subverted way. Even if you could somehow remove every feature
open to potential abuse, and thereby dodo-proof the language, the result
would be a slow, weak, ugly language that non-dodos would not want to
use. But hey, at least it would still be "best practice."

 

[Paul Rubin]

Your product gets the benefit of the doubt if it was written in
Python, but not if written in C? What if the Python interpreter
screws up?

The Python interpreter has been run in production for over a decade
modulo comparatively small incremental changes, heavily reviewed and
tested by multiple people, between versions. It's likely to be more
reliable than any one-off C program of comparable size. Further, for
a C application, the analogous situation to the Python interpreter
screwing up would be the C compiler screwing up (generating wrong
code). They have that possibility in common. But for the C
application, the screwup can be in two places (the compiler or the
application) while Python limits it to one place. The principle here
is to minimize the size of the so-called trusted codebase.

Either way, the ongoing effort (PyPy) to write Python in Python will
probably improve Python's reliability.

In the first place, you're not distinguishing C from C++. I know you
think they're the same. They're not. Python's array bounds checking
is done in C, as is the case for traditional OS kernels. C++, like
Python, offers you container types that "smell" like arrays, but do
safety checking under the hood;

Not as long as you can have pointers to those objects that contain
naked memory addresses which can be invalid.

Buffer overflows are the result of developers deliberately
choosing not to use bounds-checking.

Yes, the most common way that choice is expressed is by choosing to
code in C or C++ in the first place.

to enable that access. Even then you're trusting the hardware
developers to check for invalid indexes (bus errors and seg faults).
Should we outlaw computer hardware, too?

Again, the concept is "best practices" which is an engineering term,
not a mathematical one. Best practices are not infallible, they're
just the best normal practices observed in real-world projects of
similar scope.

"Best practices," though, is usually just what managers say when
they mean "what everybody else is doing."

No really, among other things it includes following standards that
are designed by people who know what they're doing. That actually
means something.

The idea is that if you're a manager who sticks to "best practices,"
then if everything goes to hell, the problem can't be linked
specifically to you.

That is a very valuable aspect. If I'm writing an encryption program,
best practice is to use a standard algorithm like AES instead of
concocting my own algorithm. If something does turn out to be wrong
wtih AES that makes my program fail, I'm very glad to be able to point
to the rigorous standardization process that AES went through and say
that I made the best choice I could with available information. If
for some reason I think there's an 0.002% chance that something is
wrong with AES but only an 0.001% chance that something is wrong with
my home-cooked algorithm, AES is still the obvious choice.

Same thing with internet applications, I'm much better off writing in
a type-safe language than relying on the intricacies of my unique
application code to behave properly in a language without type-safety.

The only way to keep them from writing code that will blow up
at run-time is to impose formal verification of their program, every
time they change it; that's what the C++ static type system lets you
do, in a limited and easily subverted way.

Why not get rid of the limitations and easy subversions? And I can't
agree that using a basic type like "int x[10];" is somehow a
subversion. Subversion means you have to go to some lengths to break
the type safety.

Even if you could somehow remove every feature open to potential
abuse, and thereby dodo-proof the language, the result would be a
slow, weak, ugly language that non-dodos would not want to use. But
hey, at least it would still be "best practice."

Enforcing type safety is certainly an enormous step towards
dodo-proofing a language, it hugely helps the reliability of real
programs, and is one of the reasons Python and Java are displacing C++.

 

[Steven D'Aprano]

Every time somebody uses
lambda here, they seem to get a bunch "why are you using lambda?"
responses.

Not from me.

I even use "named anonymous functions" *cough* by assigning lambda
functions to names:

foo = lambda x: x+1

 

[Steven D'Aprano]

Who feeds you this stuff?



Mercy, a whole book?

Well, that might not be "zillions", but it is sure a lot of pitfalls.

My current favorite book of language-specific pitfalls:

http://www.javapuzzlers.com/

I'm not sure what you are trying to communicate here.

Wait a few years. A Python Puzzlers book will surely be on the shelves
eventually.

You wish.

If so, then it will either be an incredibly short book, or the language
"Python" in 2012 will be nothing like Python now.

Here are some of the top results when Googling "python
pitfalls:"

http://zephyrfalcon.org/labs/python_pitfalls.html
http://evanjones.ca/python-pitfall-scope.html

Yeah, let's have a look at those. From the first link:

1. Inconsistent indentation
2. Assignment, aka names and objects
3. The += operator
4. Class attributes vs instance attributes
5. Mutable default arguments
6. UnboundLocalError
7. Floating point rounding errors
8. String concatenation
9. Binary mode for files
10. Catching multiple exceptions

Ten pitfalls. You might be able to pad that out to a twenty page leaflet,
including an index. Number ten is already gone from Python 3; numbers 7
and 9 aren't Python specific at all; most of the others are only a
pitfall if you come to Python thinking that Python is C or Java, or
whatever other language you're used to.

The second link is just bizarre. It claims that Python "does not have
nested, block variable scopes like I am accustomed to in nearly every
other programming language I use, like C, C#, C++, Objective C, Perl, and
Java." (Notice the common denominator?) But that's nonsense -- Python
does have nested block variable scopes. This "pitfall", if it is one, is
that Python doesn't create a new scope for if-blocks. Coming from a
Pascal background instead of C, it never even occurred to me that if-
blocks could be a new scope, or that anyone would possibly want it to be.

Again, the real pitfall is not Python, but imagining that Python is C.

Maybe C++ needs better pub. The guy who wrote the first of those
articles says elswhere on his web site:

"My Python pitfalls article seems to be a popular read. It's
written from a somewhat negative viewpoint: things that can go
wrong. (Of course, that's useful; you're never going to do these
things wrong again. Right? ;-) To counter-balance this, I think
there should an article with a more positive viewpoint as well. So
I was wondering, if I could collect 10 "anti-pitfalls"; parts of
the Python language that are especially clever, elegant and
intuitive."


Good luck with that.

I've seen Python criticized as "ugly" precisely because it doesn't have a
trick-based view of the world. In many ways, it's a dull language,
borrowing solid old concepts from many other languages & styles: boring
syntax, unsurprising semantics, few automatic coercions, etc etc. But
that's one of the things I like about it. -- Tim Peters

But even given that, here's a good start, the first four things that came
to mind:

1. Program structure follows indentation. No more wars over where to put
the braces. The presentation *is* the structure.

2. Namespaces.

3. Generators and iterators. Especially generators.

4. Exceptions. try blocks are fast, don't be scared to use them.

 

[Arnaud Delobelle]

On Sat, 23 Feb 2008 19:45:45 -0800, Jeff Schwab wrote:
The second link is just bizarre. It claims that Python "does not have
nested, block variable scopes like I am accustomed to in nearly every
other programming language I use, like C, C#, C++, Objective C, Perl, and
Java." (Notice the common denominator?) But that's nonsense -- Python
does have nested block variable scopes. This "pitfall", if it is one, is
that Python doesn't create a new scope for if-blocks. Coming from a
Pascal background instead of C, it never even occurred to me that if-
blocks could be a new scope, or that anyone would possibly want it to be.

I haven't read the link, but it is true that python < 3 is deficient
in that area: names in an enclosing scope can be accessed but not
rebound. In the latter case the interpreter assumes them to be local
names. Python 3000 fixes this with the new 'nonlocal' compilation
directive. E.g.

(python 2.5)
... def tick():
... acc += step
... return acc
... return tick
...Traceback (most recent call last):
File "<stdin>", line 1, in <module>

(python 3.0)
... def tick():
... nonlocal acc
... acc += step
... return acc
... return tick
...

 

[Matthew Woodcraft]

No, why would you think so? If you want something special to happen
when the close() fails (as you indeed would if you were writing
important data), you have to say somehow that you want your special code
called. What syntax would you like to see? Here's what the C++ would
look like, supposing you have a type LoggingCloser that calls close and
logs any failure:

Sure, but this LoggingCloser isn't what you described as the 'most
traditional, easiest way to open a file in C++'.

The more general point is that, where you want to take action on
cleanup and you need to handle errors from this action, it usually
isn't enough to have one class per type of resource which knows what
clean up action is required (because the required response to the error
usually varies).

Additionally, in languages where exceptions are the usual way to report
errors but exceptions from destructors are not managed pleasantly, the
error recovery code can't use the language's normal mechanisms for
dealing with the error response.

-M-

 

[Lou Pecora]

Just some anecdotal confirmation:

I personally used C++ for about 90% of my code for 10 years. During that
time, I was chugging the C++ Kool-Aid so hard I almost peed myself. I still
think that C++ is a beautiful language, but I have also come to think that
starting a program with C++ is a premature optimization.

Yes, I came to the same conclusion and now start all projects in Python,
then add C/C++ extensions for optimization.

I think that very few Python programmers today started with Python. Most of
them came to Python for a reason.

Exactly right in my case. In fact this observation is directly related
to the one your the previous paragraph. Python is a good language in
which to start a progrom.

 

[Jeff Schwab]

Sure, but this LoggingCloser isn't what you described as the 'most
traditional, easiest way to open a file in C++'.

It's not there to open the file, it's there to check the error code on
close. That's what you said you wanted.

The more general point is that, where you want to take action on
cleanup and you need to handle errors from this action, it usually
isn't enough to have one class per type of resource which knows what
clean up action is required (because the required response to the error
usually varies).

If there is variation, you can either pass some data to the responsible
object's constructor, or you can give it "policies" for what to do in
different situations.

http://en.wikipedia.org/wiki/Policy-based_design

Additionally, in languages where exceptions are the usual way to report
errors but exceptions from destructors are not managed pleasantly, the
error recovery code can't use the language's normal mechanisms for
dealing with the error response.

Huh? Why not?

 

[Lou Pecora]

In C++ memory is just another resource which you can handle just like
any other one, possibly using RAII.

Ok, I'll bite. Here's a straightforward Python expression:

a = [f(x) + g(y) for x,y in izip(m1, m2) if h(x,y).frob() == 7]

Consider how many intermediate objects are being allocated in figuring
out that listcomp. Do you REALLY want to manage all the deallocation
with something like RAII?

What makes you think that a translation of a similar expression would
involve explicit dynamic allocation at all? Barring bugs, here's an
equivalent example:

#include <iostream>
#include <map>
#include <vector>

[cut a lot of C++ code]

I realize the original point was about dynamic allocation and GC, but
for me the raw juxtaposition of the *one* line of clear Python code with
the equivalent mass of C++ code is shocking. Thanks for that.

 

[Larry Bugbee]

No, disagree.

The main reason why C++ has declined in usage is because it never got
the kind of corporate marketing enjoyed by Java and C#.

I'm inclined to disagree for two reasons. C++ is a very complex
language. Java (and the later C#) less so. Couple that with reduced
debugging time due to garbage collection and fewer pointer problems, a
lot of us decided a factor of 2x in personal productivity was worth
it. Runtime was initially an impediment, and still is for desktop
applications, but the trade was worth it.

Corporate marketing, and corporate attention in general, saw to it
that Java was well equipped with libraries and frameworks addressing
enterprise application needs. ...but the *big* reason Java won over C+
+ is because your application became stable sooner. ...with arguably
fewer problems later.

And the migration to Python is due in large part because of an
additional factor of 3-4x in personal productivity (over Java).
Improvements in runtime performance wouldn't hurt, but for many
applications that's not an issue. (If optional data typing were
offered, Python's penetration in the enterprise space would be even
higher, and I suspect there would be performance gains as well.)

Larry

 

[Jeff Schwab]

And the migration to Python is due in large part because of an
additional factor of 3-4x in personal productivity (over Java).
Improvements in runtime performance wouldn't hurt, but for many
applications that's not an issue. (If optional data typing were

You mean static data typing, right? Are there any known holes in the
dynamic type system that still need to be plugged? (I haven't heard of
any.)

 

[Larry Bugbee]

You mean static data typing, right? Are there any known holes in the

dynamic type system that still need to be plugged? (I haven't heard of
any.)

My apologies. You are right, I meant optional, static typing. Thanks
for the catch Jeff.

Python's dynamic typing is just fine. But if I know the type, I want
the ability to nail it. ...local variables, arguments, return values,
etc And if I don't know or care, I'd leave it to dynamic typing.

The need for a lot of doc goes away, those new to the language that
think it's a big deal can be explicit, and as I suggested earlier, I
think the byte code interpreter could be made a lot smarter and
faster. ...and corporate acceptance would follow.

Larry

 

[Larry Bugbee]

PS: And tools like ShedSkin and Pyrex benefit.

 

[Aahz]

What I would like is not so much a new Python implementation, as a
vehicle to help demonstrate a few things to other Python users.
Recently, I've had a few replies in tones that imply I'm on the brink of
entering several kill-files, mostly because I express disagreement with
a few closely held beliefs of some other c.l.p posters. For example,
the following are widely held opinions with which I disagree:

(1) Python is a gotcha-free language.

HAHAHAHAHAHAHA -- please, do provide some evidence for this assertion.

(2) C++ is basically the same language as C, but more complicated.

Hardly. Again, you need to provide evidence for this assertion. What I
would say is that C++ has *all* the gotchas of C plus many of its own.

(3) Garbage collection is at least as desirable a language feature as
deterministic destruction.

Enh. There probably are some people who claim that, but I can't think
of any off-hand. However, I suspect that part of the problem is that
you don't really understand Python: Python primarily relies on reference
counting, which combined with other features in Python makes cycle
creation rather easy, and there's no real option other than garbage
collection for cleaining up cycles.

(4) Static typing is inferior to dynamic typing.

There's some truth to what you say, but you are entirely misrepresenting
what people actually say: static typing is inferior to unit tests, and
dynamic typing requires less boilerplate than static typing and is
therefore easier to write.

In other ways, though, the Python community is just blindingly ignorant,
arrogant, and argumentative.

Absolutely. But your blind ignorance and arrogant argumentation isn't
helping any.

I expect my use of Python to increase in the coming years, so I want
the best possible relationship with other regular users, especially on
Usenet.

Then maybe you should start by learning Python and what people actually
say about it.

 

[Terry Reedy]

| On Sat, 23 Feb 2008 19:35:30 -0800, Jeff Schwab wrote:
|
| > Every time somebody uses
| > lambda here, they seem to get a bunch "why are you using lambda?"
| > responses.

I think you are overgeneralizing ;-)
I use 'em.

| Not from me.
|
| I even use "named anonymous functions" *cough* by assigning lambda
| functions to names:
|
| foo = lambda x: x+1

Even though I consider the above to be clearly inferior to

def foo(x): return x+1

since the latter names the function 'foo' instead of the generic
'<lambda>'.
The only other situation in which I would critisize lambda usage are
multi-line (or super-long-line) monstrousities that are not intentionally
'obfuscated Python'.

tjr

 

[Carl Banks]

Enh. There probably are some people who claim that, but I can't think
of any off-hand.

I am most certainly claiming it; in fact I'm claiming that GC far more
desirable, because the cost of deterministic destruction is too high.

The costs are:
1. extra effort (which Jeff Schwab denied but I don't remotely buy it)
2. mistakes are easier to make and more catastrophic when made

With deterministic destruction, you have to worry about the lifetime
of every single object, and that requires effort.

Either you are literally keeping track of all the object and manually
freeing and deleting them like in C, or you are sticking to a rigid
disciplined style that (mostly) guards against practices that lead to
invalid pointers and memory leaks. The discipline itself takes
effort, but the rigid style also limits expressiveness, often
severely, and so you have to expend more effort writing the same
thing.

Point is, one way or another, you are paying for deterministic
destruction with more effort.

Even that innocuous object on the local stack frame can bite you if
you pass it to a function that, expecting a heap object, stores a
pointer. Which means you have to keep track of what the functions
you're calling do, and you have to be deliberate about what the
functions you're writing do, so still more effort, yadda yadda.

I don't really think I have to say much with regard to error-proneness
compared to GC, nor the cost of mistakes Invalid pointer dereferences
and segfaults don't happen in Python (aside from extensions), and
memory leaks only happen when you accidentally keep objects alive.
Mistakes are far more common and dangerous in C++.

And subtle I might add. A destroyed object that is improperly
derferenced might "work" for awhile since its memory hadn't been
overwritten, then suddenly crash and burn in slightly different
circumstances (such as if it's running a different computer, like, for
instance, the customer's).

And tracking down and fixing those bugs, compared to the sorts of bugs
you get from garbage collection? You guessed it--more effort.

Deterministic destruction would be a great thing if it weren't for all
the drawbacks. Until then I will choose the path of lesser effort by
letting GC take care of memory, closing my own files, and having lots
of expressive power and freedom.


Carl Banks

 

[Nicola Musatti]

a = [f(x) + g(y) for x,y in izip(m1, m2) if h(x,y).frob() == 7] [...]
There you replace one line of code with 40+ lines to get around the
absence of GC. Sounds bug-prone among other things.

Come on, you didn't define f, g, izip, h or frob either. It's more
like 5 to one. Furthermore your code is more compact due to the
existence of list comprehensions, not because of GC. Had you written a
loop the difference would be smaller.

No I don't need a loop if I don't use the listcomp. I could say

a = map(lambda x,y: f(x)+g(y), ifilter(lambda x,y: h(x,y).frob==7,
izip(m1, m2)))

the listcomp is basically syntax sugar for that.

The issue here is that Python is simply more expressive than C++,
which doesn't support creation of higher order functions like that.

I see. The C++'s closest equivalent would probably be something like

int func(int x, int y) { return f(x) + g(y); }
bool filter(int x, int y) { return h(x,y).frob() == 7; }

transform_if(m1.begin(), m1.end(), m2.begin(),
std::back_inserter(a), func, filter);

Assuming that instead of izip you had the following, which is more in
style with how the C++ standard library usually works:

template <typename InIter1T, typename InIter2T,
typename OutIterT, typename FuncT, typename FilterT>
inline void transform_if(InIter1T begin1, InIter1T end1,
InIter2T begin2, OutIterT out, FuncT func, FilterT filter)
{
while ( begin1 != end1 )
{
if ( filter(*begin1, *begin2) )
*out++ = func(*begin1, *begin2);
++begin1;
++begin2;
}
}

Nameless, local function definitions and a form of type inference
should officially become part of C++ in little over a year. These will
help achieve closer expressiveness for similar statements, but the
lack of native tuples in the language will probably make it impossible
to get much closer.

However, one of the consequences of programming in this style is
you allocate a lot of temporary objects which best managed by GC.

According to which metric? This statement appears as totally
gratuitous to me. You seem to forget that deallocation of local
objects only entails stack readjustment. You can hardly beat that with
dynamic memory management.

Cheers,
Nicola Musatti

 

[Nicola Musatti]

No, disagree.


I'm inclined to disagree for two reasons. C++ is a very complex
language. Java (and the later C#) less so. Couple that with reduced
debugging time due to garbage collection and fewer pointer problems, a
lot of us decided a factor of 2x in personal productivity was worth
it. Runtime was initially an impediment, and still is for desktop
applications, but the trade was worth it.

While this was probably true towards the end of the nineties, given
the standard library and Boost I find it hard to believe that a
similar increase can be accounted for just in terms of language
differences.

Corporate marketing, and corporate attention in general, saw to it
that Java was well equipped with libraries and frameworks addressing
enterprise application needs. ...but the *big* reason Java won over C+
+ is because your application became stable sooner. ...with arguably
fewer problems later.

The number of libraries you get "out of the box" appear to me as more
likely explanations for the productivity increase.

And the migration to Python is due in large part because of an
additional factor of 3-4x in personal productivity (over Java).
Improvements in runtime performance wouldn't hurt, but for many
applications that's not an issue. (If optional data typing were
offered, Python's penetration in the enterprise space would be even
higher, and I suspect there would be performance gains as well.)

This I found less hard to believe. Python is more expressive than Java
and usually requires less code for the same task. Moreover tha
availability of libraries is comparable.

Cheers,
Nicola Musatti