Does Python really follow its philosophy of "Readability counts"?

[Steven D'Aprano]

And then have to maintain a fork. No, thanks.

If you're messing with the internals, you have to do this anyway. As soon
as the internals change, your monkey-patched solution falls apart.

Why on earth couldn't I change the code of another member of my team if
that code needs changes ? The code is the whole team's ownership.

That's a model that works well when you have a small team of, say, a
dozen people. It's not a model that works when you have hundreds of
developers working on the project. The last thing I want in my projects
is cowboys who ride all over other people's code, changing internals of
parts they barely know at the drop of a hat, causing who knows what side-
effects.

But anyway, this is a red-herring. Data hiding isn't really about
managing the development process, it's about ensuring that when you and
Fred are using Barney's OrderedDict class in your code, Barney is free to
change his the implementation without your code suddenly failing. It's
also to ensure that any changes you make at runtime to the class don't
suddenly make Fred's code break.

 

[Russ P.]

Russ P. a écrit :

And then have to maintain a fork. No, thanks.

For crying out loud, how many private attributes do you need to
access? If it's a dozen, then you and your library developer are
obviously not on the same page. If it's one or two, then it's hardly a
"fork." Just take note of the one or two places where you needed to
remove the access restriction and you're done. Heck, you don't even
need to do that, because you will be warned automatically anyway when
you get the new version of the library (unless those private
attributes are changed to public).

Why on earth couldn't I change the code of another member of my team if
that code needs changes ? The code is the whole team's ownership.

OK, fine, you can change the code of another member of the team. Are
you going to check with him first, or just do it? The point is that
changing an interface requires agreement of the team members who use
that interface, whether on the calling or the implementation side of
it. If you change interfaces without getting agreement with the other
team members, you probably won't be on the team for long. And without
access restrictions, accessing _private is equivalent to changing the
interface.

Now and FWIW,  in this case (our own code), I just don't need to "mess
with internals" - I just just change what needs to be changed.


I totally fails to find any evidence of this assertion in the above
"demonstration".


Your opinion.


My my my. If you don't trust your programmers, then indeed, don't use
Python. What can I say (and what do I care ?). But once again, relying
on the language's access restriction to manage *security* is, well, kind
of funny, you know ?

Are you seriously saying that if you were managing the production of a
major financial software package with hundreds of developers, you
would just "trust" them all to have free access to the most sensitive
and critical parts of the program? Now *that's*, well, kind of funny,
you know?

Would you give all those developers your password to get into the
system? No? Wait a minute ... you mean you wouldn't "trust" them with
your password? But what about "openness"? Are you some sort of fascist
or what?

 

[Steven D'Aprano]

Are you seriously saying that if you were managing the production of a
major financial software package with hundreds of developers, you would
just "trust" them all to have free access to the most sensitive and
critical parts of the program? Now *that's*, well, kind of funny, you
know?

I think this is a red-herring. Probably my fault -- I was the first one
to mention access controls for banking software. I meant it as an analog
to data hiding, rather than implying that one can or should use private/
protected attributes to implement data hiding.

Private attributes are a form of data hiding, but not all data hiding can
be implemented as private attributes.

 

[Steve Holden]

Russ P. a écrit : [...]

Mr. D'Aprano gave an excellent example of a large banking program.
Without enforced encapsulation, anyone on the development team has
access to the entire program and could potentially sneak in fraudulent
code much more easily than if encapsulation were enforced by the
language.

My my my. If you don't trust your programmers, then indeed, don't use
Python. What can I say (and what do I care ?). But once again, relying
on the language's access restriction to manage *security* is, well, kind
of funny, you know ?

I have to say that I thought the example was somewhat bogus. Any
development team that is even slightly concerned about the possibility
of logic bombs in the code will try to mitigate that possibility by the
use of code inspections.

You might have a chance to sell this to a clueless pointy haired boss -
I mean, that enforced access restriction will make Python more suitable
for some big enterprizey project. As far as I'm concerned, I'm not
buying it.

I'm not sure that there's much to be gained by this level of dogmatism
on either side. "Enforced encapsulation" has been "implemented" in both
C++ and Java, and both have proved to be circumventable. Just the same,
talk of "clueless pointy-haired bosses" is unlikely to be convincing.
Even the pointy-haired types never recognize themselves as such.

Annotations *have* made it into 3.0, so it's possible that the might
become usable. Remember, they'll always be optional, so those who don't
want to use them won't lose anything at all.

[...]

regards
Steve

 

[Tim Rowe]

Having once been a more type-A, I labored for a couple of years trying
to build a restricted language that provably terminated for work on an
object-oriented database research.

I was careful to say that it was the /use/ of the language that is
restricted; it's still possible to write undecidable programs, there's
just an obligation on you to show that you haven't.

I finally gave it up as a bad idea,
because, in practice, we don't care if a loop will terminate or not in
database work; a transaction that takes a year to commit is equivalent
to an infinite loop for all applications that I have interacted with
(and yes, I have worked allowing four day transactions to commit).

True -- the really serious safety critical stuff is usually real time,
and part of the proof obligations is to show that the maximum response
time is tolerable. The loop variant can help with that, too.

 

[Russ P.]

I have to say that I thought the example was somewhat bogus. Any
development team that is even slightly concerned about the possibility
of logic bombs in the code will try to mitigate that possibility by the
use of code inspections.

Of course they would, but that does not mean that access restrictions
enforced by the language are not prudent. Consider a corporation that
needs to maintain the physical security of their buildings. You can
say that locks on the doors are not sufficient. Of course they're not.
Security guards are needed too -- but that doesn't mean the locks are
not needed too. The locks may be *insufficient* by themselves, but
they are certainly not *unnecessary*. Ditto for the enforced
encapsulation for large financial or safety-critical projects.

I'm not sure that there's much to be gained by this level of dogmatism
on either side. "Enforced encapsulation" has been "implemented" in both
C++ and Java, and both have proved to be circumventable. Just the same,
talk of "clueless pointy-haired bosses" is unlikely to be convincing.
Even the pointy-haired types never recognize themselves as such.

Do you think the designers of C++, Java, Ada, and Scala would
eliminate the enforced encapsulation if they had it to do over again?
Of course not. The vast majority of the users of those languages
consider it a major plus. And those who don't wish to use it aren't
forced to use it except perhaps by their bosses or their customers.

Again, I am not saying that Python necessarily needs enforced access
restriction. I'm just saying it would be a plus if it can be added
without compromising the language in some way.

 

[Luis Zarrabeitia]

Why should it be in your power? By messing with the implementation of a
library you risk the correctness of the code of all participant coders in
that project. I not that sure it should be your power to chose when and how
to do that.

Ok, let me fix that.
It should be in _our_ power as the team of all participant coders on _our_
project to decide if we should mess with the internals or not.

What makes no sense is that it should be in the original author's power to
decide, if he is not part of _our_ team.

Do you like it better now? Wasn't it obvious in the first place?

 

[Mark Wooding]

You've built something full of user serviceable parts. You've
insisted, publicly and loudly, that the ability to modify those parts
is absolutely essential, you've rejected every effort to lock down
those internals, and then when somebody does exactly what you
encourage, you suddenly turn on them and say they're on their own.

That's pretty irresponsible behaviour.

An egregious mischaracterization.

Again, you're refusing to see the distinction between a published,
stable interface, and exposed but potentially volatile internals. If
you use my published interface and your program is broken by a change,
it's my fault and I try to fix it; if you use the internals and your
program's broken by a change, it's your fault and you're on your own.
It's pretty easy.

There are three possibilities:

* The interface does what I want it to do. Cool. I win.

* The interface doesn't do what I want it to do, but I can hack the
internal guts until it does. Cool for now, but I might get screwed
later if they change. Well, it's something to think about.

* The interface doesn't do what I want it to do, and I can't hack the
internal guts. I just lose.

If all interfaces were perfect, there wouldn't be an argument. But they
aren't. So we're sometimes left with the other two choices. Which one
looks more appealing to you?

-- [mdw]

 

[Mark Wooding]

Was this library module released in source form?

If so, then why would you care that it has enforced access
restrictions? You can just take them out, then do whatever you would
have done had they not been there to start with. I don't see how that
is any more work than figuring out what internals you need to access.
Either way you need to read and understand the code.

I could do. But in practice internal details often aren't horrifically
unstable. If I fork the library, I've now got to mess with distributing
the forked version, and keeping the patch up to date -- the codebase is
likely to be much more volatile than the internals I'm hacking on. It's
just not such a good tradeoff.

See elsewhere where I described hacking Python's `long' representation.
You'd have me fork the Python interpreter. That doesn't seem like a win
to me.

Wait ... it wasn't released in source form? Then how would you even
know what internals you need to access?

We call them `disassemblers' and `debuggers'. Sometimes even simple
experimentation is sufficient.

And why would you use something that goes against your philosophy of
openness anyway?

I try not to. Sometimes I fail.

-- [mdw]

 

[Mark Wooding]

OK, fine, you can change the code of another member of the team. Are
you going to check with him first, or just do it? The point is that
changing an interface requires agreement of the team members who use
that interface, whether on the calling or the implementation side of
it. If you change interfaces without getting agreement with the other
team members, you probably won't be on the team for long.

So far, so good.

And without access restrictions, accessing _private is equivalent to
changing the interface.

You've basically unilaterally stretched it, yes. Programmers who do
this when the implementer of the interface is three feet away deserve to
get smacked.

If you don't have an easy way to get the interface extended, you have a
pleasant choice between hacking at the guts and just being screwed.
Have a nice day.

Are you seriously saying that if you were managing the production of a
major financial software package with hundreds of developers, you
would just "trust" them all to have free access to the most sensitive
and critical parts of the program? Now *that's*, well, kind of funny,
you know?

If you're working with developers you don't trust, you're going to lose
anyway.

-- [mdw]

 

[Mark Wooding]

Annotations *have* made it into 3.0, so it's possible that the might
become usable. Remember, they'll always be optional, so those who don't
want to use them won't lose anything at all.

There's a problem here. An interface has two sides. Access control
annotations, as far as I've seen so far, look like they're only optional
on one side: that is, once specified on the implementation side, they
have a mandatory effect on the user side.

-- [mdw]

 

[Mark Wooding]

That's a model that works well when you have a small team of, say, a
dozen people. It's not a model that works when you have hundreds of
developers working on the project. The last thing I want in my projects
is cowboys who ride all over other people's code, changing internals of
parts they barely know at the drop of a hat, causing who knows what side-
effects.

Why are you assuming that Bruno would make his change in such an
irresponsible manner? Such things need discussing, with some
appropriate level of formality, with the people most involved with the
relevant code -- which might be anything between a quick chat with the
bloke across the room or an agenda item on the next team meeting to a
written proposal.

Given your apparent readiness to assume the worst of programmers at
every opportunity (e.g., your above assumption that Bruno would change
code unilaterally and secretly, assumptions elsewhere that programmers
treat any exposed wiring as being published interface), I can only
assume that you really need to get some better cow-orkers.

But anyway, this is a red-herring. Data hiding isn't really about
managing the development process, it's about ensuring that when you and
Fred are using Barney's OrderedDict class in your code, Barney is free to
change his the implementation without your code suddenly failing. It's
also to ensure that any changes you make at runtime to the class don't
suddenly make Fred's code break.

If Barney, Fred and I share a room, we can discuss what we need from
OrderedDict around the water-cooler. If we're all in different
continents, the tradeoffs look different.

If Barney's the kind of guy with quick turnaround, I might well just ask
for the necessary change. If I'll have to wait for the next release in
July (for posterity: it's January now) I'll probably hack the guts now.
If I do that, Barney's still free to change his implementation. It
might screw me, but that's my problem and not his; the alternative, with
mandatory `hiding', is that I just lose: possible breakage down the line
looks like an improvement.

Now we come on to Fred. If Fred's across the room from me then we're
back to the water-cooler. If he's on a different continent, and I know
he'll be affected, I'll probably email him. If I've never heard of him
at all, well, he might just lose when someone puts my code and Fred's
together with OrderedDict; hopefully I'll get a bug report (I ought to
have put a notice in with my distribution explaining that it uses
undocumented internals of OrderedDict, so I should be in the firing
line) and we'll sort through the wreckage. But again, if I just lost in
the second paragraph, we wouldn't even have got this far.

-- [mdw]

 

[Paul Rubin]

Now we come on to Fred. If Fred's across the room from me then we're
back to the water-cooler. If he's on a different continent, and I know
he'll be affected, I'll probably email him. If I've never heard of him
at all, well, he might just lose when someone puts my code and Fred's
together with OrderedDict...

In a large project, more probably you'd enter a change request into
some kind of tracking system, there would be discussion in the
tracking system about how to do the change; perhaps at your weekly
staff meeting you might bring up the issue with your PHB if you were
blocking on the issue, and your PHB would bring it up at the inter-PHB
meeting with Fred's PHB to bump the item's priority, and eventually
Fred would check in a change and you would use it. There is
necessarily some wasted motion in any organization of that size; good
management is about keeping the friction to a minimum and getting the
stuff done.

 

[Steven D'Aprano]

Why are you assuming that Bruno would make his change in such an
irresponsible manner?

I did? Where did I make that assumption?

What I said was that the model "The code is the whole team's ownership"
doesn't work well for large projects. *One* reason it doesn't work for
large projects is that you will invariably have cowboys who, given half a
chance, will code irresponsibly *if you let them* by encouraging the
attitude that, sure, that class written by the database backend team
belongs to everyone, never mind that you're in the UI team, go right
ahead and use whatever internals you like. There are other reasons -- do
I need to elaborate on them?

Even in a project as small as Python there is sense of *individual*
(rather than collective) code ownership, in the sense of responsibility
for specific portions of the code base. Scale up the project by a factor
of ten, and the problems caused by collective responsibility are simply
intractable.

Such things need discussing, with some
appropriate level of formality, with the people most involved with the
relevant code -- which might be anything between a quick chat with the
bloke across the room or an agenda item on the next team meeting to a
written proposal.

Yes, and when you have hundreds of developers working on the project, the
chances are quite good that there will be twenty such agenda items every
week, and then the project will bog down on arguments about what needs to
be private and what doesn't, until the project manager just makes a
blanket ruling No Access To Internals Full Stop.

And then, you know what, the project still manages to go forward. Instead
of spending 15 minutes hacking the existing Foo class to do what they
want, and then 15 hours dealing with the cascading bugs when the class
changes, people simply spend 7 hours subclassing Foo to get what they
want, and it's all good and not the end of the world.

Given your apparent readiness to assume the worst of programmers at
every opportunity (e.g., your above assumption that Bruno would change
code unilaterally and secretly, assumptions elsewhere that programmers
treat any exposed wiring as being published interface), I can only
assume that you really need to get some better cow-orkers.

Now you're just being naive. The solution to these sorts of problems
isn't "get better programmers" because even the best programmers make
mistakes (errors, and errors of judgement). Communication breaks down --
Fred is sure Barney said "Yes" while Barney is equally positive he said
"No". Or somebody has a demo to the CEO in two hours and needs to hack
something up Right Now and the guy he has to chat to is home sick with
the phone turned off, and by the time he comes back to work the little
hack is forgotten. Or whatever. There's a *bazillion* number of things
that can go wrong in big projects, and "getting better programmers" only
reduces that to something like a hundred million. It's a credit to
*everybody*, including the cowboys and the cow-orkers, that any project
of any significant size makes any progress at all.

Besides, "better programmers" are in short supply. Sometimes you have to
use who you've got, because Guido and the timbot are working somewhere
else and you can't afford them. That's why we have languages like Python:
so that *ordinary* programmers can be as productive as genius programmers
in languages like C.

 

[Steven D'Aprano]

It should be in _our_ power as the team of all participant coders on
_our_ project to decide if we should mess with the internals or not.

What makes no sense is that it should be in the original author's power
to decide, if he is not part of _our_ team.

Makes *no* sense? There's *no* good reason *at all* for the original
author to hide or protect internals?

Let's be specific here. The list implementation in CPython is an array
with a hidden field storing the current length. If this hidden field was
exposed to Python code, you could set it to a value much larger than the
actual size of the array and cause buffer overflows, and random Python
code could cause core dumps (and possibly even security exploits).

So what you're saying is that the fundamental design of Python -- to be a
high-level language that manages memory for you while avoiding common
programming errors such as buffer overflows -- makes "no sense". Is that
what you intended?

As I see it, you have two coherent positions. On the one hand, you could
be like Mark Wooding, and say that Yes you want to risk buffer overflows
by messing with the internals -- in which case I'm not sure what you see
in Python, which protects so many internals from you. Or you can say that
you made a mistake, that there are *some* good reasons to protect/hide
internals from external access.

In the second case, the next question is, why should it only be code
written in C that is allowed that protection?

 

[Mark Wooding]

I did? Where did I make that assumption?

I inferred it from the juxtaposition, apparently in error. Sorry.

What I said was that the model "The code is the whole team's ownership"
doesn't work well for large projects. *One* reason it doesn't work for
large projects is that you will invariably have cowboys who, given half a
chance, will code irresponsibly *if you let them* by encouraging the
attitude that, sure, that class written by the database backend team
belongs to everyone, never mind that you're in the UI team, go right
ahead and use whatever internals you like.

`Cowboys' will code irresponsibly anyway; they need reeducating gently
with a stick.

Yes, and when you have hundreds of developers working on the project,
the chances are quite good that there will be twenty such agenda items
every week, and then the project will bog down on arguments about what
needs to be private and what doesn't, until the project manager just
makes a blanket ruling No Access To Internals Full Stop.

Err... you've wandered off the track here. The purpose of the proposed
changes is to expose the right interfaces so that people don't need to
fiddle with internals. If I've discovered that I need to dredge
something out of the middle of your module, I have three choices:
(a) somehow get your module changed so that it hands me the information
I want through a documented interface, (b) suck your module's brains out
through a straw whether the interface lets me or not, or (c) lose. The
agenda items are supporting (a), which I thought you were in favour of;
you certainly seem to be against (b), and (c) doesn't look appealing.

And then, you know what, the project still manages to go
forward. Instead of spending 15 minutes hacking the existing Foo class
to do what they want, and then 15 hours dealing with the cascading
bugs when the class changes, people simply spend 7 hours subclassing
Foo to get what they want, and it's all good and not the end of the
world.

Subclassing hardly fixes these sorts of problems well. Hardly any class
libraries are documented sufficiently well enough to support subclassing
properly. Have a look at the CLOS MOP specification

http://www.lisp.org/mop/index.html

for a rare example of how to do it right.

Besides, "better programmers" are in short supply. Sometimes you have
to use who you've got, because Guido and the timbot are working
somewhere else and you can't afford them. That's why we have languages
like Python: so that *ordinary* programmers can be as productive as
genius programmers in languages like C.

Why waste genius programmers on churning out C? Just imagine how much
they could do with something like Python!

-- [mdw]

 

[Aaron Brady]

I inferred it from the juxtaposition, apparently in error.  Sorry.


`Cowboys' will code irresponsibly anyway; they need reeducating gently
with a stick.

A lot of this is a question of how tight you want the handcuffs.
Stricter backgrounds and bigger projects make you want tighter ones.
You can't jump right from tight handcuffs to loose ones. You'll tend
to like them looser over time. A language looser than Python in
unheard of.

Google says: 'No results found for "looser than Python".'

 

[Steve Holden]

I inferred it from the juxtaposition, apparently in error. Sorry.


`Cowboys' will code irresponsibly anyway; they need reeducating gently
with a stick.

Unfortunately on large projects it's impossible to avoid cowboys, and
some of them are completely lost to reason.

[...]

regards
Steve

 

[Mark Wooding]

Let's be specific here. The list implementation in CPython is an array
with a hidden field storing the current length. If this hidden field was
exposed to Python code, you could set it to a value much larger than the
actual size of the array and cause buffer overflows, and random Python
code could cause core dumps (and possibly even security exploits).
[...]

As I see it, you have two coherent positions. On the one hand, you could
be like Mark Wooding, and say that Yes you want to risk buffer overflows
by messing with the internals

Please, point out where I said that!

I'm pretty sure that the only time I commented on this particular point
(in message <[email protected]>), I said:

: Umm... I'm pretty sure that that's available via the `len' function,
: which is tied to list.__len__ (via the magic C-implemented-type mangler,
: in C). Though it's read-only -- and this is a shame, 'cos it'd be nice
: to be able to adjust the length of a list in ways which are more
: convenient than
:
: * deleting or assigning to a trailing slice, or
: * augmenting or assigning to a trailing zero-width slice
:
: (Perl has supported assigning to $#ARRAY for a long time. Maybe that's
: a good argument against it.)

While I realise I didn't spell it out, the semantics I had in mind where

foo.len = n

means

if n < 0:
raise ValueError, 'don\'t be stupid'
elif len(foo) < n:
foo += [None] * (n - len(foo))
else:
foo[n:] = []

(I'm not fussy what the new array slots get filled with, but it seems
sensible to be clear. Perl's semantics are more complicated: if you
decrease $#foo and then increase it again you get the old values back.
Common Lisp users will recall the idea of a fill-pointer.)

If there's anything unsafe about that, I'll be surprised.

-- in which case I'm not sure what you see in Python, which protects
so many internals from you. Or you can say that you made a mistake,
that there are *some* good reasons to protect/hide internals from
external access.

Safety is good. Escape hatches are good, too.

In the second case, the next question is, why should it only be code
written in C that is allowed that protection?

Because Python code can't cause those sorts of problems without
resorting to the escape hatches (e.g., ctypes). And, very
significantly, because C code /needs/ that protection and Python
basically doesn't.

The basic difference is that C code is fundamentally brittle: if you
mess up its invariants, it can crash horribly and possibly allow its
brain to be taken over by evil people. Python code is fundamentally
robust. The worst that can happen[1] is that the interpreter raises an
exception. This makes it ideally suited to having a more relaxed
attitude to life. And that, in turn, makes it approachable,
hackable interactively, fun!

[1] Assuming that (a) the Python implementation and C extensions are
correct, and (b) that the code in question isn't using the escape
hatches.

-- [mdw]

 

[Rhodri James]

In a large project, more probably you'd enter a change request into
some kind of tracking system, there would be discussion in the
tracking system about how to do the change; perhaps at your weekly
staff meeting you might bring up the issue with your PHB if you were
blocking on the issue, and your PHB would bring it up at the inter-PHB
meeting with Fred's PHB to bump the item's priority, and eventually
Fred would check in a change and you would use it. There is
necessarily some wasted motion in any organization of that size; good
management is about keeping the friction to a minimum and getting the
stuff done.

My experience with medium-sized organisations (50-100 people) is that
either you talk to Fred directly, or it doesn't happen. In particular
the more people (especially PHBs) that get involved, the slower the
change will come and the less like your original requirement it will
look. Each person, no matter how technically adept, has a significant
chance of misunderstanding what it is you need and/or expressing it
poorly to the next person in line.

If I need something that goes 'ping' by Friday, and get something
that goes 'pong' three weeks later (together with a note from Fred
telling me that three-and-fourpence isn't going to get me into any
of the good dance clubs), I'd have been better off taking a hacksaw
to the internals myself and presenting my fix for Fred's
consideration. Python allows me to do this without the fuss and
drama that strict encapsulation seems to posit is necessary,
instead of delaying an entire release cycle because the process
doesn't trust me.