Why does python not have a mechanism for data hiding?

[Sh4wn]

Hi,

first, python is one of my fav languages, and i'll definitely keep
developing with it. But, there's 1 one thing what I -really- miss:
data hiding. I know member vars are private when you prefix them with
2 underscores, but I hate prefixing my vars, I'd rather add a keyword
before it.

Python advertises himself as a full OOP language, but why does it miss
one of the basic principles of OOP? Will it ever be added to python?

Thanks in advance,
Lucas

 

[Fuzzyman]

From whom are you trying to hide your attributes?

Actually, 'data hiding', although vastly overused by the static crowd
can be a reasonable thing to want.

For example, at Resolver Systems we expose the spreadsheet object
model to our users. It hasa public, documented, API - plus a host of
undocumented internally used methods.

We would really *much* rather hide these, because anything our
customers start using (whether documented or not) we will probably
have to continue supporting and maintaining.

The 'we told you not to use that' approach, when applied to paying
customers doesn't really work... all they see is that you broke their
spreadsheet code by changing your API.

You can make members truly private by proxying, but it is a bit
ungainly.

Michael Foord
http://www.ironpythoninaction.com/

 

[Roy Smith]

The 'we told you not to use that' approach, when applied to paying
customers doesn't really work... all they see is that you broke their
spreadsheet code by changing your API.

I hear what you're saying, friend, and I feel your pain! Life gets so much
more complicated when you've got paying customers. It's especially
complicated when your customers have a market cap an order of magnitude
greater than your own

 

[Ville M. Vainio]

The 'we told you not to use that' approach, when applied to paying
customers doesn't really work... all they see is that you broke
their spreadsheet code by changing your API.

And the customer point of view is quite reasonable - they have a job
to do, and a limited timeframe; sometimes accessing privates directly
is much better than waiting for updates from vendor. Bad designs (as
far as choosing publics goes) happen.

Even if their softare breaks on upgrade, you can quite clearly point
out that they used an internal api - and they will keep on using the
old version until they solve the problem. Everybody wins.

Perhaps a lint-like validation tool would be optimal for this
problem...

 

[Fuzzyman]

And the customer point of view is quite reasonable - they have a job
to do, and a limited timeframe; sometimes accessing privates directly
is much better than waiting for updates from vendor. Bad designs (as
far as choosing publics goes) happen.

They will use whatever they find, whether it is the best way to
achieve a goal or not. Once they start using it they will expect us to
maintain it - and us telling them it wasn't intended to be used by
them in the first place won't cut it.

Even if their softare breaks on upgrade, you can quite clearly point
out that they used an internal api - and they will keep on using the
old version until they solve the problem. Everybody wins.

Not if they are waiting for a fix or new feature in the upgrade - and
we can't refactor because they are relying on APIs that we want to
remove.

We very much lose.

Perhaps a lint-like validation tool would be optimal for this
problem...

So we can refuse to execute their code if they use private APIs?

Proxying so that we can really hide our internal APIs is a better
solution.

This hasn't happened to us yet (our application has only been in
commercial use since January), but it is one of the reasons that
Microsoft's COM APIs grew so wide and wild, and caused them very real
problems in trying to improve them. We are keen to avoid the same
situation.

Michael Foord
http://www.ironpythoninaction.com/

 

[bearophileHUGS]

Ben Finney:

In Python, the philosophy "we're all consenting adults here" applies.<

Michael Foord:

They will use whatever they find, whether it is the best way to
achieve a goal or not. Once they start using it they will expect us to
maintain it - and us telling them it wasn't intended to be used by
them in the first place won't cut it.

So they will use the methods with one or two underscores too. And
imply that you want to document them too.
They don't seem adult enough, then ;-)
Internal API hiding seems quite less useful if you don't need to let
customers manage your code.

Bye,
bearophile

 

[Fuzzyman]

Ben Finney:


Michael Foord:


So they will use the methods with one or two underscores too. And
imply that you want to document them too.
They don't seem adult enough, then ;-)

Great! We'll just tell them that...

Michael

 

[Sh4wn]

From whom are you trying to hide your attributes?

In Python, the philosophy "we're all consenting adults here" applies.
You shouldn't pretend to know, at the time you write it, all the uses
to which your code will be put. Barriers such as enforced "private"
attributes will only cause resentment when people, despite your
anticipations, *need* to access them and are then forced to hack their
way around them.

If you want the users of your code to know that an attribute should
not be used as a public API for the code, use the convention of naming
the attribute with a single leading underscore. This is a string
signal that the attribute is part of the implementation, not the
interface. The reader is then on notice that they should not rely on
that attribute; but they are not *prohibited* from using it if
necessary to their ends.


Who taught you that enforced restrictions on attribute access was a
"basic principle" of OO?

Perhaps you're confusing the "encapsulation" and "abstraction"
principles for enforced access restrictions; they're not.


I hope not.

--
 \            "Why was I with her? She reminds me of you. In fact, she |
  `\             reminds me more of you than you do!"  -- Groucho Marx |
_o__)                                                                  |
Ben Finney

Well for me, it the ideal way to make sure it contains so 'wrong'
data. You can create getter/setters, and in some cases only a getter
and validate the value given by the user. Then you'll not have to
worry about the data in the rest of your class, which makes life a lot
easier IMO.

 

[Paddy]

Actually, 'data hiding', although vastly overused by the static crowd
can be a reasonable thing to want.

For example, at Resolver Systems we expose the spreadsheet object
model to our users. It hasa public, documented, API - plus a host of
undocumented internally used methods.

We would really *much* rather hide these, because anything our
customers start using (whether documented or not) we will probably
have to continue supporting and maintaining.

The 'we told you not to use that' approach, when applied to paying
customers doesn't really work... all they see is that you broke their
spreadsheet code by changing your API.

"We told you not to use it in the API Docs"
"Those names with leading undoerscores means _DO_NOT_USE_IT"
"THose methods whose docstrings say DO NOT USE EXTERNALLY"

And if they still use them, then they'd be problematic no matter what
language was used.

Customers ey?
Can't live without 'em...
.... Actually that's customers Sir!



- Paddy.

 

[Terry Reedy]

|| > For example, at Resolver Systems we expose the spreadsheet object
| > model to our users. It hasa public, documented, API - plus a host of
| > undocumented internally used methods.
| >
| > We would really *much* rather hide these, because anything our
| > customers start using (whether documented or not) we will probably
| > have to continue supporting and maintaining.
| >
| > The 'we told you not to use that' approach, when applied to paying
| > customers doesn't really work... all they see is that you broke their
| > spreadsheet code by changing your API.

Python was not really written with 'difficult' customers in mind ;-)

One could largely hide private vars with a program that substituted random
names for single _ names, and removed the doc strings for functions,
classes, and methods with such names.

Such a program could even put such names in a separate module imported as
'_private_do_not_use_'.

 

[Aahz]

first, python is one of my fav languages, and i'll definitely keep
developing with it. But, there's 1 one thing what I -really- miss:
data hiding. I know member vars are private when you prefix them with
2 underscores, but I hate prefixing my vars, I'd rather add a keyword
before it.

Python advertises himself as a full OOP language, but why does it miss
one of the basic principles of OOP? Will it ever be added to python?

Not directly addressing your point except insofar as it appears that
your understanding of OOP may be limited:

"...some experts might say a C++ program is not object-oriented without
inheritance and virtual functions. As one of the early Smalltalk
implementors myself, I can say they are full of themselves." --zconcept

 

[Ville M. Vainio]

So we can refuse to execute their code if they use private APIs?

No, but it could complain and point out the exact offending lines,
pointing their development effort to right direction.

Proxying so that we can really hide our internal APIs is a better
solution.

How about pyprotocols and other interface packages?

Proxying is pretty workable too, and it easily makes sense that the
official API objects should be different from the logic executing
objects.

 

[Fuzzyman]

|| > For example, at Resolver Systems we expose the spreadsheet object
| > model to our users. It hasa public, documented, API - plus a host of
| > undocumented internally used methods.
| >
| > We would really *much* rather hide these, because anything our
| > customers start using (whether documented or not) we will probably
| > have to continue supporting and maintaining.
| >
| > The 'we told you not to use that' approach, when applied to paying
| > customers doesn't really work... all they see is that you broke their
| > spreadsheet code by changing your API.

Python was not really written with 'difficult' customers in mind ;-)

True. It's extremely suited to what we do though.Minor difficulties
like this are vastly outweighed by advantages. The difficulties are
real though.

One could largely hide private vars with a program that substituted random
names for single _ names, and removed the doc strings for functions,
classes, and methods with such names.

We need to *use* those names to display the spreadsheet once the
calculation has finished (and their code has run).

Such a program could even put such names in a separate module imported as
'_private_do_not_use_'.

Splitting more of the functionality out is probably part of the best
solution.

Michael Foord
http://www.ironpythoninaction.com/

 

[Ville M. Vainio]

Or if you code in C++ and they *really* need to get at something you
made private they will still get at it. I've been there and done
that: 'private' in languages which have it is rarely an advantage
and frequently a pain.

Indeed. In C++, they recommend the "pimpl idiom" (private
implementation), which actually has real advantages ;-)

 

[Fuzzyman]

No, but it could complain and point out the exact offending lines,
pointing their development effort to right direction.


How about pyprotocols and other interface packages?

We're using IronPython. I haven't looked at pyprotocols but I know the
Zope interface package is at least partly written in C. Our
spreadsheet object model is very 'performance sensitive', so that's a
consideration. We should definitely explore the prior art in this area
before we implement anything ourselves.

Thanks

Michael Foord
http://www.ironpythoninaction.com/

 

[Joe P. Cool]

From whom are you trying to hide your attributes?

I saw this "don't need it" pattern in discussions about the ternary
"if..else" expression and about "except/finally on the same block
level".
Now Python has both. Actually it is very useful to be able to
distinguish
between inside and outside. This is obvious for real world things e.g.
your
TV. Nobody likes to open the rear cover to switch the channel. Similar
arguments apply to software objects. "data hiding" is a harsh name, I
would
call it "telling what matters". The need for this becomes
indispensable in
really big software packages like the Eclipse framework with approx.
100000
classes. If you cannot tell the difference between inside and outside
you
are lost.

In Python, the philosophy "we're all consenting adults here" applies.

Please don't sell a missing feature as a philosophy. Say you don't
need/want
it. But don't call it philosophy.

You shouldn't pretend to know, at the time you write it, all the uses
to which your code will be put.

It's *your* *decision* which uses will be available. Your explanation
appears
to me as a fear to decide.

If you want the users of your code to know that an attribute should
not be used as a public API for the code, use the convention of naming
the attribute with a single leading underscore.

Littering your class definition with dozens of underscores is exactly
the
line noise we love to criticize in Perl.

Who taught you that enforced restrictions on attribute access was a
"basic principle" of OO?

Nearly every introduction to OOP? Please don't tell me that
encapsulation
does not mean "enforced restriction". If the language has no syntactic
support for encapsulation then it does not have encapsulation. Similar
argument applies to Java where programmers sometimes claim that Java
"has"
properties because of setCrap/getCrap.

__
Joe

 

[Terry Reedy]

| Actually it is very useful to be able to distinguish
| between inside and outside.

Which Python allows one to do.

| This is obvious for real world things e.g. your
| TV. Nobody likes to open the rear cover to switch the channel.

The original issue was that users might open the case to do something
unplanned and that they would come to depend on and clamor for the
maintenance of internal details that the manufacturer wants to change. But
in the real world, most customers know and accept that custom alterations
and extensions of a product my be model specific. Some things I buy even
have a disclaimer that user-visible details might be different from what
the instructions say, and only guarantee that the functionality will be as
good or better than specified.

[...]
| Please don't sell a missing feature as a philosophy.

I won't if you don't claim that a feature is missing because you don't like
its implementation. To most of us, replacing the nearly never used '__'
with a keyword would be a auful change.

| Please don't tell me that encapsulation does not mean "enforced
restriction".

There are obviously degrees of enforced restriction. Take your TV example.
Most users will respect a 'do not open' request by the manufacturer
(Python's '_'.). But some will unscrew anyway. So some makers use
fasteners that are resistent to opening without specialized tools from
specialized sources -- or perhaps specialized knowledge (Python's '__').
(This is very frustrating if one needs to open such a system.) Or some
mark the screws so they can detect and punish entry. At least one (Apple)
has intentionally altered internals to punish entry and alteration. Or
there is the 'embed in epoxy' method.

If you want something equivalent to epoxy embedding, design it, implement
it, and share it (if you want). Perhaps one could make an Opaque extension
class, perhaps even usable as a mixin class rather than as a single-base
class.

Terry Jan Reedy

 

[miller.paul.w]

Python advertises himself as a full OOP language, but why does it miss
one of the basic principles of OOP? Will it ever be added to python?

Others have already answered this directly, but I'd like to mention
that languages I know of which have this feature also have a feature
for getting around it. (e.g. C++ and friend classes) I don't know
about you, but I don't want features in the language that make me want
to circumvent them. Do you?

 

[Russ P.]

I saw this "don't need it" pattern in discussions about the ternary
"if..else" expression and about "except/finally on the same block
level".
Now Python has both. Actually it is very useful to be able to
distinguish
between inside and outside. This is obvious for real world things e.g.
your
TV. Nobody likes to open the rear cover to switch the channel. Similar
arguments apply to software objects. "data hiding" is a harsh name, I
would
call it "telling what matters". The need for this becomes
indispensable in
really big software packages like the Eclipse framework with approx.
100000
classes. If you cannot tell the difference between inside and outside
you
are lost.


Please don't sell a missing feature as a philosophy. Say you don't
need/want
it. But don't call it philosophy.


It's *your* *decision* which uses will be available. Your explanation
appears
to me as a fear to decide.


Littering your class definition with dozens of underscores is exactly
the
line noise we love to criticize in Perl.



Nearly every introduction to OOP? Please don't tell me that
encapsulation
does not mean "enforced restriction". If the language has no syntactic
support for encapsulation then it does not have encapsulation. Similar
argument applies to Java where programmers sometimes claim that Java
"has"
properties because of setCrap/getCrap.

__
Joe

Dear Mr. Cool,

Excellent post. I agree with you entirely.

I've always considered the underscore conventions in Python an ugly
hack in an otherwise elegant language. I often avoid them even when
they technically belong just because I don't like the way they make my
code look.

Your analogy of the TV set is nice. The front control panel should be
clearely distinguished from the covered control panel and certainly
from the guts. It's more than just a matter of keeping the user from
doing something stupid -- it's a matter of design style.

I am also bothered a bit by the seeming inconsistency of the rules for
the single underscore. When used at file scope, they make the variable
or function invisible outside the module, but when used at class
scope, the "underscored" variables or functions are still fully
visible. For those who claim that the client should be left to decide
what to use, why is the client prohibited from using underscored
variables at file scope?

I may be full of beans here, but I would like to make a suggestion.
Why not add a keyword such as "private," and let it be the equivalent
of "protected" in C++? That would make member variables and functions
visible in inherited classes. The encapsulation wouldn't be as
airtight as "private" in C++, so clients could still get access if
they really need it in a pinch, but they would know they are bending
the rules slightly. I realize that single underscores do that already,
but they are just unsightly.

 

[Neil Hodgson]

Russ P.:

The encapsulation wouldn't be as
airtight as "private" in C++, so clients could still get access ...

C++ private is not airtight as you have access to pointer arithmetic
as well as
#define private public

Neil