function v. method

[Bruno Desthuilliers]

danielx a écrit :

And machine language is an abstraction of pushing electrons around
circuits. I'm not sure I see your point, unless it is simply that
Python is easier than asm.

Python is very hi-level, and very often well-written Python code is it's
own better documentation.

 

[danielx]

danielx wrote:
At first I was going to post the following:

<!-- beginning of my original post -->

(snip)

<!-- end of my original post, with ending censored -->

but then I tried this:


res = Foo.__dict__['func']
res is dan

True

And it all started to make sense. The surprising thing turned out to be
not so surprising: When the expression Foo.func gets evaluated, we get
a method which is just a wrapper around dan. Therefore, f is not dan!
This is still a little bit of magic,

FWIW, the function class implements the descriptor protocol... Here's
the "magic".

which gets me thinking again about
the stuff I self-censored. Since the dot syntax does something special
and unexpected in my case,

"unexpected" ? Did you ever wondered how the instance or class was
passed as first arg when doing method calls ?

Not knowing what's going on during method calls is exactly what
motivated me to post.

why not use some more dot-magic to implement
privates?

What for ? What makes you think we need language-inforced access
restriction ?

I knew someone would bring this up. The motivation would not be to
provide restriction, but to help maintain clean api's. If you intended
for users to use only a subset of the methods in your class, why not
help them learn your api by presenting the stuff they can use not along
side the stuff they should not?

Obviously, such things would be omitted from your docs, but users also
learn by interacting with Python, which is really one of Python's great
virtues. When supporting documents aren't sufficient to learn an api
(I'm sure this never happens, so just humor me), you can always turn to
interactive Python. This is exactly what it's there for. If nothing is
hidden, a user could be easily mislead to believe he can use a method
when he really shouldn't.

if you prefix with a single underscore, that tells the user, DON'T MESS
WITH ME FROM OUTSIDE! I AM AN IMPLEMENTATION DETAIL!

and it gets ommited from all the doc generation

if you prefix with a double underscore, then they have to go even
FARTHER out of their way to shoot themselves in the foot.

Python takes the stance of "personal responsiblity" when it comes to
access control. Which in NO WAY dimishes its "robustness" or anything
else.

just read the DailyWTF.com, incompentent people will abuse any language
features in any language, and will figure out how to break programatic
access control no matter how it is implemented. Matter of fact, Java
which in another thread someone was ADAMANT that did not expose private
anything from reflection ( and was wrong ) specifically allows you to
access all the private members, functions, everything. You just need to
tell it to turn all the "safeties" off.

From the api:

public void setAccessible(boolean flag)
throws SecurityException

Set the accessible flag for this object to the indicated boolean value.
A value of true indicates that the reflected object should suppress
Java language access checking when it is used. A value of false
indicates that the reflected object should enforce Java language access
checks.

Setting the accessible flag in a reflected object permits sophisticated
applications with sufficient privilege, such as Java Object
Serialization or other persistence mechanisms, to manipulate objects in
a manner that would normally be prohibited.

so anything added to Python to enforce "access control" would
immediately be forced to provide some means to over-ride the checks for
pickle and the like. Not to even mention the argument that it would
break crap loads of existing code base.

Sigh. I TOTALLY realize that Python works by politeness and not
enforcement. I think you are misinterpreting why I think this would be
a good idea. My concern is not with control, but with convenience. My
suggestion was that privates would only be invisible if you use the dot
syntax (ie if you are an external user); they would not be invisible
altogether (they would still be in __dict__ with no name games).

One problem which was brought up about this was that self.meth and
outsider.meth would have to be interpretted differently. I suspect (but
I haven't finished my reading assignment ) that you could find a good
way around this.

With respect to breaking stuff. I'm not sure why that would be
necessary. If current code does not say any member is private,
everything that was visible before (ie everything) would still be
visible after.

Last thing. You mentioned that auto doc generation omits
underscore-prefixed and name mangled members. dir on the other hand
does not. Maybe this suggests only a minor future improvement.

 

[danielx]

danielx a écrit :

Python is very hi-level, and very often well-written Python code is it's
own better documentation.

Yes, Python is very easy to read, but who's supposed to be reading it?
Maintainers or users? I'm really against code acting as its own
documentation. Of course the code is authoritative, but should we
really EXPECT it to serve as a reference to users??

Appearantly, this view puts me in the minority...

 

[danielx]

To bad for you.


Oh yes ?


Either you imported with the "from othermodule import *" form (which you
shouldn't do), and you *don't* have the implementation of othermodule,
or your used the "import othermodule" form, in which case it's pretty
obvious which names belongs to othermodule.

Have any other, possibly valid, reason ?


Too bad for you. Choose another language then... PHP, Perl, Ruby ?-)

"Too bad for you": While you have a valid point that this contention is
really just arbitrary (just like all conventions), could we be a little
gentler?

Personally, I don't think it looks very good either, but you just have
to deal with it if you're going to use the language properly.

 

[Bruno Desthuilliers]

danielx a écrit :

(snip)
Sigh. I TOTALLY realize that Python works by politeness and not
enforcement. I think you are misinterpreting why I think this would be
a good idea. My concern is not with control, but with convenience.

Having free access to implementation is convenient IMHO.

My
suggestion was that privates would only be invisible if you use the dot
syntax (ie if you are an external user); they would not be invisible
altogether (they would still be in __dict__ with no name games).

How would this work for class attributes ? (implementation methods,
implementation descriptors etc...)

Also, it would impact lookup perfs, which is already a somewhat weak
point in Python.

 

[Bruno Desthuilliers]

danielx a écrit :

Yes, Python is very easy to read, but who's supposed to be reading it?
Maintainers or users?

In an ideal world, nobody !-)

I'm really against code acting as its own
documentation.

I'm really for it, because :

Of course the code is authoritative,
indeed.

but should we
really EXPECT it to serve as a reference to users??

Unless you're able to maintain a good, accurate and always up to date
documentation, you can be sure users (ie other programmers) will turn to
the code. Also, there are cases where even a pretty good doc is not
enough, and you really have to turn to the code to know for sure how to
best implement something.

FWIW, reading source code can be very instructive...

 

[Antoon Pardon]

To bad for you.

I'll survive.

Oh yes ?


Either you imported with the "from othermodule import *" form (which you
shouldn't do), and you *don't* have the implementation of othermodule,
or your used the "import othermodule" form, in which case it's pretty
obvious which names belongs to othermodule.

As far as I understand the _name convention is often defended with the
argument that it stands out. Now if you have to go and look at the
import statements to make a disticntion, then it seems that the
way _names standout isn't that usefull.

Too bad for you. Choose another language then... PHP, Perl, Ruby ?-)

Not a very strong argument. Whether or not someone has a legitimat point
of criticism against Python or some of the conventions used. You can
always reply: Too bad, better choose another language then.

 

[Antoon Pardon]

you don't understand what "implementation detail" means, it means it is
NOT part of the public API and no client code should ever use it.

If you reference _vara in your code and it is in someone elses module
you don't understand YOU ARE NOT SUPPOSED TO DO THAT!

Why do you assume that in my example the other module is
not understood?

Users of your module should NEVER KNOW any of the _ or __ stuff exists
to begin with.


You should NEVER use it.

Well that may be your view, but AFAICS it is not the view of
the python community. Because each time some mechanism is
proposed for real private variable, people oppose it, they
want people to have access to what are supposed to be
private variables.

 

[Gerhard Fiedler]

Absolutely. I didn't mean the user was breaking abstraction (let's not
blame the victim). I was saying that we should really have more
sympathy for him.

I have all the sympathy in the world for him... after all, he's me

But one reason why I try to write (and insist on as much as possible from
people writing for or with me) self-documenting code is that wherever you
have documentation and code separated (which is the case of API docs), you
can bet (without losing) that sooner or later code and doc will diverge.
This has a probability that approaches 1

So, I agree with you that good API docs are a good thing, as they tell me
everything I need to know without having to wade through tons of
implementation details that may be interesting but don't serve my immediate
need (of having to use the API). But reality seems to be (and mine so far
definitely is) that these docs, even the good ones, are not completely in
alignment with the reality of the code. (We all know that code has bugs...
and the API always describes, at best, how the code /should/ work. It never
describes how it actually works, including the bugs <g> (this
notwithstanding the bugs that have been elevated to features and henceforth
been described in the API docs).

So... the final authority /is/ the code. I don't see an alternative. For
me, good abstraction doesn't mean I don't have to read the sources; good
abstraction means (among other things) that I can read the sources easily.

Gerhard

 

[Bruno Desthuilliers]

I'll survive.




As far as I understand the _name convention is often defended with the
argument that it stands out.

It does.

Now if you have to go and look at the
import statements to make a disticntion, then it seems that the
way _names standout isn't that usefull.

Please re-read the Fine Manual and try to understand what I wrote above.
You don't have to parse the import statements to know if you're dealing
with a local implementation detail or implementation of another module,
unless you do braindead renames of imported symbols.

Anyway, messing with another module's implementation should be *very* rare.

Not a very strong argument. Whether or not someone has a legitimat point
of criticism against Python or some of the conventions used. You can
always reply: Too bad, better choose another language then.

There are very few chances this convention will change anytime soon. You
may not like it for any good or bad reason, the fact is that you have to
live with it.

FWIW, the choice of other languages I proposed as an alternative is not
totally innocent : they all are possible replacements for Python, and
they all have their share of ugly cryptic notations... I'm not myself a
big fan of the leading underscore, but it's certainly a lesser evil when
compared to $php_vars or others @myperlishstuff. Or with C++
m_myMemberVar FWIW.

 

[Bruno Desthuilliers]

Why do you assume that in my example the other module is
not understood?

He doesn't, but it's clear that *you* didn't understand what
fuzzylollipop meant !-)

Hint : add a comma at the right place so parsing is unambigous:
"""
If you reference _vara in your code and it is in someone elses module,
you don't understand YOU ARE NOT SUPPOSED TO DO THAT!
"""

Well that may be your view, but AFAICS it is not the view of
the python community. Because each time some mechanism is
proposed for real private variable, people oppose it, they
want people to have access to what are supposed to be
private variables.

I'd express it in a somewhat different way: the view of the Python
community is that language-inforced access restrictions are useless and
annoying, IOW an unnecessary pain. Which doesn't imply that messing with
implementation of other modules is actually *encouraged*. Having the
possibility to easily do so is quite handy when that's the only/less
worse solution, but is not a recommended approach in general.

 

[fuzzylollipop]

I have all the sympathy in the world for him... after all, he's me

But one reason why I try to write (and insist on as much as possible from
people writing for or with me) self-documenting code is that wherever you
have documentation and code separated (which is the case of API docs), you
can bet (without losing) that sooner or later code and doc will diverge.
This has a probability that approaches 1

So, I agree with you that good API docs are a good thing, as they tell me
everything I need to know without having to wade through tons of
implementation details that may be interesting but don't serve my immediate
need (of having to use the API). But reality seems to be (and mine so far
definitely is) that these docs, even the good ones, are not completely in
alignment with the reality of the code. (We all know that code has bugs...
and the API always describes, at best, how the code /should/ work. It never
describes how it actually works, including the bugs <g> (this
notwithstanding the bugs that have been elevated to features and henceforth
been described in the API docs).

So... the final authority /is/ the code. I don't see an alternative. For
me, good abstraction doesn't mean I don't have to read the sources; good
abstraction means (among other things) that I can read the sources easily.

Gerhard

having auto generated docs, which means the documentatin is in the code
as doc strings, javadoc, reflex, or whatever format is the BEST way of
doing API documentation, period.

 

[fuzzylollipop]

Why do you assume that in my example the other module is
not understood?


Well that may be your view, but AFAICS it is not the view of
the python community. Because each time some mechanism is
proposed for real private variable, people oppose it, they
want people to have access to what are supposed to be
private variables.

actually you are really way off base, the _ and __ convention IS the
INTENDED way of doing private and "really private" documentation of
members in Python.

I didn't make this up, it is in the official Python documentation.

You need to read my previous response for COMPREHENSION one more time.
There is LESS THAN ZERO value in having a runtime enforcement of member
access control.

Python does have ALREADY have an OFFICAL mechanism for private members,
prefix your names with _ or __. Both are ommited from autogenerated
docuementation and both are OFFICALLY not supposed to be used.

 

[Gerhard Fiedler]

having auto generated docs, which means the documentatin is in the code
as doc strings, javadoc, reflex, or whatever format is the BEST way of
doing API documentation, period.

It may be the best way, no contest to that. But it still is not the code.
(I actually think that possibly the current way of embedding javadoc-like
documentation into sources is only a stepping stone into the direction
generally pointed to by what Wirth called "literate programming". In any
case, I'd rather not call it the "best way, period"; calling it the "best
currently widely supported way" seems more appropriate to me.)

Even doc strings tend to get out of sync with the code. And even doc
strings document (at best) what the code should do, not what it actually
does. And even doc strings are not always complete in describing the
functionality.

So auto generated docs are also incomplete and out of sync with the code,
sometimes more, sometimes less, sometimes so little that it is not
relevant. But you can't know how much out of sync they are from reading the
docs alone. So when push comes to shove (or so the saying goes? , the
code is the authority. Even with auto generated docs. Period... ?!? <g>

Gerhard

 

[Gerhard Fiedler]

(I actually think that possibly the current way of embedding javadoc-like
documentation into sources is only a stepping stone into the direction
generally pointed to by what Wirth called "literate programming".

That was Knuth, not Wirth. But this is not really that relevant.

Gerhard

 

[Gerhard Fiedler]

Could you elaborate on that a little or point me in the right direction to
read up on it? I'm currently re-writing a large lump of my coding and trying
to use best practice. I thought it was considered good practice to make
stuff private (in this case using __ ) that wasn't intened to be accessed
from outside the function/class?

I think fuzzylollipop meant that such members should not be used from the
outside of the class; that is, they should be considered implementation,
not API. (Which is why apparently autogenerated docs leave them out.)

Gerhard

 

[Alex Martelli]

I think fuzzylollipop meant that such members should not be used from the
outside of the class; that is, they should be considered implementation,
not API. (Which is why apparently autogenerated docs leave them out.)

Unfortunately, there is a slight ambiguity here. Consider for example
the wonderful Queue class. Its methods _put, _get are not documented in
help(Queue.Queue) nor on <http://docs.python.org/lib/QueueObjects.html>,
respecting the "private" convention.

But if you read Queue.py, you'll see...:

# Override these methods to implement other queue organizations
# (e.g. stack or priority queue).
# These will only be called with appropriate locks held

...

# Put a new item in the queue
def _put(self, item):
...

etc -- i.e., Queue.py itself strongly appears to consider these methods
*protected* (in C++ parlance), NOT *private*. Indeed, the only reason
these methods are factored out is exactly as "hook methods" (meant to be
overridden by subclasses), in a beautiful example of the "Template
Method" design pattern (in the specific variant in which the methods MAY
but DON'T HAVE TO be overridden, because the base class, Queue.Queue,
already provides them with useful functionality -- a LIFO queue).

Unfortunately Python does not have any specific naming convention to
indicate "methods that should never be CALLED by client code, but may be
usefully OVERRIDDEN in subclasses", so the leading-underscore one does
double duty -- and since the simpler interpretation "this is a private
implementation detail, ignore it completely" is so much more common than
the one about "this is a hook method which you should override in a
subclass if you want to tweak some detail of functionality", it's all
too easy to forget about the latter case (fortunately Queue.Queue is
there to remind us of it.


Alex