function v. method

[danielx]

At first I was going to post the following:

<!-- beginning of my original post -->

I just discovered the inspect module, which contains the isfunction and
ismethod functions. For some reason, I used to be under the impression
that Python methods are no different from Python functions. Naturally,
I wondered why both of these needed to exist (I also later discovered
the isroutine function). So I started to experiment at prompt. Here's
what I did:
.... def meth(self): pass
....True

Imagine my surprise. Why would Python do this?

<!-- end of my original post, with ending censored -->

but then I tried this:

res = Foo.__dict__['func']
res is dan

True

And it all started to make sense. The surprising thing turned out to be
not so surprising: When the expression Foo.func gets evaluated, we get
a method which is just a wrapper around dan. Therefore, f is not dan!

This is still a little bit of magic, which gets me thinking again about
the stuff I self-censored. Since the dot syntax does something special
and unexpected in my case, why not use some more dot-magic to implement
privates? Privates don't have to be entirely absent from Klass.__dict__
(which would make Python not introspective); they can just be invisible
when using the dot-syntax.

BTW, I am aware of Python's name mangling feature.

 

[bayerj]

I guess the python devs are not interested in implementing something
that would require new syntax and does not give something entirely new
to the language.

The good thing about python is, that the devs are only implementing
ideas that are very cool. There are a lot of cool (!= very cool) ideas
in rejected peps - but they were never implemented for good reasons.

If you *really* need privates, just use the naming convention.

 

[Leif K-Brooks]

This is still a little bit of magic, which gets me thinking again about
the stuff I self-censored. Since the dot syntax does something special
and unexpected in my case, why not use some more dot-magic to implement
privates? Privates don't have to be entirely absent from Klass.__dict__
(which would make Python not introspective); they can just be invisible
when using the dot-syntax.

You can do this now, kind of:
.... x = property()
.... def doStuffWithX(self):
.... self.__dict__['x'] = 123
.... print self.__dict__['x']
....Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: unreadable attribute

If you're proposing that self.x and bar.x should give different results,
then that's quite a bit more magic than property() and methods use. They
both use the descriptor API; for more information on that, read
<http://python.org/download/releases/2.2.3/descrintro/>.

 

[Duncan Booth]

True

Imagine my surprise. Why would Python do this?

Nothing magical happens at the point you said. The assignment is just an
assignment and you saw that the function was stored unchanged when you
looked in the class's dictionary.

The magic happens when you access a member of a class or an instance. If
the value is a descriptor then its __get__ method is called. Roughly
speaking, Foo.func is equivalent to:

Foo.__dict__['func'].__get__(None, Foo)

<unbound method Foo.dan>

Python does this so that it can support other descriptor types such as
property, classmethod, staticmethod.

You can see this happening if you define your own descriptor:
def __get__(self, *args):
print "__get__ called", args
return None

__get__ called (None, <class __main__.Foo at 0x00B3F930>)

http://docs.python.org/ref/descriptor-invocation.html has a description of
this. Although it claims "Note that descriptors are only invoked for new
style objects or classes (ones that subclass object() or type())." the
descriptor mechanism is partially implemented for old style classes.
Several aspects of descriptors don't work properly though in old-style
classes which is one reason why you should always use new-style classes.

Privates don't have to be entirely absent from Klass.__dict__
(which would make Python not introspective); they can just be invisible
when using the dot-syntax.

You could implement that using a data descriptor, but if you are going to
prevent access to your private variables using the dot operator, then your
code is going to look pretty silly with a lot of references to
self.__dict__['theprivate'] which doesn't gain anything in readability over
self.__theprivate.

 

[Bruno Desthuilliers]

At first I was going to post the following:

<!-- beginning of my original post -->
(snip)

<!-- end of my original post, with ending censored -->

but then I tried this:

res = Foo.__dict__['func']
res is dan

True

And it all started to make sense. The surprising thing turned out to be
not so surprising: When the expression Foo.func gets evaluated, we get
a method which is just a wrapper around dan. Therefore, f is not dan!
This is still a little bit of magic,

FWIW, the function class implements the descriptor protocol... Here's
the "magic".

which gets me thinking again about
the stuff I self-censored. Since the dot syntax does something special
and unexpected in my case,

"unexpected" ? Did you ever wondered how the instance or class was
passed as first arg when doing method calls ?

why not use some more dot-magic to implement
privates?

What for ? What makes you think we need language-inforced access
restriction ?

(snip)

BTW, I am aware of Python's name mangling feature.

Name mangling is mainly here to protect from accidental overridding. The
convention for implementation attributes is single-leading-underscore.

 

[danielx]

At first I was going to post the following:

<!-- beginning of my original post -->
(snip)

<!-- end of my original post, with ending censored -->

but then I tried this:

res = Foo.__dict__['func']
res is dan

True

And it all started to make sense. The surprising thing turned out to be
not so surprising: When the expression Foo.func gets evaluated, we get
a method which is just a wrapper around dan. Therefore, f is not dan!
This is still a little bit of magic,

FWIW, the function class implements the descriptor protocol... Here's
the "magic".

which gets me thinking again about
the stuff I self-censored. Since the dot syntax does something special
and unexpected in my case,

"unexpected" ? Did you ever wondered how the instance or class was
passed as first arg when doing method calls ?

Not knowing what's going on during method calls is exactly what
motivated me to post.

What for ? What makes you think we need language-inforced access
restriction ?

I knew someone would bring this up. The motivation would not be to
provide restriction, but to help maintain clean api's. If you intended
for users to use only a subset of the methods in your class, why not
help them learn your api by presenting the stuff they can use not along
side the stuff they should not?

Obviously, such things would be omitted from your docs, but users also
learn by interacting with Python, which is really one of Python's great
virtues. When supporting documents aren't sufficient to learn an api
(I'm sure this never happens, so just humor me), you can always turn to
interactive Python. This is exactly what it's there for. If nothing is
hidden, a user could be easily mislead to believe he can use a method
when he really shouldn't.

 

[danielx]

This is still a little bit of magic, which gets me thinking again about
the stuff I self-censored. Since the dot syntax does something special
and unexpected in my case, why not use some more dot-magic to implement
privates? Privates don't have to be entirely absent from Klass.__dict__
(which would make Python not introspective); they can just be invisible
when using the dot-syntax.

You can do this now, kind of:
... x = property()
... def doStuffWithX(self):
... self.__dict__['x'] = 123
... print self.__dict__['x']
...Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: unreadable attribute

If you're proposing that self.x and bar.x should give different results,
then that's quite a bit more magic than property() and methods use. They

Yes, I had not considered that very carefully...

both use the descriptor API; for more information on that, read
<http://python.org/download/releases/2.2.3/descrintro/>.

Let me finish reading that before I get back to your point.

 

[danielx]

True

Imagine my surprise. Why would Python do this?

Nothing magical happens at the point you said. The assignment is just an
assignment and you saw that the function was stored unchanged when you
looked in the class's dictionary.

The magic happens when you access a member of a class or an instance. If
the value is a descriptor then its __get__ method is called. Roughly
speaking, Foo.func is equivalent to:

Foo.__dict__['func'].__get__(None, Foo)

<unbound method Foo.dan>

Python does this so that it can support other descriptor types such as
property, classmethod, staticmethod.

You can see this happening if you define your own descriptor:
def __get__(self, *args):
print "__get__ called", args
return None

__get__ called (None, <class __main__.Foo at 0x00B3F930>)

http://docs.python.org/ref/descriptor-invocation.html has a description of
this. Although it claims "Note that descriptors are only invoked for new
style objects or classes (ones that subclass object() or type())." the
descriptor mechanism is partially implemented for old style classes.
Several aspects of descriptors don't work properly though in old-style
classes which is one reason why you should always use new-style classes.

Privates don't have to be entirely absent from Klass.__dict__
(which would make Python not introspective); they can just be invisible
when using the dot-syntax.

You could implement that using a data descriptor, but if you are going to
prevent access to your private variables using the dot operator, then your
code is going to look pretty silly with a lot of references to
self.__dict__['theprivate'] which doesn't gain anything in readability over
self.__theprivate.

I believe you are talking about the same thing as Leif, but I can't
quite tell. I'll need to read your paper as well .

 

[Bruno Desthuilliers]

(snip)


Not knowing what's going on during method calls is exactly what
motivated me to post.

!-)

Ok, so now you have to read about descriptors, __getattribute__ and
__setattr__.

I knew someone would bring this up.
Indeed.

The motivation would not be to
provide restriction, but to help maintain clean api's. If you intended
for users to use only a subset of the methods in your class, why not
help them learn your api by presenting the stuff they can use not along
side the stuff they should not?

Obviously, such things would be omitted from your docs, but users also
learn by interacting with Python, which is really one of Python's great
virtues. When supporting documents aren't sufficient to learn an api
(I'm sure this never happens, so just humor me), you can always turn to
interactive Python.

....and source code...

This is exactly what it's there for. If nothing is
hidden, a user could be easily mislead to believe he can use a method
when he really shouldn't.

Single leading underscore means "implementation, don't touch or you're
on your own".

 

[danielx]

!-)

Ok, so now you have to read about descriptors, __getattribute__ and
__setattr__.


...and source code...

*shudders* What happened to all the goodness of abstraction?

Single leading underscore means "implementation, don't touch or you're
on your own".

I'll remember that. I had forgotten what the convention was for
labeling things "do not touch".

 

[fuzzylollipop]

At first I was going to post the following:

<!-- beginning of my original post -->
(snip)

<!-- end of my original post, with ending censored -->

but then I tried this:


res = Foo.__dict__['func']
res is dan

True

And it all started to make sense. The surprising thing turned out to be
not so surprising: When the expression Foo.func gets evaluated, we get
a method which is just a wrapper around dan. Therefore, f is not dan!
This is still a little bit of magic,

FWIW, the function class implements the descriptor protocol... Here's
the "magic".

which gets me thinking again about
the stuff I self-censored. Since the dot syntax does something special
and unexpected in my case,

"unexpected" ? Did you ever wondered how the instance or class was
passed as first arg when doing method calls ?

Not knowing what's going on during method calls is exactly what
motivated me to post.

What for ? What makes you think we need language-inforced access
restriction ?

I knew someone would bring this up. The motivation would not be to
provide restriction, but to help maintain clean api's. If you intended
for users to use only a subset of the methods in your class, why not
help them learn your api by presenting the stuff they can use not along
side the stuff they should not?

Obviously, such things would be omitted from your docs, but users also
learn by interacting with Python, which is really one of Python's great
virtues. When supporting documents aren't sufficient to learn an api
(I'm sure this never happens, so just humor me), you can always turn to
interactive Python. This is exactly what it's there for. If nothing is
hidden, a user could be easily mislead to believe he can use a method
when he really shouldn't.

if you prefix with a single underscore, that tells the user, DON'T MESS
WITH ME FROM OUTSIDE! I AM AN IMPLEMENTATION DETAIL!

and it gets ommited from all the doc generation

if you prefix with a double underscore, then they have to go even
FARTHER out of their way to shoot themselves in the foot.

Python takes the stance of "personal responsiblity" when it comes to
access control. Which in NO WAY dimishes its "robustness" or anything
else.

just read the DailyWTF.com, incompentent people will abuse any language
features in any language, and will figure out how to break programatic
access control no matter how it is implemented. Matter of fact, Java
which in another thread someone was ADAMANT that did not expose private
anything from reflection ( and was wrong ) specifically allows you to
access all the private members, functions, everything. You just need to
tell it to turn all the "safeties" off.

From the api:

public void setAccessible(boolean flag)
throws SecurityException

Set the accessible flag for this object to the indicated boolean value.
A value of true indicates that the reflected object should suppress
Java language access checking when it is used. A value of false
indicates that the reflected object should enforce Java language access
checks.

Setting the accessible flag in a reflected object permits sophisticated
applications with sufficient privilege, such as Java Object
Serialization or other persistence mechanisms, to manipulate objects in
a manner that would normally be prohibited.

so anything added to Python to enforce "access control" would
immediately be forced to provide some means to over-ride the checks for
pickle and the like. Not to even mention the argument that it would
break crap loads of existing code base.

 

[Alex Martelli]

*shudders* What happened to all the goodness of abstraction?

<http://www.joelonsoftware.com/articles/LeakyAbstractions.htm>


Alex

 

[Diez B. Roggisch]

<http://www.joelonsoftware.com/articles/LeakyAbstractions.htm>

Misses an l at the end:

http://www.joelonsoftware.com/articles/LeakyAbstractions.html

Diez

 

[Bruno Desthuilliers]

*shudders* What happened to all the goodness of abstraction?

Compared to machine language, Python source code is really abstration.

 

[Antoon Pardon]

danielx wrote:
At first I was going to post the following:

<!-- beginning of my original post -->

(snip)

<!-- end of my original post, with ending censored -->

but then I tried this:


res = Foo.__dict__['func']
res is dan

True

And it all started to make sense. The surprising thing turned out to be
not so surprising: When the expression Foo.func gets evaluated, we get
a method which is just a wrapper around dan. Therefore, f is not dan!
This is still a little bit of magic,

FWIW, the function class implements the descriptor protocol... Here's
the "magic".

which gets me thinking again about
the stuff I self-censored. Since the dot syntax does something special
and unexpected in my case,

"unexpected" ? Did you ever wondered how the instance or class was
passed as first arg when doing method calls ?

Not knowing what's going on during method calls is exactly what
motivated me to post.

why not use some more dot-magic to implement
privates?

What for ? What makes you think we need language-inforced access
restriction ?

I knew someone would bring this up. The motivation would not be to
provide restriction, but to help maintain clean api's. If you intended
for users to use only a subset of the methods in your class, why not
help them learn your api by presenting the stuff they can use not along
side the stuff they should not?

Obviously, such things would be omitted from your docs, but users also
learn by interacting with Python, which is really one of Python's great
virtues. When supporting documents aren't sufficient to learn an api
(I'm sure this never happens, so just humor me), you can always turn to
interactive Python. This is exactly what it's there for. If nothing is
hidden, a user could be easily mislead to believe he can use a method
when he really shouldn't.

if you prefix with a single underscore, that tells the user, DON'T MESS
WITH ME FROM OUTSIDE! I AM AN IMPLEMENTATION DETAIL!

Personnaly I don't like this convention. It isn't clear enough.

Suppose I am writing my own module, I use an underscore, to
mark variables which are an implementation detail for my
module.

Now I need to import an other module in my module and need access
to an implementation variable from that module. So now I have
variables with an underscore which have two different meanings:

1) This is an implemantation detail of this module, It is the
users of my module who have to be extra carefull using it.

2) This is an implemantation detail of the other module,
I should be extra carefull using it.

And I find variable starting or ending with an underscore ugly.

 

[Bruno Desthuilliers]

Personnaly I don't like this convention.

To bad for you.

It isn't clear enough.

Oh yes ?

Suppose I am writing my own module, I use an underscore, to
mark variables which are an implementation detail for my
module.

Now I need to import an other module in my module and need access
to an implementation variable from that module.

So now I have
variables with an underscore which have two different meanings:

1) This is an implemantation detail of this module, It is the
users of my module who have to be extra carefull using it.

2) This is an implemantation detail of the other module,
I should be extra carefull using it.

Either you imported with the "from othermodule import *" form (which you
shouldn't do), and you *don't* have the implementation of othermodule,
or your used the "import othermodule" form, in which case it's pretty
obvious which names belongs to othermodule.

Have any other, possibly valid, reason ?

And I find variable starting or ending with an underscore ugly.

Too bad for you. Choose another language then... PHP, Perl, Ruby ?-)

 

[Gerhard Fiedler]

*shudders* What happened to all the goodness of abstraction?

Abstraction as you seem to use it requires complete docs of the interface.
Which is what you said you don't have... So the original abstractor broke
the abstraction when publishing insufficient docs, not the one who looks
into the sources to find out what actually happens.

(This independently of the merits of abstraction.)

Gerhard

 

[fuzzylollipop]

Suppose I am writing my own module, I use an underscore, to
mark variables which are an implementation detail for my
module.

Now I need to import an other module in my module and need access
to an implementation variable from that module. So now I have
variables with an underscore which have two different meanings:

you don't understand what "implementation detail" means, it means it is
NOT part of the public API and no client code should ever use it.

If you reference _vara in your code and it is in someone elses module
you don't understand YOU ARE NOT SUPPOSED TO DO THAT!

1) This is an implemantation detail of this module, It is the
users of my module who have to be extra carefull using it.

Users of your module should NEVER KNOW any of the _ or __ stuff exists
to begin with.

2) This is an implemantation detail of the other module,
I should be extra carefull using it.

You should NEVER use it.

And I find variable starting or ending with an underscore ugly.

But like my previous post states, you can mark stuff private in Java
all you want, you can still access. Having the compiler or runtime
enforce that just adds an extra step to turn that enforcement off.

You can do the same thing, in C++.

So WHY does a language need this enforcement.

And the "large code base" needs the "robustness" is not a valid
argument.

Since Apple has proven this NOT to be the case with Cocoa which is in
Objective-C and "knows about access control but doesn't really use
them"

Anyone that things a language HAS to have these constructs enforced by
the compiler or runtime, lacks disipline.

Python has exactly what every other language EFFECTIVELY has, a common
way to document or "tag" what is _private but might need "Friend"
access, and what is REALLY __private.

If you don't like this, then that is your problem go use a language
that gives you that false sense of secuirty, don't expect it to change.

 

[danielx]

Compared to machine language, Python source code is really abstration.

And machine language is an abstraction of pushing electrons around
circuits. I'm not sure I see your point, unless it is simply that
Python is easier than asm.

 

[danielx]

Abstraction as you seem to use it requires complete docs of the interface.
Which is what you said you don't have... So the original abstractor broke
the abstraction when publishing insufficient docs, not the one who looks
into the sources to find out what actually happens.

Absolutely. I didn't mean the user was breaking abstraction (let's not
blame the victim). I was saying that we should really have more
sympathy for him.