magic names in python

[per9000]

Hi,

I recently started working a lot more in python than I have done in
the past. And I discovered something that totally removed the pretty
pink clouds of beautifulness that had surrounded my previous python
experiences: magic names (I felt almost as sad as when I discovered
the strange pink worms that eat you in nethack, not to mention the
mind flayers - I really hate them).

I guess all programming languages have magic names to some extent
(f.x. classes in the "C-family" have constructors that must have the
same name as the class (foo::foo) instead of foo.__init__).

I just used a search engine a little on this topic and I found no
comprehensive list of magic names in python.

So my questions:
* is there a comprehensive list of magic names in python (so far i
know of __init__ and __repr__)?
* are these lists complete or can magic names be added over time (to
the python "core")?
* are magic names the same in different python versions?

I also tried (selected parts of(?)) the unittest package for use in
Zope and it seemed functions that I created for my test with the magic
prefix "test" were magic, other functions were not.

So another question emerges:
* is the use of magic names encouraged and/or part of good coding
practice.

Live long and prosper,
Per

--

Per Erik Strandberg
home: www.pererikstrandberg.se
work: www.incf.org
also: www.spongswedencare.se

 

[Marc 'BlackJack' Rintsch]

I just used a search engine a little on this topic and I found no
comprehensive list of magic names in python.

They are usually mentioned in parts of the docs where the relevant
functionality is explained. For example in `Special method names`_ in the
reference manual.

And of course all covered in the reference manual are under
`_ (underscore)`_ in its index. In the `library reference's index`_
you'll find some more that are used by modules.

... _`_ (underscore)`: http://docs.python.org/ref/genindex.html#letter-_
... _library reference's index: http://docs.python.org/lib/genindex.html
... _Special method names: http://docs.python.org/ref/specialnames.html

* are these lists complete or can magic names be added over time (to
the python "core")?

Magic names can be added over time. For example the ``with`` statement
introduced `__enter__()` and `__exit__()` in Python 2.5. And some special
names may even change were the docs say so or if they aren't mentioned at
all in the docs.

* are magic names the same in different python versions?
Yes.

So another question emerges:
* is the use of magic names encouraged and/or part of good coding
practice.

What do you mean by "use"? Implement them to override behavior? Yes,
that's their purpose. Invent new magic names? No of course not, they are
special for a reason: preventing name clashes with the user's names.

Ciao,
Marc 'BlackJack' Rintsch

 

[Josiah Carlson]

So my questions:
* is there a comprehensive list of magic names in python (so far i
know of __init__ and __repr__)?
* are these lists complete or can magic names be added over time (to
the python "core")?
* are magic names the same in different python versions?

I don't believe that there is a full list of all __magic__ methods. The
operator module has a fairly extensive listing of functions that call
such methods, but I know that some have been left out.

Among those that I remember off the top of my head while typing this
message...

__init__
__new__
__str__
__repr__
__len__
__nonzero__
__hash__
__cmp__ (__eq__, __ne__, __lt__, __gt__, __le__, __ge__)
__getattr__
__setattr__
__delattr__
__getitem__
__setitem__
__delitem__
__iter__
__neg__
__not__

There's also the not-magic named, but still somewhat magic .next()
method on iterators/generators.

- Josiah

 

[per9000]

[...]

So another question emerges:
* is the use of magic names encouraged and/or part of good coding
practice.

What do you mean by "use"? Implement them to override behavior? Yes,
that's their purpose. Invent new magic names? No of course not, they are
special for a reason: preventing name clashes with the user's names.

[in my taste: UGLY]
I think of something like this: Perhaps I create a class that works
with a lot of files, and with inheritance new types of files can be
opened and worked with.

When it is time for an instance of this class to die the files need to
be closed - perhaps "right now" and not when it is suitable for the
garbage collector. To facilitate this I create a number of functions
with names like close_*_file (f.x. close_indata_file,
close_outdata_file, close_error_file etc). (compare with [PyTest|
unittest] "test*" names.)

If this class has been inherited to some other class that works with
more than one indata file perhaps we want to magically close all files
that are open by calling all function in this instance that has names
matching "close_*_file".

I would consider this an ugly way of solving it.

[in my taste: NICER]
I'd perhaps add file-handles to some list (and encourage inherited
classes to use this list) and close everything in the list. I would
not use magic wildcard names.

So in other words: Do I need to go to yellow alert, and watch out for
magic names in non-core add-on packages to python? Or should I just
RTFM a little more? (I hate RTFM'ing.)

Live long and prosper,
Per

--

Per Erik Strandberg
home: www.pererikstrandberg.se
work: www.incf.org
also: www.spongswedencare.se

 

[Marc 'BlackJack' Rintsch]

[...]

So another question emerges:
* is the use of magic names encouraged and/or part of good coding
practice.

What do you mean by "use"? Implement them to override behavior? Yes,
that's their purpose. Invent new magic names? No of course not, they are
special for a reason: preventing name clashes with the user's names.

[in my taste: UGLY]
I think of something like this: Perhaps I create a class that works
with a lot of files, and with inheritance new types of files can be
opened and worked with.

When it is time for an instance of this class to die the files need to
be closed - perhaps "right now" and not when it is suitable for the
garbage collector. To facilitate this I create a number of functions
with names like close_*_file (f.x. close_indata_file,
close_outdata_file, close_error_file etc). (compare with [PyTest|
unittest] "test*" names.)

If this class has been inherited to some other class that works with
more than one indata file perhaps we want to magically close all files
that are open by calling all function in this instance that has names
matching "close_*_file".

I would consider this an ugly way of solving it.

[in my taste: NICER]
I'd perhaps add file-handles to some list (and encourage inherited
classes to use this list) and close everything in the list. I would
not use magic wildcard names.

So in other words: Do I need to go to yellow alert, and watch out for
magic names in non-core add-on packages to python? Or should I just
RTFM a little more? (I hate RTFM'ing.)

Now I'm a little confused. What does this have to do with magic names? I
thought you are talking about names that start and end with two
underscores (`__magic__`)!?

Ciao,
Marc 'BlackJack' Rintsch

 

[Diez B. Roggisch]

[...]

So another question emerges:
* is the use of magic names encouraged and/or part of good coding
practice.

What do you mean by "use"? Implement them to override behavior? Yes,
that's their purpose. Invent new magic names? No of course not, they
are
special for a reason: preventing name clashes with the user's names.

[in my taste: UGLY]
I think of something like this: Perhaps I create a class that works
with a lot of files, and with inheritance new types of files can be
opened and worked with.

When it is time for an instance of this class to die the files need to
be closed - perhaps "right now" and not when it is suitable for the
garbage collector. To facilitate this I create a number of functions
with names like close_*_file (f.x. close_indata_file,
close_outdata_file, close_error_file etc). (compare with [PyTest|
unittest] "test*" names.)

If this class has been inherited to some other class that works with
more than one indata file perhaps we want to magically close all files
that are open by calling all function in this instance that has names
matching "close_*_file".

I would consider this an ugly way of solving it.

[in my taste: NICER]
I'd perhaps add file-handles to some list (and encourage inherited
classes to use this list) and close everything in the list. I would
not use magic wildcard names.

This is a completely different topic than the usage of "magic" names in
python to allow e.g. operator overloading and the like.

So in other words: Do I need to go to yellow alert, and watch out for
magic names in non-core add-on packages to python? Or should I just
RTFM a little more? (I hate RTFM'ing.)

Certainly RTFM.

Diez

 

[markacy]

Hi,

I recently started working a lot more in python than I have done in
the past. And I discovered something that totally removed the pretty
pink clouds of beautifulness that had surrounded my previous python
experiences: magic names (I felt almost as sad as when I discovered
the strange pink worms that eat you in nethack, not to mention the
mind flayers - I really hate them).

I guess all programming languages have magic names to some extent
(f.x. classes in the "C-family" have constructors that must have the
same name as the class (foo::foo) instead of foo.__init__).

I just used a search engine a little on this topic and I found no
comprehensive list of magic names in python.

So my questions:
* is there a comprehensive list of magic names in python (so far i
know of __init__ and __repr__)?
* are these lists complete or can magic names be added over time (to
the python "core")?
* are magic names the same in different python versions?

I also tried (selected parts of(?)) the unittest package for use in
Zope and it seemed functions that I created for my test with the magic
prefix "test" were magic, other functions were not.

So another question emerges:
* is the use of magic names encouraged and/or part of good coding
practice.

Live long and prosper,
Per

--

Per Erik Strandberg
home:www.pererikstrandberg.se
work:www.incf.org
also:www.spongswedencare.se

On "magic" methods and their usage You can read here:
http://diveintopython.org/object_oriented_framework/special_class_methods.html
By the way - this is a very good book on python.
Cheers,
Marek

 

[Facundo Batista]

I don't believe that there is a full list of all __magic__ methods. The
operator module has a fairly extensive listing of functions that call
such methods, but I know that some have been left out.

There IS a full documentation of this special methods::

http://docs.python.org/dev/ref/specialnames.html

Note that they're named "special", not "magic".

The phrase "I'm not a wizard, I just use Python" is one of the best,
ever.

Regards,

 

[Lenard Lindstrom]

Hi,

I recently started working a lot more in python than I have done in
the past. And I discovered something that totally removed the pretty
pink clouds of beautifulness that had surrounded my previous python
experiences: magic names (I felt almost as sad as when I discovered
the strange pink worms that eat you in nethack, not to mention the
mind flayers - I really hate them).

I guess all programming languages have magic names to some extent
(f.x. classes in the "C-family" have constructors that must have the
same name as the class (foo::foo) instead of foo.__init__).

I just used a search engine a little on this topic and I found no
comprehensive list of magic names in python.

So my questions:
* is there a comprehensive list of magic names in python (so far i
know of __init__ and __repr__)?
* are these lists complete or can magic names be added over time (to
the python "core")?
* are magic names the same in different python versions?

I also tried (selected parts of(?)) the unittest package for use in
Zope and it seemed functions that I created for my test with the magic
prefix "test" were magic, other functions were not.

So another question emerges:
* is the use of magic names encouraged and/or part of good coding
practice.

What is "magic" about __init__ and __repr__? They are identifiers just
like "foo" or "JustAnotherClass". They have no special meaning to the
Python compiler. The leading and trailing double underscores represent
no special incantation. It is just a naming convention.

So a number of method names like __init__ and __repr__ have a
pre-defined usage. In every other respect they are just normal methods.
They can be called directly. When dir() is called on a class they show
up along with other method names. They can appear in a class declaration
or added later after the class is created, just like other methods. They
differ in that sometimes they will be called implicitly, such as when
creating a new object or by the print statement.

As to whether to use them: absolutely. __init__ is the preferred way to
initialize a new object. Providing a __repr__ or __str__ method lets
objects display descriptive and meaningful information about themselves
in print statements and exception messages. And various other "special
methods" allow for user defined versions of Python's built-in types. In
fact most built-in types can be extended by subclassing and overriding
their special methods.

Back to "magic names". Python is a consistent language. What may be
special cased - "magic" - in another language is handled normally with
Python's general purpose dynamic type system. In fact I can think of
only one name I would call "magic": __future__.

from __future__ import <feature>

does have special meaning to the Python compiler. It does more than
import a module.

 

[Steven D'Aprano]

What is "magic" about __init__ and __repr__? They are identifiers just
like "foo" or "JustAnotherClass". They have no special meaning to the
Python compiler. The leading and trailing double underscores represent
no special incantation. It is just a naming convention.

That's not quite true, as you point out:

So a number of method names like __init__ and __repr__ have a
pre-defined usage.

That makes them magic, in the best possible way.

In every other respect they are just normal methods.

That is *almost* true. Or, to put it another way, that is wrong. Some of
the double-underscore magic methods are automatically called on the class
instead of the instance, bypassing normal inheritance.

.... def foo(self):
.... return "foo called from the class"
.... def __str__(self):
.... return "__str__ called from the class"
....'__str__ called from the class'

Now see what happens when we add methods to the instance.
'foo called from the class'

So calling a random method like foo() goes through the usual procedure:
check the instance, then check the class. Now let's try it with a magic
method.

'__str__ called from the class'



[snip]

Back to "magic names". Python is a consistent language. What may be
special cased - "magic" - in another language is handled normally with
Python's general purpose dynamic type system. In fact I can think of
only one name I would call "magic": __future__.

Certainly I think __future__ is *more* magic than other double-underscore
names.

 

[per9000]

[...]

Now I'm a little confused. What does this have to do with magic names? I
thought you are talking about names that start and end with two
underscores (`__magic__`)!?

Indeed I am talking about two things at once - you are right. Since I
have problems with both of these "features" I perhaps do not separate
them very well.

I received lots of nice links to __special__ names in python, thanks.
When it comes to 'dive into python' this is like the fifth time I
heard of it so I really have to get it soon.

Still, I have problems with "magic" functions, similar to magic
numbers: http://en.wikipedia.org/wiki/Magic_number_(programming)
f.x. calling all member-functions of a class to close files as
illustrated in a previous post, or PyUnits magic "test*"-names:
http://pyunit.sourceforge.net/pyunit.html#RUNNING_CMD (there are
options here but I fell into the bear-trap of course)

Guessing from your replies this is a general problem/issue/feature and
as I understand it there is no special pythonic recommendations in
this area, right?

Also, I guess that use of magic numbers and magic should be documented
and are thus discovered by RTFM'ing.

Additional comments are welcome, or course.

All your base are belong to us,
Per

--

Per Erik Strandberg
home: www.pererikstrandberg.se
work: www.incf.org
also: www.spongswedencare.se

 

[Marc 'BlackJack' Rintsch]

Still, I have problems with "magic" functions, similar to magic
numbers: http://en.wikipedia.org/wiki/Magic_number_(programming)
f.x. calling all member-functions of a class to close files as
illustrated in a previous post, or PyUnits magic "test*"-names:
http://pyunit.sourceforge.net/pyunit.html#RUNNING_CMD (there are
options here but I fell into the bear-trap of course)

I don't see much similarity here. While magic numbers are quite
meaningless on their own, the name prefix carries at least *some*
explanation and therefore is much less magic. Maybe not even magic at all
in the sense of the Wikipedia article.

Guessing from your replies this is a general problem/issue/feature and
as I understand it there is no special pythonic recommendations in
this area, right?

Except documenting what's going on and not over(ab)using the reflective
and dynamic nature of Python to the point where the source starts to get
unreadable and too magic, AFAIK there's no special "pythonic"
recommendation.

Ciao,
Marc 'BlackJack' Rintsch

 

[ai]

I don't think it is magic. If you think it is magic, can you talk
about what's the better way and how can you implement the functions
without any magic. I can't image a lauguage without special names.
There are some special names even in Lisp. I guess you just hate the
'__'.

 

[per9000]

I don't think it is magic. If you think it is magic, can you talk
about what's the better way and how can you implement the functions
without any magic. [...]

Well, in a sense I guess all reserved words can be considered magic,
but in Python you can kill it (AFAIK that is pretty unique to python)8, 8.0)

I guess the amount and flavor of the magic is a matter of personal
taste - coming from a C*-background I pretty much like
myClass.myClass() instead of myClass.__init__() - but these are
equivalent and I guess I just have to live with the way python has
__special__ names and print a list of the cases I end up with often to
keep my memory from forgetting them.

There are some special names even in Lisp. I guess you just hate the
'__'.

Yes and no. Indeed, I find the overuse of underscores ugly, but I can
accept some use of __special__ functions. AFAIK there must be reserved
words in a programming language - f.x. names of ctors in oo-languages.
The exceptions are perhaps brainf**k and whitespace that has special
chars instead.

Also I find it unfortunate that the name mangling and underscores are
the way to make members "less public". But I can live with that - I
just have to read more python code to get the hang of it.
[]-|--<

/Per

--

Per Erik Strandberg
home: www.pererikstrandberg.se
work: www.incf.org
also: www.spongswedencare.se

 

[Lenard Lindstrom]

That's not quite true, as you point out:

Disassemble the code object of a class statement. "__init__" is just an
identifier, a Python name. So there is no "magic" here.

That makes them magic, in the best possible way.

I was careful to use the word "usage". Maybe I should have used
"protocol" or "specification" here. A method named "write" is understood
to have a particular definition and purpose. Is it "magic"? These
methods are simply callbacks. And yes, these callbacks make Python flexible.

That is *almost* true. Or, to put it another way, that is wrong. Some of
the double-underscore magic methods are automatically called on the class
instead of the instance, bypassing normal inheritance.

They exist as Python functions in the class dictionary. They have no
hidden flags that distinguish them from other class level function
attributes.

.... def foo(self):
.... return "foo called from the class"
.... def __str__(self):
.... return "__str__ called from the class"
....
'__str__ called from the class'

Now see what happens when we add methods to the instance.

'foo called from the class'

So calling a random method like foo() goes through the usual procedure:
check the instance, then check the class. Now let's try it with a magic
method.


'__str__ called from the class'

Yes, method lookup for an operation differs from attribute lookup in
new-style classes. But it is a property of new-style classes, not the
methods themselves. For instance, classic classes behave differently:
def __str__(self):
return "__str__ called from the class"

'__str__ called from the class'


I do admit that the special methods are given special treatment by the
type and ClassType types to ensure they are called by their
corresponding operations. But this special treatment is restricted to
the types themselves. In CPython an extension type can be written from
scratch that treats special methods exactly as a new-style class does.
Or an extension type can implement a completely novel approach. The
point is, at some level, the machinery which defines special method
"magic" is accessible to the programmer. So is it really "magic" or
advanced technology?

This comes down to the original posting not defining a "magic name". It
does not mention what is so objectionable about __init__ and __repr__. I
am claiming they are not as "magical" as they may first appear.

 

[Steven D'Aprano]

Disassemble the code object of a class statement. "__init__" is just an
identifier, a Python name. So there is no "magic" here.

It is not JUST an identifier, it is an identifier with a special, dare I
say *magic*, meaning.

I suspect we have different ideas of what a "magic name" is. You seem to
be under the impression that magic names are a bad thing, and that
therefore one has to minimize their number. Hence you are glossing over
the differences and emphasizing the similarities between methods (say) foo
and __init__.

I don't consider "magic name" in this content to be anything to be ashamed
of, so I'm not concerned about downplaying the differences. Yes, there are
significant similarities between __init__ and random methods you write
yourself, but the differences are equally, if not more, significant.

I was careful to use the word "usage". Maybe I should have used
"protocol" or "specification" here. A method named "write" is understood
to have a particular definition and purpose. Is it "magic"?

It could be, depending on the context.

If we're discussing file objects compared to str objects, then file
objects have a method "write" and str objects don't, and the method is
just another method.

But in the context of some third function, which takes a file-like object
with a "write" method, then yes it is magic, because any type of object
will work so long as it has a method "write" with the right semantics. Not
very much magic, but a little.

These
methods are simply callbacks. And yes, these callbacks make Python flexible.


They exist as Python functions in the class dictionary. They have no
hidden flags that distinguish them from other class level function
attributes.

They don't need special flags, because the Python compiler itself already
knows about the existence of them and takes special actions with them.

[snip]

I do admit that the special methods are given special treatment by the
type and ClassType types to ensure they are called by their
corresponding operations. But this special treatment is restricted to
the types themselves. In CPython an extension type can be written from
scratch that treats special methods exactly as a new-style class does.
Or an extension type can implement a completely novel approach. The
point is, at some level, the machinery which defines special method
"magic" is accessible to the programmer. So is it really "magic" or
advanced technology?

Does it matter? As Clarke said, any sufficiently advanced technology is
indistinguishable from magic... or to put it another way, as Larry Niven
did, any sufficiently advanced magic is indistinguishable from technology.

Since I don't actually believe in magic, as in "wishing makes it so", of
course it is technology. But most Python programmers aren't capable of, or
have any interest in, writing extension types in C or hacking the compiler.

This comes down to the original posting not defining a "magic name". It
does not mention what is so objectionable about __init__ and __repr__. I
am claiming they are not as "magical" as they may first appear.

I don't believe the Original Poster considers there is anything
objectionable about magic names like __init__ and __repr__, and even if he
does, I certainly don't.

Magic names in this context are those methods which have pre-defined
meanings to the compiler. That is mostly double-underscore methods like
__init__, but also methods like iterator.next(). If you ask yourself "What
methods do I have to over-ride to make a customized sub-class of a
built-in type?", the answer will include mostly magic names. If you ask
"What methods do I have to write for my class to work with Python
operators?", the answer will also include magic names like __add__,
__mul__, and __xor__.

 

[Lenard Lindstrom]

It is not JUST an identifier, it is an identifier with a special, dare I
say *magic*, meaning.

I suspect we have different ideas of what a "magic name" is. You seem to
be under the impression that magic names are a bad thing, and that
therefore one has to minimize their number. Hence you are glossing over
the differences and emphasizing the similarities between methods (say) foo
and __init__.

No, I don't consider magic in a programming language a bad thing.
Overloaded operators is one feature I like in a language. Using special,
or magic, methods to do this is elegant. And it would appear we are
working with the same definition as regarding methods: a magic name
method is an identifier recognized by the compiler as a special case. So
a C++ constructor is a magic name. But by the same definition Python
lacks magic methods. They have nothing to do with the compiler. They are
a run-time feature.

I don't consider "magic name" in this content to be anything to be ashamed
of, so I'm not concerned about downplaying the differences. Yes, there are
significant similarities between __init__ and random methods you write
yourself, but the differences are equally, if not more, significant.

I'm not contesting that some identifiers have special meaning or are
treated differently under certain conditions. Nor am I trying to hide
the fact. But in the context of Python I don't find it remarkable. The
Python type system is quite abstract, and class behavior is kept
separate from the byte code interpreter. It is in the built-in types
that special methods are given significance. And through these special
methods one has access to the inner workings of Python's type system and
modify the special behavior. So if that's magic, then it is a very
approachable magic.

It could be, depending on the context.

If we're discussing file objects compared to str objects, then file
objects have a method "write" and str objects don't, and the method is
just another method.

But in the context of some third function, which takes a file-like object
with a "write" method, then yes it is magic, because any type of object
will work so long as it has a method "write" with the right semantics. Not
very much magic, but a little.

Okay.


They don't need special flags, because the Python compiler itself already
knows about the existence of them and takes special actions with them.

By compiler I mean the parser and byte code generator, not the
interpreter or built-in objects. And the compiler knows nothing about
special Python methods. Neither does the byte code interpreter. By
defining ones own meta-meta class one can make __init__ and __new__
unremarkable.

[snip]

>> [snip]
>> In CPython an extension type can be written from
scratch that treats special methods exactly as a new-style class does.
Or an extension type can implement a completely novel approach. The
point is, at some level, the machinery which defines special method
"magic" is accessible to the programmer. So is it really "magic" or
advanced technology?

Does it matter? As Clarke said, any sufficiently advanced technology is
indistinguishable from magic... or to put it another way, as Larry Niven
did, any sufficiently advanced magic is indistinguishable from technology.

Since I don't actually believe in magic, as in "wishing makes it so", of
course it is technology. But most Python programmers aren't capable of, or
have any interest in, writing extension types in C or hacking the compiler.

If certain magic behaviors depend on the compiler then that magic may be
unavailable to the programmer. In Python assignment to None is now a
syntax error. I would consider this real magic since Python has no way
to declare compile time constants. But I can turn __build__ into a
special method that is called at class creation. Special methods are not
strictly limited to the ones already defined. And yes, in the sense that
magic means using obscure language features to define new behavior, this
is magic. Finally, nothing I have mentioned so far requires "hacking the
compiler".

I don't believe the Original Poster considers there is anything
objectionable about magic names like __init__ and __repr__, and even if he
does, I certainly don't.

I am not claiming that __init__ or __repr__ are objectionable. I am
suggesting that calling them "magic names" in the context of Python is
an exaggeration and overlooks the true power of the language.

Magic names in this context are those methods which have pre-defined
meanings to the compiler. That is mostly double-underscore methods like
__init__, but also methods like iterator.next(). If you ask yourself "What
methods do I have to over-ride to make a customized sub-class of a
built-in type?", the answer will include mostly magic names. If you ask
"What methods do I have to write for my class to work with Python
operators?", the answer will also include magic names like __add__,
__mul__, and __xor__.

I assumed you considered magic names as any pre-defined identifiers that
have a special meaning within the language. I am guessing that's how
others use the term. In this respect I agree that special methods are
magic names.

But in a programming language labeling a special case as "magic" is not
descriptive. It says nothing about what makes the special case unusual
or noteworthy. Is the quality unique to certain pre-defined identifiers?
Does it make use of obscure language features? Is it a glaring
inconsistency in an otherwise uniform language?

So I would say to anyone encountering Python magic for the first time to
reserve judgment until you understanding it. Then it may not seem so
peculiar after all. Don't make assumptions based on other languages.