What is the status of the __subclasses__ method?

[Ruud de Jong]

The question I have is: how safe / future proof / portable is the
use of the __subclasses__ method that exists for new-style classes?

Background: I thought I had found an easy-to-understand
application for metaclasses: making classes instantly aware
of their subclasses. The context was that I needed a class
hierarchy, and I wanted to be able to instantiate subclasses
by calling the root class. Which exact subclass would be
instantiated would depend on the arguments with which the root
class would be called. In effect using the root class as a factory.

I did not want to build the logic of argument-to-subclass
mapping in the root class. Instead I wanted the root class
to pass the argument (recursively) to its subclasses, and if
a subclass would accept the argument then an object instantiation
of that subclass would be returned. To do this, each class
in the hierarchy obviously needs to be aware of its subclasses.

I also did not want to revisit the base classes each time
I would create a new subclass -- I wanted the mere act
of subclassing a base class to be enough for the base class
to be aware of the subclass. So, being the kind of lazy
person who prefers to spend several hours on an interesting
problem to avoid 5 minutes of routine work, I wrote a metaclass
that would do this for me. Quite straightforward really.
Only after playing around with this for a while, and inspecting
the results, I happened to notice that new-style classes in
Python have a __subclasses__ method. This method does what
the name implies: Klass.__subclasses__() returns a list of
the direct subclasses of Klass.

I don't really mind the lost time for the creation of the
metaclass -- that was fun, and I learned some new things
along the way. But what bothers me is that there's absolutely
nothing about this method in the Python documentation.
The excellent "Python in a Nutshell" book also does not
mention it. It's almost as if this method is not meant
to be used by mortals.

When I searched python.org for "__subclasses__" I found
some more information. The __subclasses__ method appears
to exist to allow modifications of the superclass to be
percolated down to its children, mainly for speed reasons,
if I understand Tim Peter's explanation correctly
(http://mail.python.org/pipermail/python-list/2003-August/176360.html).
I also found several warnings that the __subclasses__ method
only returns classes that have been accessed previously (which
would defeat my purpose) See e.g.
http://mail.python.org/pipermail/python-bugs-list/2003-June/018299.html.
However, when I experimented with the __subclasses__ method
for my application, I always found that the subclass was known
as soon as it was defined.

So again, my question. Is it safe to use the __subclasses__
method for the purpose I described above? Or would it be safer
to revert to my home-grown metaclass solution?

Ruud de Jong

 

[Michael Hudson]

The question I have is: how safe / future proof / portable is the
use of the __subclasses__ method that exists for new-style classes?

Hmm. I think it's unlikely to go away.

When I searched python.org for "__subclasses__" I found
some more information. The __subclasses__ method appears
to exist to allow modifications of the superclass to be
percolated down to its children, mainly for speed reasons,
if I understand Tim Peter's explanation correctly
(http://mail.python.org/pipermail/python-list/2003-August/176360.html).
Yep.

I also found several warnings that the __subclasses__ method only
returns classes that have been accessed previously (which would
defeat my purpose)

Hum, you're being overly paranoid here. Python defined types are
PyType_Ready-ed as soon as they are created.

So again, my question. Is it safe to use the __subclasses__
method for the purpose I described above?

Well, I can't see into the future, but I'd feel secure using it.

I'm not sure what you describe is a very tasteful use of it though...

Or would it be safer to revert to my home-grown metaclass solution?

You might want to keep it around, I guess...

Cheers,
mwh

 

[Ruud de Jong]

Michael Hudson schreef:

Hmm. I think it's unlikely to go away.




Yep.

What I wanted to avoid is that I would rely on something that is
not part of the Python *language*, but rather of a Python
*implementation*. But then, the distinction between these two
is not always clear -- if a construct is present in every
implementation, what is the difference with it being a part
of the language?

Well, raising the question is answering it -- as long as a
construct is not officially part of the language, that construct
can in principle change or disappear if the language maintainers
find that necessary.

Hum, you're being overly paranoid here. Python defined types are
PyType_Ready-ed as soon as they are created.

Well, I guess that such cautious behavior has become sort of
second nature for me -- a consequence of having worked some
twenty years on a huge software system (think multi-million lines
of C code), with thousands of colleagues adding, deleting and
modifying code simultaneously. In spite of strict processes and
procedures, too often somebody would make themselves dependent on
some construct in another part of the system that was never
meant for public usage. And when that construct was modified
to add e.g. new functionality or for performance improvement,
all hell would break loose -- because it would break the
functionality that "illegally" used the construct (and that
had been working fine for the customers)

With this conditioning I tend to be *very* careful about
using undocumented features, like __subclasses__.

Anyway, if I understand you correctly, as long as the subclasses
are implemented in Python, they will be known to their parent
immediately. And from your other remarks I get the
impression that the __subclasses__ method is a rather permanent
feature in every Python implementation.

Well, I can't see into the future, but I'd feel secure using it.

I'm not sure what you describe is a very tasteful use of it though...

Well, not being a native english speaker, I don't quite know how to
interpret this latter remark. Please elaborate. Are you objecting
to the use of the __subclasses__ method? Or to the general mechanism
I described? Or maybe I really am paranoid, and I am reading things
that aren't there

Here is the essence of the use of __subclasses__ in the application
I referred to. Note that is a variant of the Chain-of-Responsibility
pattern, applied to the Factory Method. I know that I abused the
NotImplementedError -- in the actual application it is a custom-
defined exception. But for illustration purposes I wanted to avoid
such non-essential details.
def __new__(cls, line):
for subclass in cls.__subclasses__():
try:
return subclass.__new__(subclass, line)
except NotImplementedError:
continue
raise NotImplementedError

pass

def __new__(cls, line):
if line.startswith('RCV:'):
return object.__new__(cls)
raise NotImplementedError

def __new__(cls, line):
if line.startswith('SND:'):
return object.__new__(cls)
raise NotImplementedError


The thing to note is that I can add subclasses at will, and
never have to revisit the Event root class. This means better
maintainability and extensibility.

You might want to keep it around, I guess...

I will. I joins my collection of APL one-liners and my FORTRAN V
preprocessor

Regards,

Ruud de Jong

 

[Michael Hudson]

Michael Hudson schreef:


What I wanted to avoid is that I would rely on something that is
not part of the Python *language*, but rather of a Python
*implementation*. But then, the distinction between these two
is not always clear
Indeed.

-- if a construct is present in every implementation, what is the
difference with it being a part of the language?

Well, raising the question is answering it -- as long as a
construct is not officially part of the language, that construct
can in principle change or disappear if the language maintainers
find that necessary.

I'm not sure this attitude really applies to Python. There's no real
definition of what "officially part of the language" means.

We don't take things away for the fun of it. __subclasses__() would
only disappear if some major internal restructuring happened and it
becamse massively inconvenient to keep it. But this applies (probably
most of the time with less force) to just about anything else!

Well, I guess that such cautious behavior has become sort of
second nature for me -- a consequence of having worked some
twenty years on a huge software system (think multi-million lines
of C code), with thousands of colleagues adding, deleting and
modifying code simultaneously. In spite of strict processes and
procedures, too often somebody would make themselves dependent on
some construct in another part of the system that was never
meant for public usage.

It's exposed to Python. It it was really meant to be internal, that
wouldn't have happened.

Anyway, if I understand you correctly, as long as the subclasses
are implemented in Python, they will be known to their parent
immediately.
Yes.

And from your other remarks I get the impression that the
__subclasses__ method is a rather permanent feature in every Python
implementation.

Well, it's only present in one implementation at the moment, for what
that's worth...

Well, not being a native english speaker, I don't quite know how to
interpret this latter remark. Please elaborate. Are you objecting
to the use of the __subclasses__ method?
No.

Or to the general mechanism I described?

I didn't really spend long trying to understand what you said you were
trying to do, but it sounded a little strange. That's all.

Here is the essence of the use of __subclasses__ in the application
I referred to. Note that is a variant of the Chain-of-Responsibility
pattern, applied to the Factory Method. I know that I abused the
NotImplementedError -- in the actual application it is a custom-
defined exception. But for illustration purposes I wanted to avoid
such non-essential details.

def __new__(cls, line):
for subclass in cls.__subclasses__():
try:
return subclass.__new__(subclass, line)
except NotImplementedError:
continue
raise NotImplementedError


def __new__(cls, line):
if line.startswith('RCV:'):
return object.__new__(cls)
raise NotImplementedError


def __new__(cls, line):
if line.startswith('SND:'):
return object.__new__(cls)
raise NotImplementedError

That's cute.

The thing to note is that I can add subclasses at will, and never
have to revisit the Event root class. This means better
maintainability and extensibility.

Yes. I've actually used __subclasses__ for a somewhat similar
purpose, having a Resource class whose subclasses know how to find all
instances of each resource type and so being able to find all
resources by looking through subclasses.

I will note that there's some thing *slightly* odd going on here,
which perhaps is highlighted by considering what happens if you
subclass OutgoingMsgEvent. In the code you posted, this subclass
wouldn't be picked up. You could traverse the subclass graph easily
enough, but it seems to me that "is able to instantiate events" is
more like a "class-instance" relationship (flat) than a
"class-subclass" relationship (possibly nested) and so it might be
more appropriate to make all your Event classes instances of a custom
metaclass that keeps track of its instances and then create events by
calling a method on the metaclass.

But, whatever.

I will. I joins my collection of APL one-liners and my FORTRAN V
preprocessor

Cheers,
mwh

 

[Terry Reedy]

I'm not sure this attitude really applies to Python. There's no real
definition of what "officially part of the language" means.

I think 'in the Python Reference Manual' qualifies pretty well as defining
the language. The Python Library Reference fleshes out the current PSF
CPython distribution. Any discrepancies between these and the (C)Python
implementation are considered bugs to be fixed. Guido has (as I remember)
occasionally omitted (and rejected patches about) items that he wants to
remain unofficial implementation details subject to possible change.

Someone using an undocumented implementation details like __subclasses__()
might especially want to test code against beta releases as they come out
to either protest a change or start adjusting. But the OP can relex
somewhat since code-breaking changes are not made intentionally without
some clear benefit otherwise.

Terry J. Reedy