What is the status of the __subclasses__ method?

Discussion in 'Python' started by Ruud de Jong, Feb 27, 2004.

  1. Ruud de Jong

    Ruud de Jong Guest

    The question I have is: how safe / future proof / portable is the
    use of the __subclasses__ method that exists for new-style classes?

    Background: I thought I had found an easy-to-understand
    application for metaclasses: making classes instantly aware
    of their subclasses. The context was that I needed a class
    hierarchy, and I wanted to be able to instantiate subclasses
    by calling the root class. Which exact subclass would be
    instantiated would depend on the arguments with which the root
    class would be called. In effect using the root class as a factory.

    I did not want to build the logic of argument-to-subclass
    mapping in the root class. Instead I wanted the root class
    to pass the argument (recursively) to its subclasses, and if
    a subclass would accept the argument then an object instantiation
    of that subclass would be returned. To do this, each class
    in the hierarchy obviously needs to be aware of its subclasses.

    I also did not want to revisit the base classes each time
    I would create a new subclass -- I wanted the mere act
    of subclassing a base class to be enough for the base class
    to be aware of the subclass. So, being the kind of lazy
    person who prefers to spend several hours on an interesting
    problem to avoid 5 minutes of routine work, I wrote a metaclass
    that would do this for me. Quite straightforward really.
    Only after playing around with this for a while, and inspecting
    the results, I happened to notice that new-style classes in
    Python have a __subclasses__ method. This method does what
    the name implies: Klass.__subclasses__() returns a list of
    the direct subclasses of Klass.

    I don't really mind the lost time for the creation of the
    metaclass -- that was fun, and I learned some new things
    along the way. But what bothers me is that there's absolutely
    nothing about this method in the Python documentation.
    The excellent "Python in a Nutshell" book also does not
    mention it. It's almost as if this method is not meant
    to be used by mortals.

    When I searched python.org for "__subclasses__" I found
    some more information. The __subclasses__ method appears
    to exist to allow modifications of the superclass to be
    percolated down to its children, mainly for speed reasons,
    if I understand Tim Peter's explanation correctly
    (http://mail.python.org/pipermail/python-list/2003-August/176360.html).
    I also found several warnings that the __subclasses__ method
    only returns classes that have been accessed previously (which
    would defeat my purpose) See e.g.
    http://mail.python.org/pipermail/python-bugs-list/2003-June/018299.html.
    However, when I experimented with the __subclasses__ method
    for my application, I always found that the subclass was known
    as soon as it was defined.

    So again, my question. Is it safe to use the __subclasses__
    method for the purpose I described above? Or would it be safer
    to revert to my home-grown metaclass solution?

    Ruud de Jong
    Ruud de Jong, Feb 27, 2004
    #1
    1. Advertising

  2. Ruud de Jong <> writes:

    > The question I have is: how safe / future proof / portable is the
    > use of the __subclasses__ method that exists for new-style classes?


    Hmm. I think it's unlikely to go away.

    > When I searched python.org for "__subclasses__" I found
    > some more information. The __subclasses__ method appears
    > to exist to allow modifications of the superclass to be
    > percolated down to its children, mainly for speed reasons,
    > if I understand Tim Peter's explanation correctly
    > (http://mail.python.org/pipermail/python-list/2003-August/176360.html).


    Yep.

    > I also found several warnings that the __subclasses__ method only
    > returns classes that have been accessed previously (which would
    > defeat my purpose)


    Hum, you're being overly paranoid here. Python defined types are
    PyType_Ready-ed as soon as they are created.

    > So again, my question. Is it safe to use the __subclasses__
    > method for the purpose I described above?


    Well, I can't see into the future, but I'd feel secure using it.

    I'm not sure what you describe is a very tasteful use of it though...

    > Or would it be safer to revert to my home-grown metaclass solution?


    You might want to keep it around, I guess...

    Cheers,
    mwh

    --
    I would hereby duly point you at the website for the current pedal
    powered submarine world underwater speed record, except I've lost
    the URL. -- Callas, cam.misc
    Michael Hudson, Feb 27, 2004
    #2
    1. Advertising

  3. Ruud de Jong

    Ruud de Jong Guest

    Michael Hudson schreef:

    > Ruud de Jong <> writes:
    >
    >
    >>The question I have is: how safe / future proof / portable is the
    >>use of the __subclasses__ method that exists for new-style classes?

    >
    >
    > Hmm. I think it's unlikely to go away.
    >
    >
    >>When I searched python.org for "__subclasses__" I found
    >>some more information. The __subclasses__ method appears
    >>to exist to allow modifications of the superclass to be
    >>percolated down to its children, mainly for speed reasons,
    >>if I understand Tim Peter's explanation correctly
    >>(http://mail.python.org/pipermail/python-list/2003-August/176360.html).

    >
    >
    > Yep.


    What I wanted to avoid is that I would rely on something that is
    not part of the Python *language*, but rather of a Python
    *implementation*. But then, the distinction between these two
    is not always clear -- if a construct is present in every
    implementation, what is the difference with it being a part
    of the language?

    Well, raising the question is answering it -- as long as a
    construct is not officially part of the language, that construct
    can in principle change or disappear if the language maintainers
    find that necessary.


    >>I also found several warnings that the __subclasses__ method only
    >>returns classes that have been accessed previously (which would
    >>defeat my purpose)

    >
    >
    > Hum, you're being overly paranoid here. Python defined types are
    > PyType_Ready-ed as soon as they are created.
    >


    Well, I guess that such cautious behavior has become sort of
    second nature for me -- a consequence of having worked some
    twenty years on a huge software system (think multi-million lines
    of C code), with thousands of colleagues adding, deleting and
    modifying code simultaneously. In spite of strict processes and
    procedures, too often somebody would make themselves dependent on
    some construct in another part of the system that was never
    meant for public usage. And when that construct was modified
    to add e.g. new functionality or for performance improvement,
    all hell would break loose -- because it would break the
    functionality that "illegally" used the construct (and that
    had been working fine for the customers)

    With this conditioning I tend to be *very* careful about
    using undocumented features, like __subclasses__.

    Anyway, if I understand you correctly, as long as the subclasses
    are implemented in Python, they will be known to their parent
    immediately. And from your other remarks I get the
    impression that the __subclasses__ method is a rather permanent
    feature in every Python implementation.

    >
    >>So again, my question. Is it safe to use the __subclasses__
    >>method for the purpose I described above?

    >
    >
    > Well, I can't see into the future, but I'd feel secure using it.
    >
    > I'm not sure what you describe is a very tasteful use of it though...
    >

    Well, not being a native english speaker, I don't quite know how to
    interpret this latter remark. Please elaborate. Are you objecting
    to the use of the __subclasses__ method? Or to the general mechanism
    I described? Or maybe I really am paranoid, and I am reading things
    that aren't there :)

    Here is the essence of the use of __subclasses__ in the application
    I referred to. Note that is a variant of the Chain-of-Responsibility
    pattern, applied to the Factory Method. I know that I abused the
    NotImplementedError -- in the actual application it is a custom-
    defined exception. But for illustration purposes I wanted to avoid
    such non-essential details.

    >>> class Event(object):

    def __new__(cls, line):
    for subclass in cls.__subclasses__():
    try:
    return subclass.__new__(subclass, line)
    except NotImplementedError:
    continue
    raise NotImplementedError


    >>> class MsgEvent(Event):

    pass

    >>> class IncomingMsgEvent(MsgEvent):

    def __new__(cls, line):
    if line.startswith('RCV:'):
    return object.__new__(cls)
    raise NotImplementedError


    >>> class OutgoingMsgEvent(MsgEvent):

    def __new__(cls, line):
    if line.startswith('SND:'):
    return object.__new__(cls)
    raise NotImplementedError


    >>> x = Event('RCV: blah blah')
    >>> x

    <__main__.IncomingMsgEvent object at 0x00AD2D30>
    >>>


    The thing to note is that I can add subclasses at will, and
    never have to revisit the Event root class. This means better
    maintainability and extensibility.

    >
    >>Or would it be safer to revert to my home-grown metaclass solution?

    >
    >
    > You might want to keep it around, I guess...
    >

    I will. I joins my collection of APL one-liners and my FORTRAN V
    preprocessor :)

    Regards,

    Ruud de Jong
    Ruud de Jong, Feb 28, 2004
    #3
  4. Ruud de Jong <> writes:

    > Michael Hudson schreef:
    >
    > > Ruud de Jong <> writes:
    > >
    > >>The question I have is: how safe / future proof / portable is the
    > >>use of the __subclasses__ method that exists for new-style classes?

    > > Hmm. I think it's unlikely to go away.
    > >
    > >>When I searched python.org for "__subclasses__" I found
    > >>some more information. The __subclasses__ method appears
    > >>to exist to allow modifications of the superclass to be
    > >>percolated down to its children, mainly for speed reasons,
    > >>if I understand Tim Peter's explanation correctly
    > >> (http://mail.python.org/pipermail/python-list/2003-August/176360.html).

    > > Yep.

    >
    > What I wanted to avoid is that I would rely on something that is
    > not part of the Python *language*, but rather of a Python
    > *implementation*. But then, the distinction between these two
    > is not always clear


    Indeed.

    > -- if a construct is present in every implementation, what is the
    > difference with it being a part of the language?
    >
    > Well, raising the question is answering it -- as long as a
    > construct is not officially part of the language, that construct
    > can in principle change or disappear if the language maintainers
    > find that necessary.


    I'm not sure this attitude really applies to Python. There's no real
    definition of what "officially part of the language" means.

    We don't take things away for the fun of it. __subclasses__() would
    only disappear if some major internal restructuring happened and it
    becamse massively inconvenient to keep it. But this applies (probably
    most of the time with less force) to just about anything else!

    > >>I also found several warnings that the __subclasses__ method only
    > >>returns classes that have been accessed previously (which would
    > >>defeat my purpose)

    > > Hum, you're being overly paranoid here. Python defined types are
    > > PyType_Ready-ed as soon as they are created.
    > >

    >
    > Well, I guess that such cautious behavior has become sort of
    > second nature for me -- a consequence of having worked some
    > twenty years on a huge software system (think multi-million lines
    > of C code), with thousands of colleagues adding, deleting and
    > modifying code simultaneously. In spite of strict processes and
    > procedures, too often somebody would make themselves dependent on
    > some construct in another part of the system that was never
    > meant for public usage.


    It's exposed to Python. It it was really meant to be internal, that
    wouldn't have happened.

    > Anyway, if I understand you correctly, as long as the subclasses
    > are implemented in Python, they will be known to their parent
    > immediately.


    Yes.

    > And from your other remarks I get the impression that the
    > __subclasses__ method is a rather permanent feature in every Python
    > implementation.


    Well, it's only present in one implementation at the moment, for what
    that's worth...

    > >
    > >>So again, my question. Is it safe to use the __subclasses__
    > >> method for the purpose I described above?

    > > Well, I can't see into the future, but I'd feel secure using it.
    > > I'm not sure what you describe is a very tasteful use of it
    > > though...
    > >

    > Well, not being a native english speaker, I don't quite know how to
    > interpret this latter remark. Please elaborate. Are you objecting
    > to the use of the __subclasses__ method?


    No.

    > Or to the general mechanism I described?


    I didn't really spend long trying to understand what you said you were
    trying to do, but it sounded a little strange. That's all.

    > Here is the essence of the use of __subclasses__ in the application
    > I referred to. Note that is a variant of the Chain-of-Responsibility
    > pattern, applied to the Factory Method. I know that I abused the
    > NotImplementedError -- in the actual application it is a custom-
    > defined exception. But for illustration purposes I wanted to avoid
    > such non-essential details.
    >
    > >>> class Event(object):

    > def __new__(cls, line):
    > for subclass in cls.__subclasses__():
    > try:
    > return subclass.__new__(subclass, line)
    > except NotImplementedError:
    > continue
    > raise NotImplementedError
    >
    >
    > >>> class MsgEvent(Event):

    > pass
    >
    > >>> class IncomingMsgEvent(MsgEvent):

    > def __new__(cls, line):
    > if line.startswith('RCV:'):
    > return object.__new__(cls)
    > raise NotImplementedError
    >
    >
    > >>> class OutgoingMsgEvent(MsgEvent):

    > def __new__(cls, line):
    > if line.startswith('SND:'):
    > return object.__new__(cls)
    > raise NotImplementedError
    >
    >
    > >>> x = Event('RCV: blah blah')
    > >>> x

    > <__main__.IncomingMsgEvent object at 0x00AD2D30>
    > >>>


    That's cute.

    > The thing to note is that I can add subclasses at will, and never
    > have to revisit the Event root class. This means better
    > maintainability and extensibility.


    Yes. I've actually used __subclasses__ for a somewhat similar
    purpose, having a Resource class whose subclasses know how to find all
    instances of each resource type and so being able to find all
    resources by looking through subclasses.

    I will note that there's some thing *slightly* odd going on here,
    which perhaps is highlighted by considering what happens if you
    subclass OutgoingMsgEvent. In the code you posted, this subclass
    wouldn't be picked up. You could traverse the subclass graph easily
    enough, but it seems to me that "is able to instantiate events" is
    more like a "class-instance" relationship (flat) than a
    "class-subclass" relationship (possibly nested) and so it might be
    more appropriate to make all your Event classes instances of a custom
    metaclass that keeps track of its instances and then create events by
    calling a method on the metaclass.

    But, whatever.

    > >
    > >>Or would it be safer to revert to my home-grown metaclass solution?

    > > You might want to keep it around, I guess...
    > >

    > I will. I joins my collection of APL one-liners and my FORTRAN V
    > preprocessor :)


    :)

    Cheers,
    mwh

    --
    Linux: Horse. Like a wild horse, fun to ride. Also prone to
    throwing you and stamping you into the ground because it doesn't
    like your socks. -- Jim's pedigree of operating systems, asr
    Michael Hudson, Feb 28, 2004
    #4
  5. Ruud de Jong

    Terry Reedy Guest

    "Michael Hudson" <> wrote in message
    news:...
    > I'm not sure this attitude really applies to Python. There's no real
    > definition of what "officially part of the language" means.


    I think 'in the Python Reference Manual' qualifies pretty well as defining
    the language. The Python Library Reference fleshes out the current PSF
    CPython distribution. Any discrepancies between these and the (C)Python
    implementation are considered bugs to be fixed. Guido has (as I remember)
    occasionally omitted (and rejected patches about) items that he wants to
    remain unofficial implementation details subject to possible change.

    Someone using an undocumented implementation details like __subclasses__()
    might especially want to test code against beta releases as they come out
    to either protest a change or start adjusting. But the OP can relex
    somewhat since code-breaking changes are not made intentionally without
    some clear benefit otherwise.

    Terry J. Reedy
    Terry Reedy, Feb 28, 2004
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. saha
    Replies:
    0
    Views:
    4,042
  2. Shug
    Replies:
    13
    Views:
    890
    Philipp
    Dec 15, 2006
  3. Shug
    Replies:
    13
    Views:
    946
    Philipp
    Dec 15, 2006
  4. samwyse
    Replies:
    1
    Views:
    562
    Gabriel Genellina
    Sep 14, 2009
  5. harryos
    Replies:
    0
    Views:
    377
    harryos
    Oct 13, 2010
Loading...

Share This Page