Why has __new__ been implemented as a static method?

  • Thread starter Jurko Gospodnetiæ
  • Start date
J

Jurko Gospodnetiæ

Hi all.

I was wandering why Python implements its __new__ method as a static
and not a class method?

__new__ always accepts a cls parameter, which lead me to believe it
was a class method. Also, implementing __new__ as a class method seems
natural when thinking about __new__ as 'a method you call on a class to
create a new instance of that class'.

The difference between the two implementations is admittedly not that
significant. In most cases __new__ would still need to have its cls
parameter explicitly specified since it is most often called on a class
and not on an instance.

Having it implemented as a class method would allow this:

class X:
def __new__(cls, x):
return super().__new__(x)

instead of having to do this, as we do now:

class X:
def __new__(cls, x):
return super().__new__(cls, x)


Thinking about this some more - I guess having it as a static method
allows you to use super() and still create a new 'subclass' when asked
to instantiate a base class. Perhaps some sort of a factory pattern
implementation like so:

class Base:
def __new__(cls, x):
actual_class = get_class_for(x)
return super().__new__(actual_class, x)

If __new__ were a class method, the same logic could not be so easily
implemented as you would need to call __new__ on actual_class, and so
each actual_class would need to implement its own __new__ skipping its
direct parent's __new__ - yuck... what a mess...

Could that be the actual use case causing Guido to model __new__ as a
static method? Or was it something else?


I'm not suggesting this be changed, of course. I'm just curious as to
whether I'm not fully understanding something in this part of Python. I
generally find Python highly intuitive and it's not often something
about it takes me by surprise like this. :)

Thanks in advance.

Best regards,
Jurko Gospodnetiæ

P.S.
Normally, I'd chalk this issue up under 'bike-shedding', but it came
up while teaching others about Python and so does not feel right leaving
it as 'because that's the way it is'. :)
 
S

Steven D'Aprano

Hi all.

I was wandering why Python implements its __new__ method as a static
and not a class method?

Have you read Guido's tutorial on it?

https://www.python.org/download/releases/2.2.3/descrintro

Factoid: __new__ is a static method, not a class method. I initially
thought it would have to be a class method, and that's why I added the
classmethod primitive. Unfortunately, with class methods, upcalls don't
work right in this case, so I had to make it a static method with an
explicit class as its first argument.
[end quote]

I'm not entirely sure what he means by "upcalls", but I believe it means
to call the method further up (that is, closer to the base) of the
inheritance tree.
 
G

Gregory Ewing

Steven said:
I'm not entirely sure what he means by "upcalls", but I believe it means
to call the method further up (that is, closer to the base) of the
inheritance tree.

I think it means this:

def __new__(cls):
MyBaseClass.__new__(cls)

which wouldn't work with a class method, because
MyBaseClass.__new__ would give a *bound* method
rather than an unbound one.

Python 3's version of super() seems to work with
class methods, but Python 2's doesn't (or at least
I couldn't get it to work in a brief test). Also,
I don't think super() existed at all when __new__
was invented.
 
S

Steven D'Aprano

I think it means this:

def __new__(cls):
MyBaseClass.__new__(cls)

which wouldn't work with a class method, because MyBaseClass.__new__
would give a *bound* method rather than an unbound one.

If it were a class method, you would call it by MyBaseClass.__new__()
rather than explicitly providing the cls argument.

Python 3's version of super() seems to work with class methods, but
Python 2's doesn't (or at least I couldn't get it to work in a brief
test).

Works for me. Perhaps you got your super() call wrong?

py> class MyDict(dict):
.... @classmethod
.... def fromkeys(cls, *args, **kwargs):
.... print "Calling overridden method."
.... return super(MyDict, cls).fromkeys(*args, **kwargs)
....
py> MyDict.fromkeys('abc')
Calling overridden method.
{'a': None, 'c': None, 'b': None}
py> MyDict().fromkeys('abc')
Calling overridden method.
{'a': None, 'c': None, 'b': None}

Also, I don't think super() existed at all when __new__ was
invented.

Well, I don't know about that, but super was introduced in the same
version as new-style classes with __new__.
 
G

Gregory Ewing

Steven said:
If it were a class method, you would call it by MyBaseClass.__new__()
rather than explicitly providing the cls argument.

But that wouldn't be any good, because the base __new__
needs to receive the actual class being instantiated,
not the class that the __new__ method belongs to.
 
S

Steven D'Aprano

But that wouldn't be any good, because the base __new__ needs to receive
the actual class being instantiated, not the class that the __new__
method belongs to.


Which is exactly what method descriptors -- whether instance methods or
class descriptors -- can do. Here's an example, using Python 2.7:

class MyDict(dict):
@classmethod
def fromkeys(cls, *args, **kwargs):
print "Called from", cls
return super(MyDict, cls).fromkeys(*args, **kwargs)

class AnotherDict(MyDict):
pass


And in use:

py> MyDict.fromkeys('abc')
Called from <class '__main__.MyDict'>
{'a': None, 'c': None, 'b': None}
py> AnotherDict().fromkeys('xyz')
Called from <class '__main__.AnotherDict'>
{'y': None, 'x': None, 'z': None}


In both cases, MyDict's __new__ method receives the class doing the
calling, not the class where the method is defined.

Whatever the difficulty is with __new__, it isn't something obvious.
 
R

Rotwang

Which is exactly what method descriptors -- whether instance methods or
class descriptors -- can do. Here's an example, using Python 2.7:

class MyDict(dict):
@classmethod
def fromkeys(cls, *args, **kwargs):
print "Called from", cls
return super(MyDict, cls).fromkeys(*args, **kwargs)

class AnotherDict(MyDict):
pass


And in use:

py> MyDict.fromkeys('abc')
Called from <class '__main__.MyDict'>
{'a': None, 'c': None, 'b': None}
py> AnotherDict().fromkeys('xyz')
Called from <class '__main__.AnotherDict'>
{'y': None, 'x': None, 'z': None}


In both cases, MyDict's __new__ method receives the class doing the
calling, not the class where the method is defined.

Yes, when a classmethod bound to a subclass or an instance is called.
But this is irrelevant to Gregory's point:

If it were a class method, you would call it by MyBaseClass.__new__()
rather than explicitly providing the cls argument.


The relevant behaviour is this:
@classmethod
def m(cls):
print("Called from", cls)
@classmethod
def m(cls):
C.m()

Called from <class '__main__.C'>


If __new__ were a classmethod, then a call to MyBaseClass.__new__()
within the body of MySubClass.__new__ would pass MyBaseClass to the
underlying function, not the MySubClass. This means that

class MySubClass(MyBaseClass):
def __new__(cls):
return MyBaseClass.__new__()

would fail, since it would return an instance of MyBaseClass rather than
MySubClass.
 
I

Ian Kelly

Which is exactly what method descriptors -- whether instance methods or
class descriptors -- can do. Here's an example, using Python 2.7:

class MyDict(dict):
@classmethod
def fromkeys(cls, *args, **kwargs):
print "Called from", cls
return super(MyDict, cls).fromkeys(*args, **kwargs)

class AnotherDict(MyDict):
pass


And in use:

py> MyDict.fromkeys('abc')
Called from <class '__main__.MyDict'>
{'a': None, 'c': None, 'b': None}
py> AnotherDict().fromkeys('xyz')
Called from <class '__main__.AnotherDict'>
{'y': None, 'x': None, 'z': None}


In both cases, MyDict's __new__ method receives the class doing the
calling, not the class where the method is defined.

Whatever the difficulty is with __new__, it isn't something obvious.

You cheated on two counts here. First, you're using super; I think
Guido's comment about "upcalls" in the link you posted earlier was in
reference to calls that explicitly name the name parent class, i.e.
"dict.fromkeys()", not "super(MyDict, cls).fromkeys()".

Second, you didn't override the method in AnotherDict, so
"MyDict.fromkeys" and "AnotherDict.fromkeys" refer to the same method,
the only difference being in which class is passed to the descriptor
when it is accessed.

Compare to this:

class MyDict(dict):
@classmethod
def fromkeys(cls, *args, **kwargs):
print "MyDict Called from", cls
return dict.fromkeys(*args, **kwargs)

class AnotherDict(MyDict):
@classmethod
def fromkeys(cls, *args, **kwargs):
print "AnotherDict Called from", cls
return MyDict.fromkeys(*args, **kwargs)
AnotherDict Called from <class '__main__.AnotherDict'>
MyDict Called from <class '__main__.MyDict'>
{'a': None, 'c': None, 'b': None}
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,901
Latest member
Noble71S45

Latest Threads

Top