confused about __new__

[K. Richard Pixley]

I'm confused about the following. The idea here is that the set of
instances of some class are small and finite, so I'd like to create them
at class creation time, then hijack __new__ to simply return one of the
preexisting classes instead of creating a new one each call.

This seems to work in python3, but fails in python2 with:

Foo(23)
TypeError: unbound method __Foo__new__() must be called with Foo
instance as first argument (got type instance instead)

I don't understand. Can anyone explain?

--rich

 

[Steven D'Aprano]

I'm confused about the following. The idea here is that the set of
instances of some class are small and finite, so I'd like to create them
at class creation time, then hijack __new__ to simply return one of the
preexisting classes instead of creating a new one each call.

I'm surprised it works in Python3. Try this in Python 2 instead:

Foo.__new__ = staticmethod(__Foo__new__)

 

[K Richard Pixley]

I'm surprised it works in Python3. Try this in Python 2 instead:

Foo.__new__ = staticmethod(__Foo__new__)

Excellent. Thank you.

--rich

 

[Fredrik Tolf]

I don't understand. Can anyone explain?

I'm also a bit confused about __new__. I'd very much appreciate it if
someone could explain the following aspects of it:

* The manual (<http://docs.python.org/reference/datamodel.html>) says
that __new__ is "a static method (special-cased so you need not declare
it as such)". What does "special-cased" mean? Apparently, for
instance, in OP's case, Python did not automatically detect that it
should not be bound as a method.

* Is there any part of the manual that explains, holistically, the
greater context of object instantiation into which __new__ fits? I can
only find small parts of it spread around the documentation for __new__
and __init__, but no complete explanation of it. There are several
things I'm wondering about, like what it means to call a type object at
all; how methods, properties and the like are bound; how pickle can
instantiate a class without calling __init__; when and whether __dict__
is created and a couple of other things. Is there a complete
documentation of the process anywhere that I haven't managed to find?

 

[Ian Kelly]

I'm also a bit confused about __new__. I'd very much appreciate it if
someone could explain the following aspects of it:

 * The manual (<http://docs.python.org/reference/datamodel.html>) says
  that __new__ is "a static method (special-cased so you need not declare
  it as such)". What does "special-cased" mean? Apparently, for
  instance, in OP's case,  Python did not automatically detect that it
  should not be bound as a method.

It apparently has to do with the class creation. For the
special-casing to happen, the __new__ method has to be present when
the class is created. If it is, then it automatically gets wrapped in
a staticmethod. In the OP's case, he was adding the __new__ method
after class creation, so the wrapping did not happen automatically.
Compare:
.... __new__ = my_new
....

class Bar(object): pass ....
Bar.__new__ = my_new
Foo.__dict__['__new__']

Bar.__dict__['__new__']

 * Is there any part of the manual that explains, holistically, the
  greater context of object instantiation into which __new__ fits? I can
  only find small parts of it spread around the documentation for __new__
  and __init__, but no complete explanation of it. There are several
  things I'm wondering about, like what it means to call a type object at
  all;

I don't know of anything that organizes it that way specifically, but
I believe the Data Model reference pretty much covers what you're
looking for. From the type hierarchy, under "Callable Types":
http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy
"""
Class Types
Class types, or “new-style classes,” are callable. These objects
normally act as factories for new instances of themselves, but
variations are possible for class types that override __new__(). The
arguments of the call are passed to __new__() and, in the typical
case, to __init__() to initialize the new instance.
"""

AFAIK, that's pretty much it. When a type is called, __new__ is
called to create the new instance, and then __init__ is called to
initialize it (if __new__ returned an instance of the type).

how methods, properties and the like are bound;

When they're accessed, using the descriptor protocol, not as part of
the instantiation process. See:
http://docs.python.org/reference/datamodel.html#invoking-descriptors

how pickle can
  instantiate a class without calling __init__;

By calling the __new__ method directly instead of calling the type:
http://docs.python.org/library/pickle.html#object.__getnewargs__

when and whether __dict__
  is created and a couple of other things.

Under the hood as part of the object creation process, unless the
class uses __slots__.

 

[Lie Ryan]

I'm also a bit confused about __new__. I'd very much appreciate it if
someone could explain the following aspects of it:

* The manual (<http://docs.python.org/reference/datamodel.html>) says
that __new__ is "a static method (special-cased so you need not declare
it as such)". What does "special-cased" mean? Apparently, for
instance, in OP's case, Python did not automatically detect that it
should not be bound as a method.

If you declare new in the regular way:

class Foo(object):
def __new__(cls):
...

Python would create __new__ as a static method even without applying the
staticmethod decorator; that python does not detect to special case
__new__ when it is added dynamically is probably an oversight in Python
2.x that was fixed in Python 3.x.

 

[K Richard Pixley]

I'm also a bit confused about __new__. I'd very much appreciate it if
someone could explain the following aspects of it:

* The manual (<http://docs.python.org/reference/datamodel.html>) says
that __new__ is "a static method (special-cased so you need not declare
it as such)". What does "special-cased" mean? Apparently, for
instance, in OP's case, Python did not automatically detect that it
should not be bound as a method.

* Is there any part of the manual that explains, holistically, the
greater context of object instantiation into which __new__ fits? I can
only find small parts of it spread around the documentation for __new__
and __init__, but no complete explanation of it. There are several
things I'm wondering about, like what it means to call a type object at
all; how methods, properties and the like are bound; how pickle can
instantiate a class without calling __init__; when and whether __dict__
is created and a couple of other things. Is there a complete
documentation of the process anywhere that I haven't managed to find?

Everything I know about this stuff I got by reading the manual. Search
for __init__ and you'll find a pretty complete description of how
objects are created, how to perturb that process, how attributes are
looked up, and how to perturb that process. At least, you do in both
python 2 and python 3 current manuals, I think.

The conceptual leap for me was in recognizing that a class is just an
object. The best way, (imo, so far), to create a singleton in python is
to use the class itself as the singleton rather than ever instantiating
it. With a little work, you can even prevent it from ever being
instantiated.

The concept of a "factory" gets a little weird then, as a factory might
return an instantiation of some class but it might instead return a
class proper - perhaps even an automatically constructed class - which
is a somewhat different sort of factory.

Calling a type is a little bit different. Calling a class is how you
initiate an instantiation of that class. Calling an instantiation leads
to __call__, (which may or may not have much semantic meaning, depending
on your class). It's the difference between C() and C()(). (And the
analogy holds for other builtin types, I think).

Super() is also fairly well documented.

But __metaclass__ could use a few more examples, imo. I'm still not
entirely clear on __metaclass__. As I understand it, (which may well be
completely wrong), __metaclass_ is the means of perturbing the results
of "isinstance". You can create, (or delete), inheritance arbitrarily,
and without even requiring real, existing classes to do it. I'm not
clear on why you'd want to do this, nor why __metaclass__ is a better
mechanism than, say, any of the implementations of "interface" which
have been done in various tool kits or "mixins", which all seem to be
somewhat overlapping in functionality.

--rich

 

[Fredrik Tolf]

It apparently has to do with the class creation. For the
special-casing to happen, the __new__ method has to be present when
the class is created. If it is, then it automatically gets wrapped in
a staticmethod. In the OP's case, he was adding the __new__ method
after class creation, so the wrapping did not happen automatically.

Hmm, thank you. After trying a couple of things, it appears that the
reason OP's method worked in Python 3 is that __get__ on ordinary
functions in Python 3 simply returns the function itself when called on
type lookup, rather than Python 2's unbound method objects.

http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy
"""
Class Types
Class types, or “new-style classes,” are callable. These objects
normally act as factories for new instances of themselves, but
variations are possible for class types that override __new__(). The
arguments of the call are passed to __new__() and, in the typical
case, to __init__() to initialize the new instance.
"""

AFAIK, that's pretty much it. When a type is called, __new__ is
called to create the new instance, and then __init__ is called to
initialize it (if __new__ returned an instance of the type).

Since that description doesn't include when __dict__ is created, it isn't
complete, however. I guess it's safe to assume that __dict__ is created
inside object.__new__, but that also leaves some things unclear; see
below.

When they're accessed, using the descriptor protocol, not as part of
the instantiation process. See:
http://docs.python.org/reference/datamodel.html#invoking-descriptors

Ah, sorry. I had read that, but I had mistakenly thought that the same
instance method object was always returned, which would have implied that
it had to be stored for the instance somewhere, at some point. I see now
that that is not actually the case, however. That clears up a whole lot
for me. Thanks!

Under the hood as part of the object creation process, unless the
class uses __slots__.

And also unless the object created is of type `object' or any other
built-in type, which leaves me wondering exactly under what circumstances
__dict__ actually is created. Is it some kind of special case in
object.__new__ for types created with the `class' statement? There also
appear to be other special cases in object.__new__:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: object.__new__(dict) is not safe, use dict.__new__()

I guess my question reduces into the following: What does object.__new__
actually do, and what other special cases does it implement?

 

[Ian Kelly]

The conceptual leap for me was in recognizing that a class is just an
object.  The best way, (imo, so far), to create a singleton in python is to
use the class itself as the singleton rather than ever instantiating it.
 With a little work, you can even prevent it from ever being instantiated.

I don't think that's actually possible:

Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information..... def __new__(cls):
.... raise TypeError
.... def __init__(self):
.... raise TypeError
....<class '__main__.Foo'>

I prefer the Borg pattern for singletons myself. When you need a Borg
instance, you just instantiate the class normally, do whatever you
need with it, and then discard it normally. It doesn't matter how
many instances there are (there can even be zero) because they all
share identical state. It's better encapsulation too, since client
code does not even need to know that the class is a singleton variant;
that becomes an implementation detail.

http://www.aleax.it/Python/5ep.html
http://code.activestate.com/recipes/66531/

But __metaclass__ could use a few more examples, imo.  I'm still not
entirely clear on __metaclass__.  As I understand it, (which may well be
completely wrong), __metaclass_ is the means of perturbing the results of
"isinstance".  You can create, (or delete), inheritance arbitrarily, and
without even requiring real, existing classes to do it.

A metaclass is the class of another class (which is normally `type` if
no metaclass is specified). It's used to customize the way that the
class is created, in the same way that the class can be used to
customize the way instances are created (e.g. by overriding __new__).
You could use it to mess with the class's base classes, although I
think that usage is rare. It's more common to use it to automatically
add or modify entries in the class dict. For example, Django models
use a metaclass to collect all the fields and options declared for the
model class and to setup a default manager for the model class.

Note that many things that can be done with a metaclass can also be
done with a class decorator. The main difference as I see it is that
the metaclass is inherited by subclasses, whereas a decorator would
need to be applied individually to each subclass.

 

[K Richard Pixley]

The conceptual leap for me was in recognizing that a class is just an
object. The best way, (imo, so far), to create a singleton in python is to
use the class itself as the singleton rather than ever instantiating it.
With a little work, you can even prevent it from ever being instantiated.

I don't think that's actually possible:

Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.... def __new__(cls):
... raise TypeError
... def __init__(self):
... raise TypeError
...<class '__main__.Foo'>

Try:

class Foo(object):
def __new__(cls):
return cls

--rich

 

[Ian Kelly]

The conceptual leap for me was in recognizing that a class is just an
object.  The best way, (imo, so far), to create a singleton in pythonis
to
use the class itself as the singleton rather than ever instantiating it..
 With a little work, you can even prevent it from ever being
instantiated.

I don't think that's actually possible:

Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

class Foo:

...     def __new__(cls):
...         raise TypeError
...     def __init__(self):
...         raise TypeError
...

type(object.__new__(Foo))

<class '__main__.Foo'>

Try:

class Foo(object):
   def __new__(cls):
       return cls

Okay:

Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information..... def __new__(cls):
.... return cls
....False

 

[K Richard Pixley]

The conceptual leap for me was in recognizing that a class is just an
object. The best way, (imo, so far), to create a singleton in python is
to
use the class itself as the singleton rather than ever instantiating it.
With a little work, you can even prevent it from ever being
instantiated.

I don't think that's actually possible:

Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

class Foo:

... def __new__(cls):
... raise TypeError
... def __init__(self):
... raise TypeError
...

type(object.__new__(Foo))

<class '__main__.Foo'>

Try:

class Foo(object):
def __new__(cls):
return cls

Okay:

Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.... def __new__(cls):
... return cls
...False

I'm not seeing why you're using "object.__new__". With Foo(), it seems
fine.

(e-mail address removed)> python
Python 2.7.2 (default, Dec 12 2011, 13:05:49)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)] on
darwin
Type "help", "copyright", "credits" or "license" for more information..... def __new__(cls):
.... return cls
....4298515984

(e-mail address removed)> python3
Python 3.2.2 (default, Dec 26 2011, 15:03:08)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)] on
darwin
Type "help", "copyright", "credits" or "license" for more information..... def __new__(cls):
.... return cls
....4298841008

Are you trying to demonstrate that I haven't prevented you from
instantiating Foo? If so, then I will cede that point. I certainly
don't know enough about python internals just now to even claim to be
capable of protecting a class from a hostile user. My guess is that
short of a privileged/unprivileged split, or castrating the interpreter
and locking you into it, such a protection would not be possible.

My point was that I can indeed intercept common and convenient usage to
create a lovely singleton semantic. I can't force you to use it. (Nor
do I have any motivation to so do.)

--rich

 

[Ian Kelly]

Are you trying to demonstrate that I haven't prevented you from
instantiating Foo?  If so, then I will cede that point.  I certainly don't
know enough about python internals just now to even claim to be capable of
protecting a class from a hostile user.  My guess is that short of a
privileged/unprivileged split, or castrating the interpreter and locking you
into it, such a protection would not be possible.

Yes, that was the point. I certainly don't think that anybody should
ever be creating class instances by directly invoking object.__new__
unless they really, really know what they are doing. I was just
pointing out that I don't think it's actually possible to prevent
classes from being instantiated.

My point was that I can indeed intercept common and convenient usage to
create a lovely singleton semantic.  I can't force you to use it. (Nor do I
have any motivation to so do.)

Sure, I don't disagree with that. And the object.__new__ workaround
is a general point that doesn't apply only to singletons. Any class
that customizes its instantiation process can be messed up by doing
that.