Making immutable instances

[Antoon Pardon]

IMHO, this is usually (but not always) a mistake. (If you're programming a
missle guidance system, or it makes your program go faster it's not a
mistake )
So are PRIVATE, CONST (all types), SEALED, FINAL, etc -- even the best
programmer doesn't foresee what a user will want to do to make best use of
his components,

But maybe what the user wants no longer guarantees correct working.

and many a time. I've been annoyed (in Java and MS christelijke vorm er van.
frameworks) by not being able to access/modify/subclass a member/class that
I know is there because it has to be there (or because I can see it in the
debugger),

Maybe that it is there is just an implementation detail.

but it's not accessable because the programmer was overly clever
and had been to OOP school.

I think hiding something that is essentially an implementation detail
is good abstraction and programming practice.

A fine-grained capability architecture married
to the language and runtime where I can declare my own level cleverness to
override the programmer's would be nice, but I think Python's voluntary
DoThis, _DoThisIfYouReallyHaveTo, and __You'dBetterKnowWhatYou'reDoing__
approach is a better way to go than the undefeatable keyword approach.

I disagree.

Suppose I have the following code.

from module import __take_care__

__private_detail__ = ...

I now have two variable that are flaged the same way, but they are not.

__take_care__ is a private variable from an other module which I should
use with extreme care not to break the other package.

__private_detail__ on the other hand is just keeping private data for
my own module, which I should care about as just any other variable
in my module. It are other modules that should take special care
if they should choose to import this variable.

That is why I don't think the underscore prefixes are very usefull.
They constantly flag to take care with a specific variable, while
that variable is nothing special within the module itself.

 

[Scott David Daniels]

So, for a class that needs to set attributes in __init__ (but after
that, become immutable), how do I get around this? Should I make a
_FooFunctionality class, and then inherit from that to make Foo as the
immutable class that actually gets exported?

Typically, constants are set up in __new__ (which happens before
__init__), because by __init__ time the object is built. Remember
not to complain that you have no copy operation, because there is no
such thing as a copy of a constant (the original constant is good
enough).

--Scott David Daniels
(e-mail address removed)

 

[Ben Finney]

A type implemented in C offers different possibilities than one
implemented in Python -- no deep conceptual reason, just practical
ones.

So, in the hypothetical situation that all Python types were
implemented in pure Python, what would the ideal behaviour for
immutables be? Or would there be no immutables?

I don't think that making it explicit that you intended to document
the restriction would have changed my answer (although an explicit
acknowlegment that you're looking for restrictions against
accidental misuse rather than against determined attackers surely
would).

I'm looking for a "consenting adults" restriction: classes will have
immutable instances only where it makes sense from the class protocol.
I'm not going to lose sleep over users who go looking for trouble.

 

[Roel Schroeven]

Antoon, I don't think Mike wrote it like that

I don't even know how I spotted that, since I didn't really read that
part of the text. I guess the Dutch words caught my attention somehow.
It even took a while for me to realize that it must have been an error;
the first few seconds I was trying to find the relationship between
christianity and Java, MS or frameworks. Without success, obviously.

 

[Antoon Pardon]

Antoon, I don't think Mike wrote it like that

You are right, I don't know what happened, but a piece of text from
an other article was pasted in somehow.

My apologies to everyone and especially Mike.

 

[Mike]

There's a big difference. An immutable object has a totally different
semantic,
compared to a mutable object. If you document it to be immutable, and
maybe
even provide __eq__ /__hash__, adding attributes from it is surely an user
bug.
And surely a bug for which I'd expect an exception to be raised.

Why is it "surely" a bug? It is arguable whether adding new attributes (vs.
changing those that are already there) is, in fact, mutation.
How will adding user attibutes that your code knows nothing about break the
contracts you have specified?
If __hash__/__eq__ does not look at the new attributes, all the better - as
these attributes are just "along for the ride". (But if the hash reflected
into all attrs, things would indeed break.)
Personally, adding attributes to existing objects would not be my preferred
programming style, epsecailly in a langauge like Python with rich and
easy-to-use associative data structures, but sometimes it's the quickest or
only way to get something done -- if you are unwilling (or more likely,
unable) to modify all the levels of method signatures required to pass new
information about an object/state along.
I can see how a compile-time *warning* would be of value to show it is not
the *desired* usage pattern in the eyes of the component programmer, but
beyond that, why get in the way? Python is not Java or any other language --
sometimes I don't like the total dynamicity either (I'd like optional types
and/or type-inference), but it is what it is.

Sometimes, I play with some of my objects and I have to go back and check
documentation whether they are immutable or not, to make sure I use the
correct
usage pattern. That's fine, this is what docs are for, but couldn't Python
give
me some way to enforce this so that, if I or some other dev do the
mistake, it
doesn't go unnoticed?

It sounds like what you may want are opaque objects where you can control
access better -- if that's so, keep them all hidden, and just pass back a
handle or proxy - you could always implement restrictred/readonly or
copy-on-write semantics. (If there isn't one already, auto-proxy generation
for this sort of thing sounds like a fun python cookbook recipe...)

 

[Mike Meyer]

That is an ideal case and we all know the real world is not ideal or
every program would be bug free. And I don't think anyone would solely
relies on the language feature for spotting bugs. Whether this kind of
guard is useful is another story.

Python generally regards languages features that either require extra
typing or reduce the flexibility of the language for the purpose of
catching bugs as "not useful." Some people disagree with this, but
it's the way Python is.

<mike

 

[Mike]

Op 2005-11-24, Mike schreef <[email protected]>:
[...snip...]

...but I think Python's voluntary
DoThis, _DoThisIfYouReallyHaveTo, and __You'dBetterKnowWhatYou'reDoing__
approach is a better way to go than the undefeatable keyword approach.

I disagree.

Suppose I have the following code.

from module import __take_care__

__private_detail__ = ...

I now have two variable that are flaged the same way, but they are not.

__take_care__ is a private variable from an other module which I should
use with extreme care not to break the other package.

__private_detail__ on the other hand is just keeping private data for
my own module, which I should care about as just any other variable
in my module. It are other modules that should take special care
if they should choose to import this variable.

If you just import the module and use __private_detail__ vs.
module.__take_care__ -- that solves that problem

That is why I don't think the underscore prefixes are very usefull.
They constantly flag to take care with a specific variable, while
that variable is nothing special within the module itself.

I didn't say that the methods used were perfect or fully expressive, but
just in the right direction.
Maybe there should be a few more variations, and they should be documented
and blessed, and *tools* can honor them as they see fit. (Different syntax
hilighting, lint checking, etc.)

m

 

[Mike Meyer]

I'm looking for a "consenting adults" restriction: classes will have
immutable instances only where it makes sense from the class protocol.
I'm not going to lose sleep over users who go looking for trouble.

I think you're defining immutable differently than I do. Python
already supports attributes that clients can't rebind. You can make an
instance in which all attributes have that property. However
attributes added by a client aren't really yours. They don't change
the immutability of your attributes, or affect the behavior of your
class in any way. Such attributes don't really belong to your class;
they belong to the client.

Even in full B&D languages, it's trivial to overcome this restriction
with a subclass:
Traceback (most recent call last):

The extra class is the kind of boilerplate that Python in general has
so little of, and B&D languages so much of. In Python, I can even fix
it so *your* code uses my wrapped version:

import Finney
class Addable(Finnney.Immutable): pass
Finney.Immutable = Addable

Which means that from now on *your* code that tries to create
Immutables will actually get Addables. The inability to do this in B&D
languages is - well, painfull. That Python doesns't require the
boilerplate in a good thing.

<mike

 

[Mike]

That is an ideal case and we all know the real world is not ideal or
every program would be bug free. And I don't think anyone would solely
relies on the language feature for spotting bugs. Whether this kind of
guard is useful is another story.

This is a case where the *language* should be expressive as possible, and
the *tools* surrounding it should be of more help.
Have I ever mistyped a python variable and introduced a bug? -- many times,
and it's really annoying. It's a fault that there's no tool that realizes
that "positon" looks a hell of a lot like "position", more than it's the
compiler's fault to make such an assumption, because the compiler can't
"close the loop" with the user, but the tool can. (Or my fault for not using
such a tool - I know there's pylint, etc.)

 

[Alex Martelli]

So, in the hypothetical situation that all Python types were
implemented in pure Python, what would the ideal behaviour for
immutables be? Or would there be no immutables?

Pypy is implementing Python in Python and using just the same semantics
as CPython. Of course, there IS some sleight of hand, because there are
still two "sides" -- interpreter level and application level... stuff
implemented in interpreter level Python plays just the same role as
stuff implemented in C does in CPython.

Musing theoretically, I can see an *ideal* approach would have to meet
higher aesthetic goals than Python's -- reach for richer symmetry, e.g.,
all types existing in mutable and immutable flavors (Ruby comes sort of
close, with plain and frozen objects). Python's more about pragmaticity
than purity (and, therefore, than idealness)...

I'm looking for a "consenting adults" restriction: classes will have
immutable instances only where it makes sense from the class protocol.
I'm not going to lose sleep over users who go looking for trouble.

Yep, I understand this, now that you've explained in more detail, and
FWIW I do approve.

One more piece of advice: you might document that, in circumstances
where (using a normal Python class's instances) the obvious approach
would be to add some per-instance attribute to the instances (with
semantics totally outside the class), your users might be better advised
to employ a weakref.WeakKeyDictionary with the instances as keys.
Although a tad more cumbersome, they can get the same semantics this
way, even though it may seem "inside-out" wrt the "intuitive" approach.


Alex

 

[Giovanni Bajo]

I certainly hope you're not relying on it to catch bugs. You should do
proper testing instead. Not only will that catch pretty much all the
bugs you mention later - thus resolving you of the need to handcuff
clients of your class - it will catch lots of other bugs as well.

This sounds a little academic. Writing a real Python program without testing is
impossible or close to it, so you are really not telling me anything new. Even
*with* testing, there are bugs. I'm sure you well know this.

My feeling is that you're trying to get too much out of my words. I'm not
trying to handcuff anyone. You seem to concentrate on me trying to avoid people
adding attributes to my precious objects. It's not that. If I write a class and
want it to be immutable, it is because it has to be so. If I write a queue
class and I say that people shouldn't call pop() if it's empty, I mean it. If I
enforce it with a RuntimeError, I'm not thinking I'm handcuffing someone. I
don't see a ImmutableError to be so different from it.

I often design objects that I want to be immutable. I might keep it as a key in
dictionaries, or I might have external, non-intrusive caches of some kind
relying on the fact that the instance does not change. Of course, it *might* be
that testing uncovers the problem. Unittests tend to be pretty specific, so in
my experience they happen to miss *new* bugs created or uncovered by the
integration of components. Or if they hit it, you still have to go through
debug sessions. An ImmutableError would spot the error early.

In my view, enforcing immutability is no different from other forms of
self-checks. Do you reckon all kind of asserts are useless then? Surely they
don't help for anything that a good unittest couldn't uncover. But they help
catching bugs such as breakage of invariants immediately as they happen.
Immutability can be such an invariant.

Well, if you want to propose a change to the language, you need a good
use case to demonstrate the benefits of such a change. Do you have
such a use case? Catching bugs doesn't qualify, otherwise Python would
be radically different from what it is.

One good reason, in my opinion, is that there *are* immutable objects in
Python, among builtins. And people can easily build extension objects which are
immutable. So being impossible to write a regular object in Python which is
immutable is not orthogonal to me.

Now let me ask you a question. What is a good use case for "assert" that
justifies its introduction in the language? What is a good usecase for module
'unittest' which justifies its introduction in the standard library? Why do you
think tuples are immutable and *enforced* to be so?

I'm not convinced that immutability is that important a concept. Yeah,
you have to know about it, but it seems more like an implementation
detail than a crucial concept.

Probably it's just a matter of design styles.

I'm not sure it's more important than
things like interned strings and the sharing of small integers. Most
of the discussion of immutables here seems to be caused by newcomers
wanting to copy an idiom from another language which doesn't have
immutable variables. Their real problem is usually with binding, not
immutability.

I'm not such a newcomer, but (how funny) Python is *the* language that
introduced me to the concept of immutable objects and their importance in
design

 

[Guest]

My feeling is that you're trying to get too much out of my words. I'm
not trying to handcuff anyone. You seem to concentrate on me trying to
avoid people adding attributes to my precious objects. It's not
that. If I write a class and want it to be immutable, it is because it
has to be so. If I write a queue class and I say that people shouldn't
call pop() if it's empty, I mean it. If I enforce it with a
RuntimeError, I'm not thinking I'm handcuffing someone. I don't see a
ImmutableError to be so different from it.

But why would you call it that, when the object isn't actually
implemented as immutable?

It's pretty misleading to call an object immutable rather than saying
that it shouldn't be changed for some reason.

Throw an exception that describes why it doesn't make sense to change
that particular object instead.

As I said before, I think you're confusing the (in Python pretty
non-existent) concept of encapsulation with Python's immutable types,
which are immutable because the implementation demands it. (A fact I
hope will disappear at some point.)

 

[Paul Rubin]

As I said before, I think you're confusing the (in Python pretty
non-existent) concept of encapsulation with Python's immutable types,
which are immutable because the implementation demands it. (A fact I
hope will disappear at some point.)

What implementation demand? If Python's designers wanted mutable
strings or tuples, Python could have them. Python strings and tuples
are immutable by design, not by accident of implementation. Wanting
to make immutable class instances out of the same design
considerations is perfectly reasonable.

 

[Giovanni Bajo]

But why would you call it that, when the object isn't actually
implemented as immutable?

I think you misunderstood my message. I'm saying that an ImmutableError
wouldn't be a much different form of self-checking compared to the usual
TypeError/ValueError/RuntimeError you get when you pass wrong values/types to
methods/functions, or you violate invariants of objects. This has nothing to do
with dynamism, duck typing, dynamic binding and whatnot. Some objects have deep
invariants which can't be broken.

Why do you think we have a frozenset, for instance? By Mike's argument, we
shouldn't have it. And we should be able to use a regular mutable set as
dictionary key. Unittests will catch errors. Instead, we got two classes
instead of one, immutability is *enforced*, and sets can't be used as
dictionary keys. This is all good in my opinion, and follows the good rule of
"catch errors early".

Throw an exception that describes why it doesn't make sense to change
that particular object instead.

*How* can I do? There is no language construct which lets me specifies an
exception to throw whenever someone modifies my object. *Even* if I got partial
support for immutable types (like __new__ which can be used to initialize
immutable objects).

As I said before, I think you're confusing the (in Python pretty
non-existent) concept of encapsulation with Python's immutable types,
which are immutable because the implementation demands it. (A fact I
hope will disappear at some point.)

You seriously believe that strings, integers and tuples are immutable because
of implementation details? I believe they are part of a language design -- and
a good part of it.

 

[Mike Meyer]

My feeling is that you're trying to get too much out of my words. I'm not
trying to handcuff anyone. You seem to concentrate on me trying to avoid people
adding attributes to my precious objects.

Because *that's* the use case that you're preventing that I have
problems with.

It's not that. If I write a class and want it to be immutable, it is
because it has to be so. If I write a queue class and I say that
people shouldn't call pop() if it's empty, I mean it. If I enforce
it with a RuntimeError, I'm not thinking I'm handcuffing someone. I
don't see a ImmutableError to be so different from it.

And I have no problems with that. If you believe your class should
throw an error if someone calls an instances pop() method when it's
empty, do so.

Likewise, if you want to make it so a client can't change your
attributes, feel free to do so.

However, when you prevent a client from adding an attribute, you're
not merely making your objects immutable, you're making them
static. Python isn't a static language, it's a dynamic language. I
consider parts of it that are static to be warts.

In my view, enforcing immutability is no different from other forms of
self-checks. Do you reckon all kind of asserts are useless then? Surely they
don't help for anything that a good unittest couldn't uncover. But they help
catching bugs such as breakage of invariants immediately as they happen.
Immutability can be such an invariant.

Adding an attribute can't break an invariant. The only way an
attribute can affect an invariant is if the invariant references
it. If it references it it, then it must exist - so I can't add it.

One good reason, in my opinion, is that there *are* immutable objects in
Python, among builtins. And people can easily build extension objects which are
immutable. So being impossible to write a regular object in Python which is
immutable is not orthogonal to me.

I didn't ask for a reason, I asked for a use case. Orthogonality is a
good thing, but so is generality. So I'll argue that the correct way
to make this situation orthogonal is to make builtin objects more
general by making it possible to add attributes to all of them.

On the other hand, practicality beats purity, and doing the above has
a significant cost in the current implementation, so leave it like it
is.

Now let me ask you a question. What is a good use case for "assert" that
justifies its introduction in the language? What is a good usecase for module
'unittest' which justifies its introduction in the standard library?

Unittesting doesn't change the language, and is part of any good
development methodology.

Assert is a shorthand for "if ... ; raise ...." that vanishes when you
turn on optimization. It's cleaner and more clearly expresses the
intent of the programmer, and saves a fair amount of boilerplate in
the best case.

Why do you think tuples are immutable and *enforced* to be so?

That you can't add attributes to a tuple is a detail of the
implementation; it's true for non-immutable types as well.

I'm not such a newcomer, but (how funny) Python is *the* language that
introduced me to the concept of immutable objects and their importance in
design

Well, that would explain why you think it's so important - it's where
you first encountered it. I'd argue that it's no more important than
identity - which is what I was searching for when I talked about
interned strings sharing small integers. There are builtin types that
preserve identity for equal instances, at least under some
conditions. There are no constructs for helping you do that with
user-defined objects. Should we add them for the sake of
orthogonality? I don't think so - not without a good use case.

<mike

 

[Giovanni Bajo]

Why is it "surely" a bug? It is arguable whether adding new
attributes (vs. changing those that are already there) is, in fact,
mutation.

If we agree that *changing* attributes is a bug for those classes, we're
already a step beyond in this discussion

About adding attributes, I agree that it's kind of a grey area. Per-se, there
is nothing wrong. My experience is that they give the user a false expectation
that the object can be modified. It ends up with an object with attributes
which can't be modified because that'd break invariants, and others which are
freely modificable.

In fact, the only thing that you are adding an attribute to *that* instance,
and not to all the *equivalent* instances (by definition of __hash__/__eq__) is
questionable and bug-prone. You might end up *thinking* you have that very
instance around, while you don't. In fact, with immutable objects, you are not
supposed to think in terms of instances, but rather of values. When I see a
string like "foobar" I don't care if it's the same instance of a "foobar" I saw
before. If I could add an attribute to "foobar", I might end up believing that
whenever I see a "foobar" around, it will have that attribute.

As you said, Python has rich data structures which lets you still find good
ways to carry around additional information. So why trying so hard to get into
trouble?

It sounds like what you may want are opaque objects where you can
control access better

No that's a different issue. One example of my immutable classes is Point2D
(two attributes: x/y). Having it immutable gives it many useful properties,
such as a real value semantic.

 

[Mike Meyer]

Björn Lindström wrote:
Why do you think we have a frozenset, for instance? By Mike's argument, we
shouldn't have it.

Not *my* arguments, certainly. Not unless you're seriously
misinterpreting them.

<mike

 

[Alex Martelli]

You seriously believe that strings, integers and tuples are immutable because
of implementation details? I believe they are part of a language design -- and
a good part of it.

Definitely. When I first designed gmpy, I wanted to make its number
types mutable, thinking this might probably enhance performance, but I
double checked with Python cognoscenti first -- and the result was a
plebiscite for IMmutable numbers (at the time, I was sort of new at
Python, and didn't really get it, but now I do;-).


Alex

 

[Giovanni Bajo]

And I have no problems with that. If you believe your class should
throw an error if someone calls an instances pop() method when it's
empty, do so.

Likewise, if you want to make it so a client can't change your
attributes, feel free to do so.

However, when you prevent a client from adding an attribute, you're
not merely making your objects immutable, you're making them
static. Python isn't a static language, it's a dynamic language. I
consider parts of it that are static to be warts.

I always thought that adding an attribute was just as bad as changing the
attributes, for an immutable object. But I now see your point (eventually!),
they are pretty different issues.

But, whatever attribute you add to the instance, it should *also* be immutable
(so that, in other words, wouldn't be so different than carrying around a tuple
with the original instance and the added attribute). This said, I think I
devise that a language support for enforcing immutability could allow adding
attributes to instance -- as long as those attributes then become immutable as
well.

Well, that would explain why you think it's so important - it's where
you first encountered it.

Yes. But I'm familiar with different object semantics, and I found the
immutable objects to be a pretty good replacement of value semantics, and to
implement a kind-of pass-by-value convention in a language which only has
references.

I'd argue that it's no more important than
identity - which is what I was searching for when I talked about
interned strings sharing small integers. There are builtin types that
preserve identity for equal instances, at least under some
conditions. There are no constructs for helping you do that with
user-defined objects. Should we add them for the sake of
orthogonality? I don't think so - not without a good use case.

I don't think identity is important for immutable objects (as I wrote
elsewhere), so I don't think adding language constucts for this would prove
useful. Instead, immutable objects *are* common, and we still miss a way to
mark them as such.