LangWart: Method congestion from mutate multiplicty

[Rick Johnson]

DISCLAIMER:
This post covers a universal programming language design flaw using both Python and Ruby code examples to showcase the issue.

I really don't like to read docs when learning a language, especially a "so-called" high level language. I prefer to learn the language by interactivesessions and object introspection. Then, when i have exhausted all abilities to intuit the solution, i will roll my eyes, maybe blubber an expletive,and then reluctantly crack open a user manual.

However, learning a new language (be it by this method or by official docs)is frustrating when languages have method congestion from a need to present their users with both a method for "in-place-mutation" and method for "mutation-of-a-copy"; both sharing an almost exact spelling!

Yes i know, naming conventions can help. And one widely used convention is to use weak verbs for the "mutation of a copy" and strong verbs for "in-place mutation", consider:

py> a.reverse -> mutate 'a'
py> a.reversed -> new Array

However you will sooner or later encounter a word that does not have a proper "weak verb" variant to describe the copy-mutate action, consider:

rb> point3d.offset(vector3d) -> mutate 'point3d'
rb> point3d.offseted(vector3d) -> HUH?

The Ruby language attempted to save the programmer from the scourge of obtaining a four year degree in linguistics just to create intuitive identifiers "on-the-fly", and they tried to remove this ambiguity by employing "post-fix-punctuation" of the exclamation mark as a visual cue for in-place modification of the object:

rb> a = [1,2,3]
rb> a.reverse!()
[3,2,1]
rb> a
[3,2,1]

....think of the exclamation mark yelling out; "Hey, i will modify this object so be careful dude!" On the other hand, a method that mutates a copy will have the same identifier except /without/ the exclamation mark:

rb> a = [1,2,3]
rb> a.reverse()
[3,2,1]
rb> a
[1,2,3]

Now whilst this punctuation solves the ambiguity issue in a reasonable manner, it does not solve the congestion issue because for /every/ method that returns a copy of the object, another method will exist with an exclamationmark post-fixed that signifies object mutation. I don't like this because when i inspect the object i see redundant method names:

rb> mutators = a.methods.grep(/.*!/)
rb> copyers = a.methods.select{|x| mutators.include?(x+"!")}
rb> copyers+mutators.sort
rb> ["flatten", "transform", "collect", "sort", "map", "uniq", "offset", "reverse", "compact", "reject", "normalize", "slice", "collect!", "compact!","flatten!", "map!", "normalize!", "offset!", "reject!", "reverse!", "slice!", "sort!", "transform!", "uniq!"]

Now that's just a small subset of the member functions of the Array object!Can you imagine the mental overload induced when the entire set of methodsmust be rummaged through each and every time!!!

rb> a.methods.length
141

*look-of-disapproval*

============================================================
SOLUTION
============================================================

The solution is simple. Do not offer the "copy-mutate" methods and force all mutation to happen in-place:

py> l = [1,2,3]
py> l.reverse
py> l
[3,2,1]

If the user wants a "mutated copy" he should explicitly create a new objectand then apply the correct mutator method:

py> a1 = [1,2,3]
py> a2 = list(a1).reverse()
py> a1
[1,2,3]
py> a2
[3,2,1]

 

[Steven D'Aprano]

The solution is simple. Do not offer the "copy-mutate" methods and force
all mutation to happen in-place:

py> l = [1,2,3]
py> l.reverse
py> l
[3,2,1]

If the user wants a "mutated copy" he should explicitly create a new
object and then apply the correct mutator method:

py> a1 = [1,2,3]
py> a2 = list(a1).reverse()

Oh wow, Rick has re-discovered programming in Python during the mid to late
1990s!

I was there, and I remember what it was like. For about a month, you try
hard to follow Rick's prescription. Then you realise that with a small
helper function, you can halve the amount of code it takes to do a common
operation:

def reversed(sequence):
seq = list(sequence)
seq.reverse()
return seq


Soon you've copied this reversed() function into all your projects. And of
course, they start to diverge... in project A, you only care about lists.
In project B, you realise that you also need to support tuples and strings:


def reversed(sequence):
seq = sequence[:]
try:
seq.reverse()
except AttributeError:
seq = seq[::-1]
return seq

which in project C you realise can be shortened:

def reversed(sequence):
return sequence[::-1]


until you get to project D when you realise that you also want this to work
on dicts:

def reversed(sequence):
everything = list(sequence)
return everything[::-1]


and then in project E you wonder why reversed(string) returns a list:

def reversed(sequence):
everything = list(sequence)[::-1]
if isinstance(sequence, tuple):
return tuple(everything)
elif isinstance(sequence, str):
return ''.join(everything)
return everything


and then finally you learn about iterators and generators and become more
comfortable with a flow-based programming paradigm and generators:

def reversed(sequence):
for item in list(sequence)[::-1]:
yield item

at which point you realise that, hell, this is so useful that pretty much
everyone has implemented it a dozen times or more in their own projects,
and you start to agitate for it to be added to the builtins so that there
is *one* implementation, done *right*, that everyone can use.

And then you get told that Guido's time machine has struck again, because
Python has already had this since Python 2.4.

 

[Chris Angelico]

I really don't like to read docs when learning a language, especially a "so-called" high level language. I prefer to learn the language by interactive sessions and object introspection. Then, when i have exhausted all abilities to intuit the solution, i will roll my eyes, maybe blubber an expletive, and then reluctantly crack open a user manual.

What Rick means: "I want to claim that I've learned a new language,
but I want it to work exactly like the imaginary language in my mind,
and if it doesn't, I'm going to complain about it, rather than,
yaknow, actually learn a new language."

I have learned *many* languages in the past couple of decades. Some of
them are excellent and I keep using them (Pike). Others are excellent
and I keep talking about them (Python). Some are mediocre or poor, but
I keep using them anyway (bash). Some are not particularly enjoyable
to me and I use them only in the one application that embeds them
(Lua, Scheme, DML). And some, I'm just not going to touch any more
(Q-BASIC). But there is not a single language that hasn't taught me
something new. I'm a better C++ programmer for having learned Python;
a better Python programmer for having grokked Scheme and Lua; and,
believe it or not, a better Scheme programmer for having mastered DML.
And that's a language so obscure it doesn't even have a Wikipedia
page... just a redlink here[1].

Learning a language requires accepting something from it into your
brain, not forcing something from your brain onto the language.

ChrisA

[1] http://en.wikipedia.org/wiki/List_of_programming_languages_by_type#Extension_languages

 

[Rick Johnson]

The solution is simple. Do not offer the "copy-mutate" methods and force
all mutation to happen in-place:

py> l = [1,2,3]
py> l.reverse
py> l
[3,2,1]

If the user wants a "mutated copy" he should explicitly create a new
object and then apply the correct mutator method:

py> a1 = [1,2,3]
py> a2 = list(a1).reverse()

Oh wow, Rick has re-discovered programming in Python during the mid to late
1990s!

[...snip: long-winded, rambling, and sarcastic response simply to convey
that Python lists have had a "reversed" method for some time...]

Steven, i am quite aware of the Python list method "reversed" --which returns a copy of the current list object in reversed order--, my point is that these types of "copy-mutate" methods superfluously pollute the object namespace. Do you really want "method pairs" like these:

sort, sorted
reverse, reversed

Hell, why stop there:

append, appended
flatten, flattened
insert, inserted
map, mapped
filter, filtered
reduce, reduced
extend, extended
freeze, frozen
set, sat|setted
unique, uniqued

Is this really what you prefer? Where does the madness end Steven? At what point do you say enough is enough? And what happens if you fail to catch the infection early enough? Steven, this is a /real/ problem which has the potential to go viral!

My point was this: All mutate methods should mutate "in-place", if the programmer wishes to create a mutated copy of the object, then the programmer should /explicitly/ create a copy of the object and then apply the correct mutator method to the copy.

NO: reversed = lst.reversed() # Python
YES: reversed = list(lst).reverse() # Python

NO: reversed = a.reverse() # Ruby
YES: reversed = Array.new(a).reverse!() # Ruby

This is about consistency and keeping the number of methods from spiraling out of control because we feel the need to automate /every/ task for the programmer, when in actuality, we are doing more harm than good.

 

[Chris Angelico]

My point was this: All mutate methods should mutate "in-place", if the programmer wishes to create a mutated copy of the object, then the programmer should /explicitly/ create a copy of the object and then apply the correct mutator method to the copy.

I agree. And we can go further and declare that there is only one data
type, the simple integer; you have an infinite number of them, and all
you can do is mutate them in place. You don't need variable names
either; just have one single array that represents your whole
namespace, and work with positions in that array. And don't bother
with actual positions, even - with a single pointer, you could manage
everything.

Forget this silly mess of data types, methods, global functions, and
so on. Let's simplify things massively!

Ook. Ook!

ChrisA

 

[Mark Janssen]

My point was this: All mutate methods should mutate "in-place", if the programmer wishes to create a mutated copy of the object, then the programmer should /explicitly/ create a copy of the object and then apply the correct mutator method to the copy.

I agree. And we can go further and declare that there is only one data
[sarcasm]

I have to agree with Rick, I think requiring the user to explicitly
create a new object, which is already a good and widely-used practice,
should be the Only One Way to Do It. Guessing method names is far
suboptimal to this simple, easy idiom. As for the point Chris was
making as to making all types one, I actually agree there too, it's
just that in order to do that, python would need a unified object
model and it doesn't have one yet.

Mark

 

[Terry Reedy]

While it is true that sorted(iterable) is essentially

def sorted(iterable):
tem = list(iterable)
tem.sort
return tem

the body is not an expression and cannot be substituted in an
expression. The need for the short form was thought common enough to be
worth, *on balance*, a new builtin name. It is not surprising that not
all agree.

Reversed(iterable) is more complicated because it returns an iterator,
not a list, and looks for a class-specific __reversed__ method. I think
it is more or less equivalent to the following:

def _rev_iter(seq, n):
for i in range(n-1, -1, -1):
# many people have trouble getting the range right
yield seq

def reversed(iterable):
try:
return iterable.__reversed__()
except AttributeError:
try:
itlen = iterable.__len__
iterable.__getitem__
return _rev_iter(iterable, itlen)
except AttributeError:
raise TypeError("argument to reversed() must be a sequence")

Even if list mutation methods returned the list, which they do not and
for good reason, reversed(it) is not the same as list(it).reverse(). So
that part of the premise of this thread is wrong.

 

[Steven D'Aprano]

The solution is simple. Do not offer the "copy-mutate" methods and
force all mutation to happen in-place:

py> l = [1,2,3]
py> l.reverse
py> l
[3,2,1]

If the user wants a "mutated copy" he should explicitly create a new
object and then apply the correct mutator method:

py> a1 = [1,2,3]
py> a2 = list(a1).reverse()

Oh wow, Rick has re-discovered programming in Python during the mid to
late 1990s!

[...snip: long-winded, rambling, and sarcastic response simply to convey
that Python lists have had a "reversed" method for some time...]

Steven, i am quite aware of the Python list method "reversed" --which
returns a copy of the current list object in reversed order--,

And you have missed my point, which is that reversed(), and sorted(), were
not added to the language on a whim, but because they were requested, over
and over and over again. People who actually programmed using Python before
reversed() and sorted() were added missed them, and consequently kept
reimplementing them.

You want to go back to the Bad Old Days when everyone was reimplementing the
same few functions over and over again. I say, boo sucks to that.

my point is
that these types of "copy-mutate" methods superfluously pollute the object
namespace. Do you really want "method pairs" like these:

sort, sorted
reverse, reversed
Yes.


Hell, why stop there:

append, appended

"appended" is called list addition.

newlist = oldlist + [item_to_append]

flatten, flattened

flatten is another often requested, hard to implement correctly, function.
The only reason that Python doesn't have a flatten is that nobody can agree
on precisely what it should do.

Like map, filter, reduce, etc. flatten is not sensibly implemented as a
mutator method, but as a function.

insert, inserted

"inserted" is called addition, together with list slicing when needed.

newlist = [item_to_insert] + oldlist
newlist = oldlist[0:5] + [item_to_insert] + oldlist[5:]

map, mapped
filter, filtered
reduce, reduced

Those are nonsense. None of those are in-place mutator methods. Especially
reduce, which reduces a list to a single item. You might as well have
suggested "len, "lened".

extend, extended

Again, "extended" is spelled list addition.

Are you sure you've actually programmed in Python before? You seem awfully
ignorant of language features.


[...]

My point was this: All mutate methods should mutate "in-place",

Well duh. All mutator methods do mutate in-place, otherwise they wouldn't be
mutator methods.

if the
programmer wishes to create a mutated copy of the object, then the
programmer should /explicitly/ create a copy of the object and then apply
the correct mutator method to the copy.

Been there, done that, it sucks. That's about a dozen steps backwards to a
worse time in Python development.

 

[Steven D'Aprano]

Rick Johnson:


Ruby does not use '!' to indicate in-place modification:
http://dablog.rubypal.com/2007/8/15/bang-methods-or-danger-will-rubyist

Why am I not surprised that Rick's knowledge of Ruby is no deeper than his
knowledge of Python?

 

[Chris Angelico]

"inserted" is called addition, together with list slicing when needed.

newlist = [item_to_insert] + oldlist
newlist = oldlist[0:5] + [item_to_insert] + oldlist[5:]

Really? Wouldn't it be easier to use slice assignment on a copy?

newlist = oldlist[:]; newlist[posos] = [item_to_insert]

Actually, come to think of it, that scores about the same on
readability. Six of one, half dozen of the other.

ChrisA

 

[Oscar Benjamin]

My point was this: All mutate methods should mutate "in-place", if the programmer wishes to create a mutated copy of the object, then the programmer should /explicitly/ create a copy of the object and then apply the correct mutator method to the copy.

I agree. And we can go further and declare that there is only one data
[sarcasm]

I have to agree with Rick, I think requiring the user to explicitly
create a new object, which is already a good and widely-used practice,
should be the Only One Way to Do It.

Why should I copy a potentially large data structure just to iterate
over it in reverse order? And why on earth would you want to remove
the more efficient ways of doing this?

Guessing method names is far suboptimal to this simple, easy idiom.

There is no guessing. If the object has a __reverse__ method then it
specifically advertises that it knows how to create an iterator that
gives its values in reverse order. Otherwise __len__ and __getitem__
are used.


Oscar

 

[Tim Chase]

Those are nonsense. None of those are in-place mutator methods.
Especially reduce, which reduces a list to a single item. You might
as well have suggested "len, "lened".

And, if you want those in-place, indexing trivially comes to the
rescue again:

lst[:] = map(transform_fn, lst)
lst[:] = filter(check_fn, lst)

or, as I prefer:

lst[:] = [transform_fn(x) for x in lst]
lst[:] = [x for x in lst if check_fn(x)]

as they can be combined simply.

-tkc

 

[Steven D'Aprano]

"inserted" is called addition, together with list slicing when needed.

newlist = [item_to_insert] + oldlist
newlist = oldlist[0:5] + [item_to_insert] + oldlist[5:]

Really? Wouldn't it be easier to use slice assignment on a copy?

newlist = oldlist[:]; newlist[posos] = [item_to_insert]

I don't know about "easier", but it's two statements rather than a single
expression, which means you cannot easily include it as part of a larger
expression.

Actually, come to think of it, that scores about the same on
readability. Six of one, half dozen of the other.

Pretty much.

 

[Steven D'Aprano]

My point was this: All mutate methods should mutate "in-place", if the
programmer wishes to create a mutated copy of the object, then the
programmer should /explicitly/ create a copy of the object and then
apply the correct mutator method to the copy.

I agree. And we can go further and declare that there is only one data
[sarcasm]

I have to agree with Rick, I think requiring the user to explicitly
create a new object, which is already a good and widely-used practice,

Perhaps so, but consider how you creates new objects in Python. Very rarely
do you do so with an explicit call to the constructor. For example:

n = 5 # Yes.
# or
n = int("5") # No.

alist = some_list[1:] # Yes.
# or
alist = list()
alist.extend(some_list[1:]) # No.

items = sorted(other_things + [1]) # Yes.
# or
items = other_things[:]
items.append(1)
items.sort() # Hell no.


There are many functions or methods that create new objects, apart from the
constructor. A call like:

blist = sorted(alist)

is no less explicitly creating a new list than:

blist = list(alist)

should be the Only One Way to Do It.

Pardon me, but you've been listening to too many Perl developers. "Only One
Way To Do It" is not, and never has been, the motto of Python. You may be
thinking of the line from the Zen of Python:

py> import this
[...]
There should be one-- and preferably only one --obvious way to do it.


The emphasis is on the *obvious*, not the "only". There is an enormous
difference between prohibiting a second way to solve problems ("Only One
Way") and recommending that there should be an Obvious Way.

Guessing method names is far
suboptimal to this simple, easy idiom. As for the point Chris was
making as to making all types one, I actually agree there too,

Oh dear. Chris was being sarcastic. I thought that, even if the sarcasm
wasn't obvious, his "Ook. Ook!" at the end should have given it away:

http://www.dangermouse.net/esoteric/ook.html

it's
just that in order to do that, python would need a unified object
model and it doesn't have one yet.

I'm not sure what you mean by "unified object model", but I'm pretty sure
that Python has one. Everything is an object, with a single[1] hierarchy of
classes.



[1] Python 3 only. In Python 2, you have types, and you have old-style
classes, and they are separate.

 

[88888 Dihedral]

Steven D'Apranoæ–¼ 2013å¹´2月9日星期六UTC+8上åˆ11時36分52秒寫é“：

Rick Johnson wrote:

The solution is simple. Do not offer the "copy-mutate" methods and force

all mutation to happen in-place:
py> l = [1,2,3]

py> l.reverse

py> l

[3,2,1]

If the user wants a "mutated copy" he should explicitly create a new

object and then apply the correct mutator method:
py> a1 = [1,2,3]

py> a2 = list(a1).reverse()

Oh wow, Rick has re-discovered programming in Python during the mid to late

1990s!



I was there, and I remember what it was like. For about a month, you try

hard to follow Rick's prescription. Then you realise that with a small

helper function, you can halve the amount of code it takes to do a common

operation:



def reversed(sequence):

seq = list(sequence)

seq.reverse()

return seq





Soon you've copied this reversed() function into all your projects. And of

course, they start to diverge... in project A, you only care about lists.

In project B, you realise that you also need to support tuples and strings:





def reversed(sequence):

seq = sequence[:]

try:

seq.reverse()

except AttributeError:

seq = seq[::-1]

return seq

Will a temprary new list be formed here?
If it is not necessary, I'll prefer a reverse
generator for all lists to save the heap space
and the GC burden.

which in project C you realise can be shortened:



def reversed(sequence):

return sequence[::-1]





until you get to project D when you realise that you also want this to work

on dicts:



def reversed(sequence):

everything = list(sequence)

return everything[::-1]





and then in project E you wonder why reversed(string) returns a list:



def reversed(sequence):

everything = list(sequence)[::-1]

if isinstance(sequence, tuple):

return tuple(everything)

elif isinstance(sequence, str):

return ''.join(everything)

return everything





and then finally you learn about iterators and generators and become more

comfortable with a flow-based programming paradigm and generators:



def reversed(sequence):

for item in list(sequence)[::-1]:

yield item



at which point you realise that, hell, this is so useful that pretty much

everyone has implemented it a dozen times or more in their own projects,

and you start to agitate for it to be added to the builtins so that there

is *one* implementation, done *right*, that everyone can use.



And then you get told that Guido's time machine has struck again, because

Python has already had this since Python 2.4.

 

[Mark Janssen]

Why should I copy a potentially large data structure just to iterate
over it in reverse order? And why on earth would you want to remove
the more efficient ways of doing this?

You're right. I responded too fast. I think reversed() and sorted()
might be the only legit methods in this regard and I thank Steve
D'Aprano for pointing that out.

But Rick still has a valid point: it should not be taken as a general
practice. The point, as I see it, is that there's no clear,
documented standard on the "right way" for people to think about the
issue. The existence of sorted() and reversed() actually *misinform*
programmers as if this is the best practice. It isn't, it just that
these are very special cases (one for a real machine efficiency and
one for a very common "user efficiency") and there should probably be
documentation to make that clear, so programmers don't start going
that direction. I don't think there are other cases where such an
idiom would be recommended.

Mark

 

[Rick Johnson]

While it is true that sorted(iterable) is essentially

def sorted(iterable):
tem = list(iterable)
tem.sort
return tem

the body is not an expression and cannot be substituted in an
expression.

Yes but the body can be compressed to this single line: "list(iterable).sort()"

Reversed(iterable) is more complicated because it returns an iterator,
not a list, and looks for a class-specific __reversed__ method.
[...]

Well if you're taking the position that iterators are difficult to create i say you are exaggerating a bit. Using the for loop:

py> for LOCALVAR in SEQUENCE:
.... do_something

we can get the iterator for free. If however you want to control the iteration /without/ being locked into a loop, you can explicitly call:

py> iter(seq)

Or, if you prefer methods over global functions:

py> seq.__iter__()

Or, if python employed /true/ OOP paradigm:

py> Iterator(seq)

Even if list mutation methods returned the list, which they do not and
for good reason,

I am not proposing that in-place modification return the object.

reversed(it) is not the same as list(it).reverse(). So
that part of the premise of this thread is wrong.

Well, it's not the same /now/, because of how Python handles this operation. The status quo is to encourage the implicit idiom over the explicit, however, this behavior could be optimized to cleanly handle /explicit/ syntax only.

 

[Rick Johnson]

While it is true that sorted(iterable) is essentially

def sorted(iterable):
tem = list(iterable)
tem.sort
return tem

the body is not an expression and cannot be substituted in an
expression.

Yes but the body can be compressed to this single line: "list(iterable).sort()"

Reversed(iterable) is more complicated because it returns an iterator,
not a list, and looks for a class-specific __reversed__ method.
[...]

Well if you're taking the position that iterators are difficult to create i say you are exaggerating a bit. Using the for loop:

py> for LOCALVAR in SEQUENCE:
.... do_something

we can get the iterator for free. If however you want to control the iteration /without/ being locked into a loop, you can explicitly call:

py> iter(seq)

Or, if you prefer methods over global functions:

py> seq.__iter__()

Or, if python employed /true/ OOP paradigm:

py> Iterator(seq)

Even if list mutation methods returned the list, which they do not and
for good reason,

I am not proposing that in-place modification return the object.

reversed(it) is not the same as list(it).reverse(). So
that part of the premise of this thread is wrong.

Well, it's not the same /now/, because of how Python handles this operation. The status quo is to encourage the implicit idiom over the explicit, however, this behavior could be optimized to cleanly handle /explicit/ syntax only.

 

[Rick Johnson]

Ruby does not use '!' to indicate in-place modification:

Really?

rb> a = [1,2,3]
[1, 2, 3]
rb> a.reverse
[3, 2, 1]
rb> a
[1, 2, 3]
rb> a.reverse!
[3, 2, 1]
rb> a
[3, 2, 1]

And now we will verify that a.reverse! has not assigned 'a' to a new object

rb> a = [1,2,3]
[1, 2, 3]
rb> aID = a.object_id
78906770
rb> a.reverse!
[3, 2, 1]
rb> a
[3, 2, 1]
rb> a.object_id
78906770

I'd love to hear an explanation for that.

 

[Dennis Lee Bieber]

Oh dear. Chris was being sarcastic. I thought that, even if the sarcasm
wasn't obvious, his "Ook. Ook!" at the end should have given it away:

http://www.dangermouse.net/esoteric/ook.html

Ah... and here I thought it might have been a subtle reference to
the Librarian of the Unseen University...
http://en.wikipedia.org/wiki/Unseen_University#Librarian