Question about consistency in python language

[lechequier]

Let's say I define a list of pairs as follows:

l = [('d', 3), ('a', 2), ('b', 1)]

Can anyone explain why this does not work?
and instead I have to go:to initialize a dictionary with the given list of pairs?

when an analagous operation on strings works fine:

s = "".join(["d","o","g"])

Seems inconsistent.

thanks,
Scott

 

[Sam Pointon]

update updates the dictionary in place - it actually returns None, not
the updated dict. However, you can construct a dictionary from a list
of (key, value) pairs using dict(list). Example:

l = [('foo', 'bar'), ('baz', 'qig')]
d = dict(l)
print d

{'foo': 'bar', 'baz': 'qig'}

 

[James Stroud]

This is the difference between mutable and immutable types. In this sense it
is consistent.

If you want to do the former in one shot:

h = dict(l)

Also, you shouldn't use "1", I mean "l", as a variable name. It gets confusing
because "l", I mean "1", looks a lot like "1", I mean "l".

James

Let's say I define a list of pairs as follows:

l = [('d', 3), ('a', 2), ('b', 1)]

Can anyone explain why this does not work?

and instead I have to go:
to initialize a dictionary with the given list of pairs?

when an analagous operation on strings works fine:

s = "".join(["d","o","g"])

Seems inconsistent.

thanks,
Scott

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/

 

[Dave Benjamin]

Let's say I define a list of pairs as follows:

l = [('d', 3), ('a', 2), ('b', 1)]

Can anyone explain why this does not work?


and instead I have to go:

to initialize a dictionary with the given list of pairs?

when an analagous operation on strings works fine:

s = "".join(["d","o","g"])

Seems inconsistent.

Python is actually quite consistent in this regard: methods that modify
an object in-place return None; methods that do not modify an object
in-place return a new object instead. Since strings are immutable, they
cannot be modified in-place, so it makes sense for the "join" method to
return a new string. On the other hand, Python's dictionaries are
imperative-style and so most operations on a dictionary modify an
existing dictionary.

I was initially bothered by the phenomenon of so many methods returning
None because they could not be chained. But I have come to deeply
appreciate this style for a couple of reasons. First, it makes it clear
which methods are side-effecting (like "update") and which are not (like
"sort").

Second, it is not always clear what a good return value is for a
mutator. Perhaps {}.update() should return the dictionary, making
chaining convenient. Perhaps it should return the total number of items
after updating. Or maybe it should return the number of new keys that
were added, or a list of those keys. All of these are plausible
behaviors; the problem is that "update" is not a function. Its job is to
change something, not return something. Any possible return value would
be a convenience for certain tasks and useless for other tasks.

It's also hard to remember, in my opinion. For example, JavaScript has a
"push" method on the Array object which behaves like Python's "append"
method on lists:

js> var a = [];
js> a.push(5);
1
js> a.push(6);
2

I bet you that almost nobody knows that "push" returns the new length of
the Array. It could just as easily have returned "a" here. I could
always write "a.length", if I really needed to know the new length. This
sort of thing becomes language trivia, and when I write in JavaScript I
always ignore the result of "push" because, even if *I* know what it
returns, chances are that my readers don't.

Another reason that Python adopts the convention of returning None for
mutators is that it discourages the use of side-effecting code in
expressions. Mixing side-effects with expressions can lead to code that
is hard to read and understand. This is often debated by those who know
better and wish to write things like "h.update(a).update(b)" (method
chaining) or "while (line = file.readline()): ...". Python's decision is
pretty clear, and it's also evident in the division between statements
and expressions.

Regardless of whether you like Python's style decision or not, if you
dig deeper I think you will find that it is pretty consistent, and that
there are useful benefits to Python's way of handling side effects.

Dave

PS. If mutators had to return a value, I'd have them return "self",
probably 95% of the time. But then, it wouldn't be Python anymore. It'd
be Ruby, maybe.

 

[Bengt Richter]

Let's say I define a list of pairs as follows:

l = [('d', 3), ('a', 2), ('b', 1)]

Can anyone explain why this does not work?
and instead I have to go:to initialize a dictionary with the given list of pairs?

when an analagous operation on strings works fine:

s = "".join(["d","o","g"])

Seems inconsistent.

Join isn't that good an analogy, because the inputs and outputs are immutable.
list.sort is more comparable to dict.update. There is now a builtin sorted function
that will take a list and return a sorted one. Perhaps there will come an "updated"
function analogously for dictionaries, but I don't think the use is that common,
and you can make your own. Several ways. If you want to start with an empty dict
anyway, the dict constructor will accept a sequence of pairs (so long as the first
of every pair is hashable, which is also required for update). E.g.,

>>> dict([('d', 3), ('a', 2), ('b', 1)])

{'a': 2, 'b': 1, 'd': 3}

or the sequence of pairs as a tuple expression {'a': 2, 'b': 1, 'd': 3}

or sometimes it's handy to start with the keys and values as separate sequences
and zip them together for the dict constructor

>>> dict(zip('dab', [3,2,1]))

{'a': 2, 'b': 1, 'd': 3}

or a generator expression
{'a': 2, 'b': 1, 'd': 3}


though note that dict wants the sequence as a single argument: Traceback (most recent call last):

>>> def updated(d, seq): d=d.copy(); d.update(seq); return d ...
>>> updated({}, [('d', 3), ('a', 2), ('b', 1)])

{'a': 2, 'b': 1, 'd': 3}

One rationale for not returning a reference to the mutated value
from a mutating method is that it is too easy to think of it
as a pure expression, and forget the persistent side effect of
mutating the object. I think that was thought too bug-prone.

Regards,
Bengt Richter

 

[Mike Meyer]

Python is actually quite consistent in this regard: methods that
modify an object in-place return None;

Um, no. list.pop comes to mind as an immediate counterexample. It may
be the only one...

<mike

 

[Dave Benjamin]

Um, no. list.pop comes to mind as an immediate counterexample. It may
be the only one...

I'm sure there are counterexamples... maybe 95% is too optimistic.
Anyone want to volunteer to compile some stats?

I've never had to look up the return type of "pop" though. The only
thing even remotely ambigious about that term (at least, if you've
learned what a stack is) is whether it mutates the object, but I don't
think I've ever seen a "pop" that didn't (aside from toy examples in
formal methods / ADT related classes).

"os.system" might be a better example, since the return value could be
one of two obvious things: the status code, or the program's output.

Dave

 

[Erik Max Francis]

I'm sure there are counterexamples... maybe 95% is too optimistic.
Anyone want to volunteer to compile some stats?

Well, that's because list.pop mutates a list _and_ returns something
meaningful (not the list itself). One could modify the above statement
about consistency to say that methods which modify an object and do not
return a directed value do not return the object which was mutated, but
rather return None. So list.append returns None, but list.pop returns
the element popped.

 

[Dennis Lee Bieber]

Also, you shouldn't use "1", I mean "l", as a variable name. It gets confusing
because "l", I mean "1", looks a lot like "1", I mean "l".

In my client, 1(one) is obvious... but I (eye) and l (ell) look
identical...
lIlIlIlIl
--

 

[Kay Schluehr]

Let's say I define a list of pairs as follows:

l = [('d', 3), ('a', 2), ('b', 1)]

Can anyone explain why this does not work?
and instead I have to go:to initialize a dictionary with the given list of pairs?

when an analagous operation on strings works fine:

s = "".join(["d","o","g"])

Seems inconsistent.

If you define

sep = ""
sep.join(["d","o","g"]) "dog"
sep

''

sep is preserved and a new "dog" string is generated. Since sep is
immutable there is no way to manipulate it inplace.

On the other hand there exists no sorted() method for tuples or lists
like join() for strings but it is implemented as a function in Python24
that returns a new sorted container. I consider this as an
inconsistency across builtin types. Consistent would be following usage
pattern:

l = [1,3,2]
l.sorted() [1,2,3] # new sorted list
l.sort() # sort list inplace
l.appended(4) # new extended list [1,2,3,4]
l.append(4) # appends an element to the same list
l

[1,2,3,4]

Preserving the naming convention we would have

"".joined(["d","o","g"])

"dog"

Kay

 

[lechequier]

In a previous post, I asked about the inconsistency in usage patterns
in operating on mutable and immutable types. Thanks Dave and everyone
else for answering my question so thoughtfully and helping me to
understand the reasoning about why the different usage patterns are not
deemed to be inconsistent.

But I am still puzzled by the argument that has been given for why
methods that operate on mutable types should return None, namely, that
the designers of python didn't want the users to shoot themselves in
the foot by thinking a method simply returned a result and left the
data structure unchanged.

In the context of fundamental design principles, if you asked a random
sample of Python gurus what is more Pythonesque: preventing users from
shooting themselves in the foot or making things easier to accomplish,
my impression is that people would overwhelmingly choose the latter.
After all, the fact that Python is not strongly typed and is
interpreted rather than compiled gives plenty of ways for people to
shoot themselves in the foot but what is gained is the abilitity to do
more with less code.

But in this instance, by not allowing operations on mutable types to
return the mutated objects, it seems that the other side is being
taken, sacrificing programmer producitivity for concerns about
producing possible side effects. It is somewhat ironic, I think, that
Java, a language whose design principles clearly side on preventing
users from shooting themselves in the foot, much more so thatn Python,
generally allows you to get back the mutated object.

I'm not trying to change minds here but just to understand better how
this particular design decision fits into Python's overall design
principles.

thanks,
Scott

Let's say I define a list of pairs as follows:

l = [('d', 3), ('a', 2), ('b', 1)]

Can anyone explain why this does not work?

h = {}.update(l)

and instead I have to go:

h = {}
h.update(l)

to initialize a dictionary with the given list of pairs?

when an analagous operation on strings works fine:

s = "".join(["d","o","g"])

Seems inconsistent.

Python is actually quite consistent in this regard: methods that modify
an object in-place return None; methods that do not modify an object
in-place return a new object instead. Since strings are immutable, they
cannot be modified in-place, so it makes sense for the "join" method to
return a new string. On the other hand, Python's dictionaries are
imperative-style and so most operations on a dictionary modify an
existing dictionary.

I was initially bothered by the phenomenon of so many methods returning
None because they could not be chained. But I have come to deeply
appreciate this style for a couple of reasons. First, it makes it clear
which methods are side-effecting (like "update") and which are not (like
"sort").

Second, it is not always clear what a good return value is for a
mutator. Perhaps {}.update() should return the dictionary, making
chaining convenient. Perhaps it should return the total number of items
after updating. Or maybe it should return the number of new keys that
were added, or a list of those keys. All of these are plausible
behaviors; the problem is that "update" is not a function. Its job is to
change something, not return something. Any possible return value would
be a convenience for certain tasks and useless for other tasks.

It's also hard to remember, in my opinion. For example, JavaScript has a
"push" method on the Array object which behaves like Python's "append"
method on lists:

js> var a = [];
js> a.push(5);
1
js> a.push(6);
2

I bet you that almost nobody knows that "push" returns the new length of
the Array. It could just as easily have returned "a" here. I could
always write "a.length", if I really needed to know the new length. This
sort of thing becomes language trivia, and when I write in JavaScript I
always ignore the result of "push" because, even if *I* know what it
returns, chances are that my readers don't.

Another reason that Python adopts the convention of returning None for
mutators is that it discourages the use of side-effecting code in
expressions. Mixing side-effects with expressions can lead to code that
is hard to read and understand. This is often debated by those who know
better and wish to write things like "h.update(a).update(b)" (method
chaining) or "while (line = file.readline()): ...". Python's decision is
pretty clear, and it's also evident in the division between statements
and expressions.

Regardless of whether you like Python's style decision or not, if you
dig deeper I think you will find that it is pretty consistent, and that
there are useful benefits to Python's way of handling side effects.

Dave

PS. If mutators had to return a value, I'd have them return "self",
probably 95% of the time. But then, it wouldn't be Python anymore. It'd
be Ruby, maybe.

 

[Mike Meyer]

I've never had to look up the return type of "pop" though. The only
thing even remotely ambigious about that term (at least, if you've
learned what a stack is) is whether it mutates the object, but I don't
think I've ever seen a "pop" that didn't (aside from toy examples in
formal methods / ADT related classes).

Eiffel STACK class has a pop that dosn't return a value - at least in
SmartEiffel. On the other hand, Eiffel makes a *big* deal about the
difference between command features - which change the state - and
query features - which report on the state. That distinction even
shows up in the threading model for Eiffel.

"os.system" might be a better example, since the return value could be
one of two obvious things: the status code, or the program's output.

To me, os.system obviously returns the status code. Then again, I know
it's just the unix system() routine, which I've know *much* longer
than there's been a Python.

The point is that what's "obvious" depends on where you're coming
from. If your background is as a Unix developer, the result of
os.system is obvious. If your background is Eiffel or some other
strong ADT area, that pop has a result at all may surprise you.

<mike

 

[Mike Meyer]

If you define

sep = ""
sep.join(["d","o","g"]) "dog"
sep

''

sep is preserved and a new "dog" string is generated. Since sep is
immutable there is no way to manipulate it inplace.

On the other hand there exists no sorted() method for tuples or lists
like join() for strings but it is implemented as a function in Python24
that returns a new sorted container. I consider this as an
inconsistency across builtin types. Consistent would be following usage
pattern:

l = [1,3,2]
l.sorted() [1,2,3] # new sorted list
l.sort() # sort list inplace
l.appended(4) # new extended list [1,2,3,4]
l.append(4) # appends an element to the same list
l

[1,2,3,4]

Yes, but the function "sorted" is more useful than a list method
"sorted" in a duck typing language.

The function sorted works on all iterators. I can do:

and have it work.

If sorted were a method of a class - the it'd have to be implemented
again for every class iterable class. Either that, or you'd have to
create an abstract parent of all iterable classes to add it to - which
seems more appropriate for a B&D language than Python.

And even if you do add the abstract class, how do you make my example
work without explictly converting the iterator to a list type?

<mike

 

[Terry Reedy]

On the other hand there exists no sorted() method for tuples or lists
like join() for strings but it is implemented as a function in Python24
that returns a new sorted container. I consider this as an
inconsistency across builtin types.

The sorted function is not a list method because it is not only a list
function or even only a tuple and list function or even only a string,
tuple, list, array, or dict function. Its input is **any** iterable. The
only way to have it both be general and a method would be to have an
iterable type and to require that all iterables inherit from that type to
reap the benefit of being an iterable. All the itertools functions are
also functions and not methods of a hypothetical iterable type. 'Iterable'
is a duck type and hence functions thereof must be functions and not
methods.

Terry J. Reedy

 

[Kay Schluehr]

Yes, but the function "sorted" is more useful than a list method
"sorted" in a duck typing language.

I don't see what this has to do with "duck typing"? sorted() is simply
a generic function accepting different types. I'm not aware that
sorted() requires a specific interface of those types it accepts.

The function sorted works on all iterators. I can do:


and have it work.

If sorted were a method of a class - the it'd have to be implemented
again for every class iterable class. Either that, or you'd have to
create an abstract parent of all iterable classes to add it to - which
seems more appropriate for a B&D language than Python.

Instead of extending a class hierarchy it might even be possible to
hook a trait into the class by means of a __traits__ attribute.

http://fsl.cs.uiuc.edu/~mhills/presentations/TraitsPresentation.pdf

Generators as well as lists and tuples would provide a sortable trait.
The sorted() function could remain available for convenience.

And even if you do add the abstract class, how do you make my example
work without explictly converting the iterator to a list type?

I don't know how sorted() is implemented? A naive implementation would
in fact be nothing else then:

def sorted(iter):
l = list(iter)
l.sort()
return l

Kay

 

[Kay Schluehr]

The sorted function is not a list method because it is not only a list
function or even only a tuple and list function or even only a string,
tuple, list, array, or dict function. Its input is **any** iterable. The
only way to have it both be general and a method would be to have an
iterable type and to require that all iterables inherit from that type to
reap the benefit of being an iterable. All the itertools functions are
also functions and not methods of a hypothetical iterable type. 'Iterable'
is a duck type and hence functions thereof must be functions and not
methods.

Terry J. Reedy

So what? Then an iterable class providing the __iter__ method may be
factored out as Mike reasonably suggested ( I was wrong with my remark
about duck-typing. The __iter__ method may be the interface I claimed
not being aware of ). Or a sortable trait gets used as I would slightly
prefer - but it's not necassary. To be honest I'm not sure what all the
BDFLs Javaesque interfaces and optional static typing blabla for Py3K
should matter if it's not even possible to create obvious inheritance
hierarchies in favour for accidental generic functions?

Kay

 

[Neal Norwitz]

But I am still puzzled by the argument that has been given for why
methods that operate on mutable types should return None, namely, that
the designers of python didn't want the users to shoot themselves in
the foot by thinking a method simply returned a result and left the
data structure unchanged.

Let me try to answer your question with a question. Given this code:

>>> d = {}
>>> e = d.update([(1, 2)])

If .update() returned a dictionary, does d == e?

I'm not sure what you would guess. I am pretty sure that everyone
wouldn't agree whether d should equal e or not. If they are not equal,
that would mean a new copy would be made on each update which could be
incredibly expensive in speed and memory. It is also different from
how Python works today, since the update() method mutates the
dictionary.

In the context of fundamental design principles, if you asked a random
sample of Python gurus what is more Pythonesque: preventing users from
shooting themselves in the foot or making things easier to accomplish,
my impression is that people would overwhelmingly choose the latter.

Probably true, but ...

After all, the fact that Python is not strongly typed and is
interpreted rather than compiled gives plenty of ways for people to
shoot themselves in the foot but what is gained is the abilitity to do
more with less code.

I think most people programming Python are pretty pragmatic. There is
no single language that is ideal in all circumstances. There are
necessarily some trade-offs. Many believe that tools can help bridge
this gap. There are at least 2 tools for finding bugs (or gotchas) of
this sort: pychecker and pylint.

But in this instance, by not allowing operations on mutable types to
return the mutated objects, it seems that the other side is being
taken, sacrificing programmer producitivity for concerns about
producing possible side effects. It is somewhat ironic, I think, that
Java, a language whose design principles clearly side on preventing
users from shooting themselves in the foot, much more so thatn Python,
generally allows you to get back the mutated object.

I think Python has attempted to create an internal consistency. I
believe Java has tried to do the same. However, these aren't the same
sets of consistency. People are always going to have assumptions.
Python strives to be as intuitive as possible. However, it can't be
right 100% of the time for all people.

HTH,
n

 

[Fredrik Lundh]

After all, the fact that Python is not strongly typed and is
interpreted rather than compiled

if you think those are facts, you don't understand how Python
works.

(hint: Python's both strongly typed *and* compiled)

sacrificing programmer producitivity

the ability to save a newline here and there doesn't affect the
programmer productivity in any way whatsoever.

</F>

 

[Steve Holden]

Mike Meyer wrote:




I don't see what this has to do with "duck typing"? sorted() is simply
a generic function accepting different types. I'm not aware that
sorted() requires a specific interface of those types it accepts.

Just because you aren't aware of something doesn't stop it being true.
The argument must be iterable, and there's a specific protocol for that.

Instead of extending a class hierarchy it might even be possible to
hook a trait into the class by means of a __traits__ attribute.

http://fsl.cs.uiuc.edu/~mhills/presentations/TraitsPresentation.pdf

Generators as well as lists and tuples would provide a sortable trait.
The sorted() function could remain available for convenience.

The advantage being ... ? Perhaps you have just discovered a really
interesting hammer, and are seeing this problem as a nail?

I don't know how sorted() is implemented? A naive implementation would
in fact be nothing else then:

def sorted(iter):
l = list(iter)
l.sort()
return l

Kay

That would indeed be a naïve implementation. The implementation is, of
course, an implementation detail ;-) In this case it requires that
sort() then provides all the magic - the need for magic doesn't go away!

regards
Steve

 

[James]

Also, you shouldn't use "1", I mean "l", as a variable name. It gets confusing

because "l", I mean "1", looks a lot like "1", I mean "l".

I have seen the same warnning above significantly several times.
Is this problem originally came from the similarities between 'l' and
'1'
or from bad looking news-browser?
Forgive me if it is out of your interests.

-James GOLD