Comment on PEP-0322: Reverse Iteration Methods

[Raymond Hettinger]

Please comment on the new PEP for reverse iteration methods.
Basically, the idea looks like this:

for i in xrange(10).iter_backwards(): # 9,8,7,6,5,4,3,2,1,0
<do something with i>

The HTML version is much more readable than the ReST version.
See:
http://www.python.org/peps/pep-0322.html


Several interesting ideas surfaced in the pre-pep thread:

* Call it ireverse() instead of iter_backwards().

Good idea. This is much more pithy.


* Use object properties instead of methods.

I can't yet claim to understand what the author is really
proposing. It has something to do which providing
access to an object that responds to iter, getitem, and
getslice with reversed indices.


* using a single function that looks for an __riter__ magic
method and, if not found, use __getitem__ and __len__
to build a reverse iterator.

A workable version of this was posted.
It saves implementing some object methods at the
expense of adding a new builtin function and of
creating a new magic method name.
It is slower than direct access to the underlying object.
It crashes when applied to an infinite iterator.
It produces bizarre results when applied to mappings.


* special markers for infinite iterators

This idea is interesting but doesn't extend well
when the object is wrapped by another iterator.
Also, there is no automatic way to determine
which generators can be infinite.


Raymond Hettinger

 

[Sean Ross]

Please comment on the new PEP for reverse iteration methods.

# from PEP 322
"""
Rejected Alternative Ideas
[snip]
Add a builtin function, reverse() ...
"""

Do you mean add a function reverse() (or ireverse()) to the itertools
library, or to __builtins__?

 

[John Roth]

I think it says what needs to be said.

+1

John Roth

 

[Stephen Horne]

* Use object properties instead of methods.

I can't yet claim to understand what the author is really
proposing. It has something to do which providing
access to an object that responds to iter, getitem, and
getslice with reversed indices.

The idea is basically to have a way to get a proxy to a sequence which
behaves like the original sequence, but with indexes reversed. There
is no reason why member functions couldn't return those proxies
instead of using properties, except that when the result is sliced it
looks less ugly.

I have a demo of it for lists only (listing at end of post). It has
three properties...

backward : reversed, slicing returns a list
ibackward : reversed, slicing returns an iterator
iforward : forward, slicing returns an iterator

All properties currently support __getitem__, __setitem__,
__delitem__, __str__, __repr__, __iter__ and __len__ (they all return
an instance of the same proxy class but with different options set).

For simple iterating-over-ints, I'd use the same principles with that
object-acting-as-sliceable-set-of-all-integers idea from the PEP284
thread.


I had some problems getting the slice translation to work - there are
some little nasties with negative steps. You can write...

x [::-1]

....but trying to make it explicit...

x [len(x)-1:-1:-1]

....breaks it because of that -ve stop bound. Not being a Numeric user,
slice steps are quite a new thing to me - and that little trap just
caught me by surprise. It isn't too hard to work around it, though.
The code following isn't pretty but so far as I can tell it works.

Python 2.3 is assumed.

The question is - is the exercise worthwhile. I'm not sure even though
it's my own idea. I think it is nice in its way, and a lot more
flexible than the 'iter_backward' method proposal, but that is quite
likely a bad thing as it is probably massive overkill for the problem
being solved - and in particular the extra flexibility has costs.

What the hell, though - it was fun to play with for a bit!

Anyway, here is a quick test session...

from rilist import rilist
x=rilist(range(20))
x.backward [19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
x.backward[:10] [19, 18, 17, 16, 15, 14, 13, 12, 11, 10]
x.backward[::2] [19, 17, 15, 13, 11, 9, 7, 5, 3, 1]
del x.backward[::2]
x.backward [18, 16, 14, 12, 10, 8, 6, 4, 2, 0]
x [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
for i in x.backward :

.... print i
....
18
16
14
12
10
8
6
4
2
0

for i in x.backward [5:]:

.... print i
....
8
6
4
2
0

for i in x [5:]:

.... print i
....
10
12
14
16
18


I've done some 'ibackward' tests as well, and that seems to work - but
I haven't bothered with 'iforward'.


Here is the source - warning, it isn't very self-documenting ATM...


class c_List_Proxy (object) :
"""
Proxy to list class which supports a subset of methods with
modified behaviours, mainly...

- Slicing may create an iterator rather than a list.
- Subscripts and ranges may be reversed.
"""

def __init__ (self, p_List, p_Reverse=False, p_Iter_Slice=False) :
self.List = p_List
self.Reverse = p_Reverse
self.Iter_Slice = p_Iter_Slice

# need some support stuff

def __SliceIter__ (self, p_Slice) :
if p_Slice.step > 0 :
i = p_Slice.start
while i < p_Slice.stop :
yield self.List
i += p_Slice.step
else :
i = p_Slice.start
while i > p_Slice.stop :
yield self.List
i += p_Slice.step

def __FullIter__ (self) :
i = len (self.List) - 1
while i >= 0 :
yield self.List
i -= 1

def __KeyTransformed__ (self, p_Key) :
if self.Reverse :
if p_Key < 0 :
return (-1) - p_Key

else :
return (len (self.List) - 1) - p_Key

else :
return p_Key

def __Bound__ (self, p_Given, p_Default) :
if p_Given is None :
return p_Default
elif p_Given < 0 :
return len (self.List) + p_Given
else :
return p_Given

def __SliceFixed__ (self, p_Slice) :
l_Step = p_Slice.step or 1

if l_Step == 0 :
raise IndexError, "Step must be non-zero"

elif l_Step > 0 :
l_Start = self.__Bound__ (p_Slice.start, 0)
l_Stop = self.__Bound__ (p_Slice.stop, len (self.List))

else :
l_Start = self.__Bound__ (p_Slice.start, len (self.List) - 1)
l_Stop = self.__Bound__ (p_Slice.stop, -1)

l_Count = (((l_Stop - 1) - l_Start) // l_Step) + 1
l_Stop = l_Start + l_Count * l_Step

return slice(l_Start,l_Stop,l_Step)

def __SliceTransformed__ (self, p_Slice) :
l_Slice = self.__SliceFixed__ (p_Slice)

if self.Reverse :
l_End = len(self.List) - 1
return slice (l_End - l_Slice.start, l_End - l_Slice.stop,
-l_Slice.step)

else :
return p_Slice

# some members are trivial

def __len__ (self) :
return len (self.List)

def __iter__ (self) :
return self.__FullIter__ ()

# some members need a bit more work...

def __str__ (self) :
if self.Reverse :
return str(self.List [::-1])
else :
return str(self.List)

def __repr__ (self) :
if self.Reverse :
return repr(self.List [::-1])
else :
return repr(self.List)

def __getitem__ (self, p_Item) :
if isinstance (p_Item, slice) :
if self.Iter_Slice :
return self.__SliceIter__ (self.__SliceTransformed__ (p_Item))

else :
l_Slice = self.__SliceTransformed__ (p_Item)

if l_Slice.stop == -1 : # annoying special case
return self.List [l_Slice.start::l_Slice.step]

return self.List [l_Slice.start:l_Slice.stop:l_Slice.step]

else :
return self.List [self.__KeyTransformed__ (p_Item)]

def __setitem__ (self, p_Item, p_Value) :
if isinstance (p_Item, slice) :
l_Slice = self.__SliceTransformed__ (p_Item)

if l_Slice.stop == -1 : # annoying special case
self.List [l_Slice.start::l_Slice.step] = p_Value
else :
self.List [l_Slice.start:l_Slice.stop:l_Slice.step] = p_Value

else :
self.List [self.__KeyTransformed__ (p_Item)] = p_Value

def __delitem__ (self, p_Item) :
if isinstance (p_Item, slice) :
l_Slice = self.__SliceTransformed__ (p_Item)

if l_Slice.stop == -1 : # annoying special case
del self.List [l_Slice.start::l_Slice.step]
else :
del self.List [l_Slice.start:l_Slice.stop:l_Slice.step]

else :
del self.List [self.__KeyTransformed__ (p_Item)]

class rilist (list) :
"""
Extend normal list to provide properties giving variants of normal
indexing behaviour - specifically...

indexing slicing returns
-------- ---------------
iforward forward iterator
ibackward backward iterator
backward backward list
"""
def __IFwd__ (self) :
return c_List_Proxy (self, False, True)

def __IBack__ (self) :
return c_List_Proxy (self, True, True)

def __Back__ (self) :
return c_List_Proxy (self, True, False)

iforward = property(__IFwd__ )
ibackward = property(__IBack__)
backward = property(__Back__ )

 

[Andrew Dalke]

Raymond Hettinger:

Please comment on the new PEP for reverse iteration methods.

It says:
] Also, it only works with lists (strings, for example, do not define a
reverse method):
]
] rseqn = list(seqn)
] rseqn.reverse()
] for value in rseqn:
] print value

However, the example code will work on a string, because list(string)
returns a list, which does have a reverse.

] Extending slicing minimizes the code overhead but does nothing
] for memory efficiency, beauty, or clarity.

I don't agree with all three of those. Take for example your mhlib
use case

testfolders.reverse();
for t in testfolders:
do('mh.deletefolder(%s)' % `t`)

The PEP suggests that

for t in testfolders.iter_backwards():
do('mh.deletefolder(%s)' % `t`)

is more beautiful and clearer than

for t in testfolders[::-1]:
do('mh.deletefolder(%s)' % `t`)

(It definitely is more memory efficient if there are enough
folders.)

I don't think I agree with that. If I'm not concerned about memory
efficiency then extended slicing is nice because I don't have to
learn nor explain one more thing. ("There should be one ...")

The beauty and clarity, at least to me, only come into play
when memory is a concern, which is only rarely true for most
of the times iter_backwards is expected to be used. (Estimate
based on the use cases and my own judgement.)

] Should file objects be included? Implementing reverse iteration
] may not be easy though it would be useful on occasion.

I would suggest no - the file may not be seekable.

You suggest changing
] while _exithandlers:
] func, targs, kargs = _exithandlers.pop()
to
] for func, target, kargs in _exithandlers.iter_backwards():
] . . .
] del _exithandlers

One thing to bear in mind is that the semantics are slightly
different. In the first one, the finalizers for func, targs, kargs
*may* be called from last to first. This is the case for
CPython if there are no other references to the _exithandlers
objects. But since Python-the-language makes no guarantees
on that and Jython doesn't support it, it's a bad idea to write
code that way.

That is only a very, very minor nit. Feel free to ignore it


In difflib, I disagree that
result.sort()
return [x for score, x in result[-n:].iter_backwards()]

is easier to understand than
result.sort() # Retain only the best n.
result = result[-n:] # Move best-scorer to head of list.
result.reverse() # Strip scores.
return [x for score, x in result]

The former is more terse, but the latter, existing code is
broken down into bite-size chunks and allows verbose
documentation of each step. That isn't possible when
nearly everything is written in one line.

Also, a variant implementation may be even more concise,
which is to replace
result.append((s.ratio(), x))
with
result.append((-s.ratio(), x))
then get rid of the reverse (since the minus sign reverses the
sort order) and simply do

result.sort()
return [x for score, x in result[:n]]


] heapq.heapify() uses for i in xrange(n//2 - 1, -1, -1)

The proposed form is

for i in xrange(n//2-1).iter_backwards():

Later in your post you mention a reason against a function
which uses __riter__ magic is that it is "slower than direct
access to the underlying object."

If that's a concern then you should also point out that using
iter_backwards for cases like this is slower than the existing
solution. (It has an extra attr lookup and function call. OTOH,
doesn't the -1, -1 require two calls of the negative unary op
code, or has that been fixed? Good, it has, says dis on
some test code.)

] platform._dist_try_harder() uses for n in range(len(verfiles)-1,-1,-1)
] because the loop deletes selected elements from verfiles but needs
] to leave the rest of the list intact for further iteration. This use case
] could be addressed with itertools.ifilter() but would require the
] selection predicate to be in a lambda expression. The net result is
] less clear and readable than the original. A better reformulation is
] to replace the first line with the proposed method.

Are you sure about that? The code is

verfiles = os.listdir('/usr/lib/setup')
for n in range(len(verfiles)-1, -1, -1):
if verfiles[n][:14] != 'slack-version-':
del verfiles[n]

I think a better reformulation is

verfiles = os.listdir('/usr/lib/setup')
verfiles = [name for name in verfiles
if name.startswith("slack-version-")]

(you could put the os.listdir in the list comp. but I've
found that gets ungainly.)

] random.shuffle() uses for i in xrange(len(x)-1, 0, -1)

This isn't a use case. The xrange never returns a 0 while
the iter_backwards one will.

list(xrange(5, 0, -1)) [5, 4, 3, 2, 1]

] Rejected Alternative Ideas

There were two intermediate 'reverse' proposals. Here's the four
I know about:

- call the magic method __riter__

- call the magic method __riter__. If it doesn't exist, use
for i in xrange(len(obj)): yield obj

- call the magic method __riter__. If it doesn't exist, use
for i in itertools.count(): yield[obj[-i]]

(this has the advantage of supporting the infinite sequence
0, 1, 2, ... -3, -2, -1
)

- same as the 2nd but if len doesn't exist use
data = list(obj)
data.reverse()
for x in data: yield x

I agree with the conclusion that the last three have unexpected
interactions with mappings. However, iter *also* has unexpected
interactions with mappings.
.... def keys(self):
.... return [0, 1, 2, 4]
.... def __getitem__(self, key):
.... if key not in self.keys(): raise KeyError(key)
.... return key * 2
.... def values(self):
.... return [self[x] for x in self.keys()]
.... def items(self): return zip(self.keys(), self.values())
.... def len(self): return len(self.keys())
........ print x
....
0
2
4
Traceback (most recent call last):
File "<interactive input>", line 1, in ?

While I think I was the one who originally raised this
objection, given the way Python already behaves wrt
mappings, I retract my objection.

Another possibility, which I just thought of now, is
- call the magic method __riter__. If it doesn't exist, look
for the reverse-iterator adapter, as per PEP-0246.

I don't have any experience with the PEP to judge its
appropriateness nor usefulness. The motivation section
seems to suggest that it would be appropriate.


In your post list a set of problems with the function approaches.

It saves implementing some object methods at the
expense of adding a new builtin function and of
creating a new magic method name.
It is slower than direct access to the underlying object.
It crashes when applied to an infinite iterator.
It produces bizarre results when applied to mappings.

I would like to see this list added to the PEP. I do have'
some qualifiers for the list.

- it doesn't need to be a global function; it could be in,
say, itertools. (I know I originally suggested it as a builtin,
but I'm not saying I'm right

- (see comments above that xrange(5).iter_backwards()
is likely slower than xrange(5, -1, -1) because of the
extra lookup and call)

- the first variant, which only looks for __riter__ and raises
an exception if it doesn't exist, has exactly the same problems
with infinite iterators as calling the new method.

- similarly, it doesn't have additional problems with mappings.


Also, none of the use cases dealt with infinite sequences so
the lack of appropriate support for them shouldn't be considered
an outright rejection.

BTW, my current belief is that if this PEP leads to code
then it should be:
- a new function, either builtin or in itertools

- which uses __riter__ if present (unless we decide to
use the adapt PEP)

- else uses "for i in itertools.count(): yield[obj[-i]]"


Andrew
(e-mail address removed)

 

[Andrew Dalke]

Me:

- else uses "for i in itertools.count(): yield[obj[-i]]"

should be

- else uses "for i in itertools.count(1): yield[obj[-i]]"

Andrew
(e-mail address removed)

 

[sebastien]

The PEP has the following sentence in the rejected alternatives:

* Add a builtin function, reverse() which calls a magic method, __riter__.
I see this as more overhead for no additional benefit.

I believe that this is the better choice because it is the way forward
iteration work in python: you have the iter() function which call a
__iter__ special method.

So, I'd expect to have a riter() function which would call the
__riter__ special method.

I like riter as name to all other alternative because (iter, riter)
looks really like (find, rfind) or (index, rindex) ....

And I still think you don't need it often enough to put it in the
builtin namespace, so the function should go in the itertools module.

 

[Steve Holden]

Please comment on the new PEP for reverse iteration methods.

Looks good. But

[...]

It crashes when applied to an infinite iterator.

[...]
Hmm. My little brain is having difficulty imagining anything that won't.
What am I missing?

regards

 

[Raymond Hettinger]

So, I'd expect to have a riter() function which would call the
__riter__ special method. .. . .
And I still think you don't need it often enough to put it in the
builtin namespace, so the function should go in the itertools module.

If you make a magic method, then the function calling it
needs to be a builtin. They go hand in hand.


Raymond

 

[Raymond Hettinger]

[Andrew Dalke]

However, the example code will work on a string, because list(string)
returns a list, which does have a reverse.

Right. I'll remove the comment.

] Extending slicing minimizes the code overhead but does nothing
] for memory efficiency, beauty, or clarity.
I don't agree with all three of those.

Beauty and clarity are a matter of taste and we differ here.
To some (including me), s[::-1] looks more like line noise
than s.iter_backwards(). Also, I think the word form is
more clear. The situation becomes more acute when the
other indices have values. Compare range[n-1, 0, -1] to
range(1, n-1).iterbackwards(). Whenever negative indicies
are used, it takes more knowledge of the implementation
rules to be able to read,write, understand, and verify the code.

] Should file objects be included? Implementing reverse iteration
] may not be easy though it would be useful on occasion.

I would suggest no - the file may not be seekable.
Agreed.



One thing to bear in mind is that the semantics are slightly
different. .. . .
That is only a very, very minor nit. Feel free to ignore it

Agreed. If finalizer call order is important, then the
'while elist: elist.pop()' form is the way to go. If not,
then the for-loop is the way to go.

The former is more terse, but the latter, existing code is
broken down into bite-size chunks and allows verbose
documentation of each step. That isn't possible when
nearly everything is written in one line.

Even the former can be broken down into more lines if
that is what is needed:

result.sort()
result = result[-n]
return [x for score, x in result.iter_backwards()]

Also, a variant implementation may be even more concise,

You must be pretty confident to take on the Timbot's code ;-)

Later in your post you mention a reason against a function
which uses __riter__ magic is that it is "slower than direct
access to the underlying object."

If that's a concern then you should also point out that using
iter_backwards for cases like this is slower than the existing
solution. (It has an extra attr lookup and function call. OTOH,
doesn't the -1, -1 require two calls of the negative unary op
code, or has that been fixed? Good, it has, says dis on
some test code.)

Yes, it's true. All of the speed comments focus on the inner loop.
I'll reword to make that clear. It is definitely an issue. In Py2.2,
there was a generic sequence iterator that used __getitem__ in
iterators for lists and tuples. In Py2.3, we went to a custom
iterator for each object. There was a significant improvement
in performance.

] platform._dist_try_harder() . . .
I think a better reformulation is

verfiles = os.listdir('/usr/lib/setup')
verfiles = [name for name in verfiles
if name.startswith("slack-version-")]

I like your version better and recommend you submit it as a patch.
I'll take out the ifilter() comment.

] random.shuffle() uses for i in xrange(len(x)-1, 0, -1)
This isn't a use case. The xrange never returns a 0 while
the iter_backwards one will.

It is an important use case. The replacement code is:

for i in xrange(1,len(x)). iter_backwards()

Whenever the indices have any complexity, the forwards version,
followed by .iter_backwards() is, IMHO, much easier to mentally verify.

There were two intermediate 'reverse' proposals. Here's the four
I know about:

I'll try to add these without making the PEP as long as this thread ;-)

I agree with the conclusion that the last three have unexpected
interactions with mappings. However, iter *also* has unexpected
interactions with mappings.

Yes, but that doesn't make it less of a disadvantage.
The proposed list.iter_backwards() method does not have that
disadvantage.

In your post list a set of problems with the function approaches. ....
I would like to see this list added to the PEP.

Okay. They are added.

I do have'
some qualifiers for the list.

- it doesn't need to be a global function; it could be in,
say, itertools. (I know I originally suggested it as a builtin,
but I'm not saying I'm right

If there is to be a magic method, __riter__, then the access
function needs to be a builtin. They go hand in hand.

BTW, don't underestimate the intensity of resistance to adding
new builtins.

- (see comments above that xrange(5).iter_backwards()
is likely slower than xrange(5, -1, -1) because of the
extra lookup and call)

Reworded it to talk only to the inner loop, "approaches using
__getitem__() are slower using a custom method for
each object."

- the first variant, which only looks for __riter__ and raises
an exception if it doesn't exist, has exactly the same problems
with infinite iterators as calling the new method.

- similarly, it doesn't have additional problems with mappings.

Okay, I've moved the comments around for you.

Also, none of the use cases dealt with infinite sequences so
the lack of appropriate support for them shouldn't be considered
an outright rejection.

The use cases were just the ones I found in the library.
With generators and itertools being new, I expect
infinite iterators to become more common.

Also, I have a preference for creating something that
is as robust as possible. Making a builtin function that
doesn't apply to all objects; behaves badly with mappings;
and crashes with an infinite iterator is not my idea of robust.

I really don't understand the overpowering urge to make this a function
and add a new magic method protocol. Why isn't there a similar rage
against dict.iteritems()?

Perhaps, you'll like it better if the method is named ireverse()
instead of iter_backwards().

BTW, my current belief is that if this PEP leads to code
then it should be:
- a new function, either builtin or in itertools

- which uses __riter__ if present (unless we decide to
use the adapt PEP)

- else uses "for i in itertools.count(): yield[obj[-i]]"

Okay, we simply disagree here. That's fine.

I appreciate the thoughtful comments and careful analysis.
They have helped clarify the pep wording and helped to
define the issues.


Raymond

 

[Raymond Hettinger]

[Stephen Horne]

The question is - is the exercise worthwhile. I'm not sure even though
it's my own idea. I think it is nice in its way, and a lot more
flexible than the 'iter_backward' method proposal, but that is quite
likely a bad thing as it is probably massive overkill for the problem
being solved - and in particular the extra flexibility has costs.

What the hell, though - it was fun to play with for a bit!

Yes, it's a one ton solution to a one ounce problem,
but it was an interesting exercise and diversion.



Raymond Hettinger

 

[Raymond Hettinger]

[sebastien]

So, I'd expect to have a riter() function which would call the
__riter__ special method.

Okay, changed pep to list riter() as an alternative.

And I still think you don't need it often enough to put it in the
builtin namespace, so the function should go in the itertools module.

If you have a magic method, __riter__, then the corresponding
function needs to be a builtin. They go hand in hand. The
core parts of the language need to be usable without having
to know about itertools.


Raymond

 

[Tom Anderson]

Please comment on the new PEP for reverse iteration methods.

Looks good. But

[...]

It crashes when applied to an infinite iterator.

[...]
Hmm. My little brain is having difficulty imagining anything that won't.
What am I missing?

ideally, riter(riter(someInfiniteIterator)) would work, though!

tom

 

[Andrew Dalke]

Raymond Hettinger:

Beauty and clarity are a matter of taste and we differ here.
To some (including me), s[::-1] looks more like line noise
than s.iter_backwards().

Understood. In all this, let me make it clearthat I
have a preference for a function over a method, but won't
call it a wart of the BFDL thinks otherwise. I point these
out because I'm rather detail oriented.

You must be pretty confident to take on the Timbot's code ;-)

Well, I did say "may". I wouldn't really want to use it since
inverting the score before the sort is a tricky thing to see and
somewhat unexpected.

(I'm half expecting some enlightening comment about how
negation has subtle interactions with IEEE 754 math -- though
I thought the point of 754 was to make naive approaches like
mine usually correct.

verfiles = os.listdir('/usr/lib/setup')
verfiles = [name for name in verfiles
if name.startswith("slack-version-")]

I like your version better and recommend you submit it as a patch.
I'll take out the ifilter() comment.

But that goes against the pretty good motto of "if it ain't broke,
don't touch it." Granted, XP-style tests are supposed to
help out with that, but I don't have a /usr/lib/setup where I
can test this.

] random.shuffle() uses for i in xrange(len(x)-1, 0, -1)
This isn't a use case. The xrange never returns a 0 while
the iter_backwards one will.

It is an important use case. The replacement code is:

for i in xrange(1,len(x)). iter_backwards()

Ahh, right. Didn't think about that. You should include that
wording in the PEP.

Whenever the indices have any complexity, the forwards version,
followed by .iter_backwards() is, IMHO, much easier to mentally verify.

I agree. When I do need to go backwards I often forgot to get
both -1s in place. Ie, I'll do (n, 0) instead of (n-1, -1).
Well, don't forgot much and more, but there's some tension in
my mind as I worry if I got it correct or not.

BTW, don't underestimate the intensity of resistance to adding
new builtins.

For me it's two things: remembering what all these new things
do, and attempting to adhere to the 'don't use variables with
the same name as builtins' guiding philosophy. I don't adhere
to the latter wrt the type named 'file'.

Also, I have a preference for creating something that
is as robust as possible. Making a builtin function that
doesn't apply to all objects; behaves badly with mappings;
and crashes with an infinite iterator is not my idea of robust.

The backup which uses __getitem__ and negative offsets
does work for all objects, behaves exactly as badly as the
existing iter, and doesn't crash with an infinite iterator, no?

It's slow, but if you want the performance, make an __riter__.

I really don't understand the overpowering urge to make this a function
and add a new magic method protocol. Why isn't there a similar rage
against dict.iteritems()?

Overpowering?

My preference is for a function because of:
- similaraties to iter, which is also a function
- there are many types of iteration orderings, and not all can be
special cased methods.
- I don't see backwards iteration as all that important

As for iteritems, I was annoyed about it because it meant my
dictionary-like code would become more complicated to
implement, until I came across UserDict.DictMixin, which
lets user-defined code implement just a few methods then
builds the rest from there.

That suggests a UserList.ListMixin might be useful, but it
isn't obvious to me that it would be.

I also note that items is a method so by similarity it makes
sense that iteritems also be a method. Additionally, many
things offer forward iteration, while only dicts offer items.

Should dicts support riteritems, ritervalues?

Okay, we simply disagree here. That's fine.

Yup. I don't think I have anything useful or even interesting
to offer to this dicusssion. I also think that with these changes,
the PEP is pretty complete and solid.

Andrew
(e-mail address removed)

 

[Stephen Horne]

Please comment on the new PEP for reverse iteration methods.

One more thought.

Having an iter_backwards method on xrange seems wrong. This suggests
multiple functions/methods...

To generate backward ranges
range_backwards function
xrange_backwards funtion

for i in range_backwards (10) :
...

To iterate an object backwards
either method for appropriate objects, or
iter_backward() function and __iter_backward__ magic method
(I don't mind abbreviating the 'iter', but I think the 'backward'
needs to be kept very obvious and not reduced to an easily missed
'r' prefix).

for i in iter_backwards (seq) :
...

To enumate backwards etc
Add an enumerate_backward function which requires the input to
support __len__.

for i in enumerate_backwards (seq) :
...

For backward slicing
Add a function which translates the slice to a backward equivalent
before applying it...

for i in slice_backwards (seq, start, stop, step) :
...

 

[sebastien]

If you have a magic method, __riter__, then the corresponding
function needs to be a builtin. They go hand in hand. The
core parts of the language need to be usable without having
to know about itertools.

I don't think so:
We already have this situation for the the copy module:
copy.copy() use the __copy__ magic method and copy.deepcopy() use
__deepcopy__
I've never heard about problems with this behaviour.

In fact it's not the __riter__ magic method which will use riter()
function but the opposite, so an object which define __riter__ doesn't
need to now anything about riter().

 

[Alex Martelli]

If you have a magic method, __riter__, then the corresponding
function needs to be a builtin. They go hand in hand. The
core parts of the language need to be usable without having
to know about itertools.

I have already seen module copy presented as a counter-example to
your assertion, and I'd like to add module pickle as a second, and
I hope decisive, counter-example. It is, to put it simply, utterly
false that functions corresponding to magic methods "need to be
builtins": it's *perfectly* all right for such functions to live
in standard library modules. Oh, and let's not forget module sets:
the __as_immutable__ and __as_temporarily_immutable__ magic methods,
that are used when making a set of sets, or checking for a set's
membership in another set, can be seen as yet another example (and
in this case there is no wiggling out of it by claiming that the
magicmethod/NON-builtin correspondence is a historical/legacy thing,
as the BDFL approved module sets.py, with just this usage, so very
recently).

I second the motion that function riter, with check for __riter__
and all, should live in module itertools. Reverse iteration is a
RARE need -- far rarer than copying or even pickling -- and there
is no real reason to burden __builtins__ with a function for it.


Alex

 

[John Roth]

I agree. Importing itertools to get at riter()
isn't a significant hardship.

John Roth

 

[David Abrahams]

I have already seen module copy presented as a counter-example to
your assertion, and I'd like to add module pickle as a second, and
I hope decisive, counter-example. It is, to put it simply, utterly
false that functions corresponding to magic methods "need to be
builtins": it's *perfectly* all right for such functions to live
in standard library modules. Oh, and let's not forget module sets:
the __as_immutable__ and __as_temporarily_immutable__ magic methods,
that are used when making a set of sets, or checking for a set's
membership in another set, can be seen as yet another example (and
in this case there is no wiggling out of it by claiming that the
magicmethod/NON-builtin correspondence is a historical/legacy thing,
as the BDFL approved module sets.py, with just this usage, so very
recently).

I second the motion that function riter, with check for __riter__
and all, should live in module itertools. Reverse iteration is a
RARE need -- far rarer than copying or even pickling -- and there
is no real reason to burden __builtins__ with a function for it.

Is there any concern about the fact that a sequence of direction
changes requires building a new iterator object each time under this
proposal? I was going to implement my own bidirectional iteration
protocol by using a prev() method and use an adaptor object to swap
the meaning of next() and prev(). Python iterators (and GoF
iterators) seem poorly suited as pure sequence position indicators
because you can't reposition them without also accessing elements, so
maybe this is not such a serious issue... but my instincts tell me
that the identity of a bidirectional iterator object could be useful
if we allow the same one to be used in both directions.

FWIW, burdening builtins aside I consider the proposed syntax

for i in xrange(n).iter_backwards():

much uglier than

for i in reverse_view(xrange(n)):

but far superior to

for i in reverse(xrange(n)):

which implies in-place modification of the sequence.

Also, the idea of denying tuples the ability to reverse iterate seems
arbitrary and capricious.

Cheers,

 

[Sean Ross]

[snip]

FWIW, burdening builtins aside I consider the proposed syntax

for i in xrange(n).iter_backwards():

much uglier than

for i in reverse_view(xrange(n)):

but far superior to

for i in reverse(xrange(n)):

which implies in-place modification of the sequence.

How about

from itertools import ireverse
for i in ireverse(xrange(n)):
# suite

ireverse(), like imap(), izip(), etc., suggests that the operation is
iterative, and that no modification of the original sequence will be
performed. Others have suggested riter() (right iteration), in order to form
an association with the iter() builtin. As a matter of taste, I prefer
ireverse().

Sean