PEP 322: Reverse Iteration (second revision, please comment)

[Raymond Hettinger]

Based on the feedback here on comp.lang.python, the pep has been
updated:

www.python.org/peps/pep-0322.html


The key changes are:

* reversed() is being preferred to ireverse() as the best name.

* the sample implementation now clearly shows a check for a custom
reverse method and a guard against being applied to a mapping.

* added sample output for enumerate.__reversed__ to show
how a custom reverse method would work

* explained why the function is proposed as a builtin and why attachment
to another module or type object is not being considered further.

* expanded comments on the real world use cases to show what the
replacement code would look like.


Please continue to contribute your thoughts. I'm especially interested
in people examining their own code to verify that the new function
adds clarity and improves performance.

Also, please take a look at the revrange() alternative to see if you
prefer it or not.


Raymond Hettinger

 

[Werner Schiendl]

* reversed() is being preferred to ireverse() as the best name.

+1

even though "reverse" as a verb would be more consistent with "zip" and
"enumerate", it could be assumed to change the sequence passed as arg
(it sounds like a command)

So I agree that "reversed" is the best alternative IMHO

Also, please take a look at the revrange() alternative to see if you
prefer it or not.

Personally I think it's easier to write xrange(len(seq)-1,-1,-1) than
to remember one more function name for something that trivial.

In addition, there are rare cases where I did ever need the index.
And in that cases, enumerate is mostly what i *really* wanted.

I cannot remember to ever having had a need for reverse enumerate...

It IMHO much more beatiful to say:

for item in seq:
print item

than:

for i in xrange(len(seq)):
print seq

because the former communicates the intent (do something for all items
in the sequence).

This is one of the things I really miss when doing work in e. g. C


I'm -0 on revrange in general and -1 as a replacement for "reversed".

To me it rather adds complexity than helps...


best regards

Werner

 

[Alex Martelli]

Raymond Hettinger wrote:
...

* the sample implementation now clearly shows a check for a custom
reverse method and a guard against being applied to a mapping.

SyntaxError: 'return' with argument inside generator

So presumably said check must be recoded in the PEP as:

try:
customiter = x.__reversed__
except AttributeError:
pass
else:
for i in customiter():
yield i
return

Yeah, sometimes it WOULD be nice if "return with argument inside
generator" automagically expanded to "for ...: yield / return".

Also, please take a look at the revrange() alternative to see if you
prefer it or not.

It would deal more directly with about half of my use cases but
less directly with the other half, so that's roughly a wash. But
having revrange return a list would just perpetuate the minor wart
that range has now become, and having it NOT return a list (rather
an iterator) would violate the principle of Least Surprise wrt range.

A built-in iterator irange -- with an optional "reverse=" argument,
just like the very useful one you recently added to list.sort -- would be
just as useful as revrange for the latter's use cases, AND be very handy to
me in many more cases in which I currently use xrange and cringe each and
every time. Plus, specifically, I have some uses cases where:

if mustbereversed(...):
seq = xrange(N-1, -1, -1)
else:
seq = xrange(N)
for item in seq:
...

and those would get a somewhat minor benefit from either reverse OR
revrange, since the if/else to prepare seq would still be there,
albeit more readable along one of the two branches. However, for this
kind of use case, an irange with an optional reverse= would be PERFECT:

for item in irange(N, reverse=mustbereversed(...)):
...

now THAT would be blissful indeed.

I opine irange-w-opt-parm would be by far the best solution. E.g.,
in your use case from heapq.heapify, just about custommade for
revrange, in my opinion the three possibilities:
for i in reversed(xrange(n//2)):
...
for i in revrange(n//2):
...
for i in irange(n//2, reverse=True):
...
are just about equivalent -- the concision of revrange is a very
minor benefit even here. The vastly wider usecases of irange might
in fact even let us slowly and gradually start "discouraging" the
use of range and xrange -- that will be the day... I've pined for
an irange even since the iterator protocol appeared. And the use
of the "reverse=True" idiom will be widespread and habitual anyway
thanks to the fact that it's present in list.sort as well now.

(_and_, IMHO, irange has 100% appropriateness as a builtin, given
that it would/should soon become ***one of the most widely used
constructs in Python*** -- FAR more than revrange or reversed...!).


Alex

 

[Michele Simionato]

Based on the feedback here on comp.lang.python, the pep has been
updated:

www.python.org/peps/pep-0322.html

Thinking about the idea, I realized that my use case for "reversed"
would be very very little: definitely, not enough to justify a new
built-in. OTOH, I would not be opposed to empowering "enumerate"
which I use all the time. Something like

enumerate(iterable,'reversed')

or variations.

Just my opinion,

Michele

 

[Werner Schiendl]

A built-in iterator irange -- with an optional "reverse=" argument,
just like the very useful one you recently added to list.sort -- would be
just as useful as revrange for the latter's use cases, AND be very handy to
me in many more cases in which I currently use xrange and cringe each and
every time. Plus, specifically, I have some uses cases where:

if mustbereversed(...):
seq = xrange(N-1, -1, -1)
else:
seq = xrange(N)
for item in seq:
...

and those would get a somewhat minor benefit from either reverse OR
revrange, since the if/else to prepare seq would still be there,
albeit more readable along one of the two branches. However, for this
kind of use case, an irange with an optional reverse= would be PERFECT:

for item in irange(N, reverse=mustbereversed(...)):
...

now THAT would be blissful indeed.

+1

I'd definitely prefer such a general approach to having "revrange"

However, I think it's no direct replacement for "reversed"


regards

Werner

 

[Dang Griffith]

I cannot remember to ever having had a need for reverse enumerate...

But if you did, how about 'remunerate()' as the name for it?
--dang "artificially intelligent"

 

[David C. Fox]

Based on the feedback here on comp.lang.python, the pep has been
updated:

www.python.org/peps/pep-0322.html


The key changes are:

* reversed() is being preferred to ireverse() as the best name.

+1

(simple, useful, much better name)
David

 

[Jeremy Fincher]

* reversed() is being preferred to ireverse() as the best name.

I *really* don't like this name, especially with the list.sorted
that'll be going into 2.4. People are going to either read this right
and read sorted wrong or vice versa. Having list.sorted return a
sorted copy of its argument and having reversed return a reverse
iterator of its argument seems to be just asking for trouble
(especially when both are added in the same release).

And I really do think the name should imply that what you're getting
is an iterator. I'd much prefer a keyword argument to iter/__iter__,
but I don't think it would be backwards compatible -- old __iter__s
would fail when given a "reverse" keyword. But I do still think the
name should emphasize that we're getting an iterator. How does
reviter and __reviter__ sound? They're nice complements to iter and
__iter__, and they have precedence in

* the sample implementation now clearly shows a check for a custom
reverse method and a guard against being applied to a mapping.

I was going to offer a counter-example of a balanced binary tree being
used as a mapping, until I realized you check for __reversed__ before
doing this check. Still, though, it seems that the check there is
somewhat fragile.

* expanded comments on the real world use cases to show what the
replacement code would look like.

In your atexit example, you say "In this application popping is
required, so the new function would not help." The only reason
_run_exitfuncs pops is in order to iterate in reverse. You might look
through the standard library for the idiom:

while someList:
t = someList.pop()

Because that's exactly the idiom this will replace in a lot of
circumstances.

Also, please take a look at the revrange() alternative to see if you
prefer it or not.

I didn't see anything about revrange() in the PEP.

Jeremy

 

[Jeremy Fincher]

Oops!

In my (slightly) earlier response to this post, I said,

"and they have precedence in"

But *completely* didn't finish that thought (I had a few intervening
phone calls )

Anyway, it would have continued something like this:

And they have precedence in at least O'Caml, which provides a reviter
function on its lists.

(although, admittedly, O'Caml's not exactly a language to look to for
precedence, given its lack of popularity and the fact that it probably
won't ever gain much more than it has now. In retrospect, I probably
wouldn't have made the point at all, but having left "and they have
precedence in" I'm better off finishing the thought than just telling
people to ignore it )

Jeremy

 

[Gonçalo Rodrigues]

Based on the feedback here on comp.lang.python, the pep has been
updated:

www.python.org/peps/pep-0322.html


The key changes are:

* reversed() is being preferred to ireverse() as the best name.

Heh, I actually prefered ireverse. It conveys better that it produces
an iterator.

* the sample implementation now clearly shows a check for a custom
reverse method and a guard against being applied to a mapping.

* added sample output for enumerate.__reversed__ to show
how a custom reverse method would work

* explained why the function is proposed as a builtin and why attachment
to another module or type object is not being considered further.

* expanded comments on the real world use cases to show what the
replacement code would look like.


Please continue to contribute your thoughts. I'm especially interested
in people examining their own code to verify that the new function
adds clarity and improves performance.

I have only a couple of cases where I've needed to use reversed
iteration and in all honesty I don't think that

for index in xrange(len(lst) - 1, -1 -1):
etc.

is all that unreadable. The use cases are so few that I can very well
live with them.

And since it cannot be put inside itertools, I'm -0 on the PEP. The
use cases are so few that I don't think it justifies being put in the
builtins or the growing of Yet Another Protocol.

Just two more notes:

- Making it a method of iter sure looks bizarre. And ugly.

- Shouldn't the implementation test for __contains__ instead of
has_key? e.g.

if hasattr(x, "__contains__"):
raise ValueError("mappings do not support reverse iteration")
etc.

All my mapping-like classes overload __contains__, never has_key, but
maybe that's just me.

With my best regards,
G. Rodrigues

 

[Ron Adam]

even though "reverse" as a verb would be more consistent with "zip" and
"enumerate", it could be assumed to change the sequence passed as arg
(it sounds like a command)

So I agree that "reversed" is the best alternative IMHO

First, I apologize if I'm butting in. I'm still new to Python, but
maybe that viewpoint is needed also.

From looking at the PEP, weather to use reverse or reversed should
depend on if this function can or will be used outside of 'for in'
statements.

If it is only used within for statements, then reversed makes since,
but maybe not as a function, but a keyword used in conjunction with
'for' or 'in'... such as 'in reversed'.

for x in reversed range(items):


If it is used in other cases outside of for loops, then it should
probably be a function that returns a reversed list. In that case it
should be called reverse.

for y in reverse(range(list)):

Would be the same as:

x = reverse(list)
for y in range(x)


I don't know enough about Pythons internals to know if either
of these would cause conflicts. So this is just my opinion on how
readable and natural it looks.


_Ronald R. Adam

 

[Peter Otten]

- Shouldn't the implementation test for __contains__ instead of
has_key? e.g.

if hasattr(x, "__contains__"):
raise ValueError("mappings do not support reverse iteration")
etc.

All my mapping-like classes overload __contains__, never has_key, but
maybe that's just me.

It won't work:

hasattr([], "has_key") False

hasattr([], "__contains__") True

Peter

 

[Paul Moore]

The key changes are:

* reversed() is being preferred to ireverse() as the best name.

Good. The explanation of why reverse() is not an option helps, too.

* the sample implementation now clearly shows a check for a custom
reverse method and a guard against being applied to a mapping.

I stumbled over this, as using the existence of has_key to reject
mappings seemed odd. Surely even without this check, the sample
implementation would fail on a mapping, with a KeyError at the yield?

* added sample output for enumerate.__reversed__ to show
how a custom reverse method would work

Are you proposing to add such a custom reverse method to enumerate?

* explained why the function is proposed as a builtin and why attachment
to another module or type object is not being considered further.

This was very useful. It may not convince everyone, but it helped me
see your point of view a little better.

Please continue to contribute your thoughts. I'm especially interested
in people examining their own code to verify that the new function
adds clarity and improves performance.

I don't think I have any real cases where I'd use this. But I agree
with your comments, that this "feels like" a basic looping construct.

Also, please take a look at the revrange() alternative to see if you
prefer it or not.

Not really. It feels too much like a special case.

Overall, I'm +0 going on +1 (the main reason I'm not +1 is the fact
that I have no code that would actually *use* this feature at
present...)

Paul,

 

[Peter Otten]

I stumbled over this, as using the existence of has_key to reject
mappings seemed odd. Surely even without this check, the sample
implementation would fail on a mapping, with a KeyError at the yield?

But not reliably so. Assuming the following implementation,

def reversed(x):
i = len(x)
while i > 0:
i -= 1
yield x

where the check for a custom reverse iterator is also removed because it's a
syntax error in current Python, consider

sample = dict.fromkeys(range(3))
[i for i in reversed(sample)]

[None, None, None]

But this fails:

del sample[1]
[i for i in reversed(sample)]

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "reversed.py", line 12, in reversed
yield x
KeyError: 1
That is, you could reverse-iterate dictionaries if and only if the
dictionary d has entries for key in range(len(d)), if it were not for the
has_key attribute check, or - with the notorious

def sorted(l):
l.sort()
return l

sorted(d.keys()) == range(len(d))

Peter

 

[Raymond Hettinger]

[Raymond]
[Paul Moore]

I stumbled over this, as using the existence of has_key to reject
mappings seemed odd. Surely even without this check, the sample
implementation would fail on a mapping, with a KeyError at the yield?

Without a guard for mappings, the following would behave strangely:

d = {0:'zero', 1:'one', 3:'three'}

Peter Otten pointed-out that some user defined mappings have
__contains__ rather than has_key, so the existence of a "keys"
may make a better check.

Are you proposing to add such a custom reverse method to enumerate?

Yes, that has been requested more than once.
However, it was listed in the PEP mainly to give a clear example
of how a custom reverse could work.

This was very useful. It may not convince everyone, but it helped me
see your point of view a little better.
Thanks.



Not really. It feels too much like a special case.

Overall, I'm +0 going on +1 (the main reason I'm not +1 is the fact
that I have no code that would actually *use* this feature at
present...)

Noted.

Raymond