Top and Bottom Values [PEP: 326]

[Antoon Pardon]

I know this PEP is rejected. However I have a problem that
would benefit from having extreme values and the sample
implementation that is given in the PEP is unsatifactory
for my purpose.

I had written my own module, which works similarly but
is somewhat extended. Here is an example of how it can
be used and how I would like to use it but get stuck.

from extreme import Top

Top Top
Top + 1 Top
Top - 30 Top
Top > 1e99 True
lst = range(10)
lst[:Top]

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: slice indices must be integers or None

So this is where I am stuck. I need this Top value in
a context where it can be used as a start or stop value
in a slice. My idea was that since a stop value greater
than the length of lst in this context would simply return
a copy of lst, using Top in such a way should behave so
too. However I don't see a way of doing that.

So can someone provide ideas to get this behaviour?
If not is there a chance that PEP 326 gets revived?

 

[Lawrence D'Oliveiro]

I need this Top value in
a context where it can be used as a start or stop value
in a slice.

But the only valid values allowed for indices are 0 up to the length of the
array inclusive. Larger integers are not allowed, so why should Top be
allowed?

 

[John Machin]

But the only valid values allowed for indices are 0 up to the length of the
array inclusive. Larger integers are not allowed, so why should Top be
allowed?

Larger integers are not allowed as subscripts, but slicing is more
tolerant:

| >>> array = ['a', 'b', 'c']
| >>> array[:1]
| ['a']
| >>> array[:2]
| ['a', 'b']
| >>> array[:3]
| ['a', 'b', 'c']
| >>> array[:4]
| ['a', 'b', 'c']
| >>> array[:987654321]
| ['a', 'b', 'c']

BTW, negative subscripts are allowed, too; see e.g.
http://docs.python.org/tut/node5.html#SECTION005140000000000000000

HTH,
John

 

[Peter Otten]

I had written my own module, which works similarly but
is somewhat extended. Here is an example of how it can
be used and how I would like to use it but get stuck.

from extreme import Top

Top Top
Top + 1 Top
Top - 30 Top
Top > 1e99 True
lst = range(10)
lst[:Top]

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: slice indices must be integers or None

So this is where I am stuck. I need this Top value in
a context where it can be used as a start or stop value
in a slice. My idea was that since a stop value greater
than the length of lst in this context would simply return
a copy of lst, using Top in such a way should behave so
too. However I don't see a way of doing that.

So can someone provide ideas to get this behaviour?

.... def __index__(self):
.... return sys.maxint
....

Top = Top()
range(5)[:Top]

[0, 1, 2, 3, 4]

This can of course fail in many interesting ways...

Peter

 

[Antoon Pardon]

I had written my own module, which works similarly but
is somewhat extended. Here is an example of how it can
be used and how I would like to use it but get stuck.

from extreme import Top

Top Top
Top + 1 Top
Top - 30 Top
Top > 1e99 True
lst = range(10)
lst[:Top]

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: slice indices must be integers or None

So this is where I am stuck. I need this Top value in
a context where it can be used as a start or stop value
in a slice. My idea was that since a stop value greater
than the length of lst in this context would simply return
a copy of lst, using Top in such a way should behave so
too. However I don't see a way of doing that.

So can someone provide ideas to get this behaviour?

import sys
class Top(object):

... def __index__(self):
... return sys.maxint
...

Top = Top()
range(5)[:Top]

[0, 1, 2, 3, 4]

This can of course fail in many interesting ways...

To begin with this already fails:
.... print i
....
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: an integer is required


What bothers me a bit about the rejection of PEP 326 is that one of the
reasons stated is:

http://mail.python.org/pipermail/python-dev/2004-January/042306.html

- it is easily implemented when you really need it

Well I thought it would simplify some things for me, so I tried an
implementation and then found that some of the things that I would
want to do with it wont work. So the "is easily implemented" bit
seems not to be correct.

 

[Paul McGuire]

I had written my own module, which works similarly but
is somewhat extended. Here is an example of how it can
be used and how I would like to use it but get stuck.

from extreme import Top
Top
Top
Top + 1
Top
Top - 30
Top
Top > 1e99
True
lst = range(10)
lst[:Top]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: slice indices must be integers or None

<snip>

What about this?

-- Paul


class TopClass(int):
def __add__(self,other):
return self
def __sub__(self,other):
return self
def __mul__(self,other):
return self
def __div__(self,other):
return self
def __iadd__(self,other):
return self
def __isub__(self,other):
return self
def __imul__(self,other):
return self
def __idiv__(self,other):
return self
def __str__(self):
return "Top"

import sys
Top = TopClass(sys.maxint)

print Top
print int(Top)
print int(Top-1e9)

a = range(10)
print a[:Top]
print a[Top:]
print a[3:Top]


Prints:
Top
2147483647
2147483647
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[]
[3, 4, 5, 6, 7, 8, 9]

 

[Paul McGuire]

I had written my own module, which works similarly but
is somewhat extended. Here is an example of how it can
be used and how I would like to use it but get stuck.

from extreme import Top
Top
Top
Top + 1
Top
Top - 30
Top
Top > 1e99
True
lst = range(10)
lst[:Top]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: slice indices must be integers or None

Here's a little more refined version of my previous post:

class StubbornInt(int):
def invariantOp(self,other):
return self
__add__ = invariantOp
__sub__ = invariantOp
__mul__ = invariantOp
__div__ = invariantOp
__iadd__ = invariantOp
__isub__ = invariantOp
__imul__ = invariantOp
__idiv__ = invariantOp
__radd__ = invariantOp
__rsub__ = invariantOp
__rmul__ = invariantOp
__rdiv__ = invariantOp
def __str__(self):
return self.name

import sys
Top = StubbornInt(sys.maxint)
Top.name = "Top"
Bottom = StubbornInt(-sys.maxint-1)
Bottom.name = "Bottom"

print Top
print int(Top)
print int(Top-1e9)
print Bottom
print Bottom+sys.maxint
print int(Bottom+sys.maxint)

a = range(10)
print a[:Top]
print a[Top:]
print a[3:Top]
print 3+Top
print Top+3
print 5+2/6*Top
print a[Bottom:3]

for i in xrange(Top):
print i
if i > 10: break


prints:

Top
2147483647
2147483647
Bottom
Bottom
-2147483648
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[]
[3, 4, 5, 6, 7, 8, 9]
Top
Top
Top
0
1
2
3
4
5
6
7
8
9
10
11
[0, 1, 2]

 

[Sion Arrowsmith]

I need this Top value in
a context where it can be used as a start or stop value
in a slice.

But the only valid values allowed for indices are 0

range(5)[:-1]

[0, 1, 2, 3]

up to the length of the
array inclusive. Larger integers are not allowed, [ ... ]

range(5)[:sys.maxint]

[0, 1, 2, 3, 4]

 

[Antoon Pardon]

Antoon Pardon wrote:

I had written my own module, which works similarly but
is somewhat extended. Here is an example of how it can
be used and how I would like to use it but get stuck.

from extreme import Top
Top
Top
Top + 1
Top
Top - 30
Top
Top > 1e99
True
lst = range(10)
lst[:Top]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: slice indices must be integers or None

Here's a little more refined version of my previous post:

class StubbornInt(int):
def invariantOp(self,other):
return self
__add__ = invariantOp
__sub__ = invariantOp
__mul__ = invariantOp
__div__ = invariantOp
__iadd__ = invariantOp
__isub__ = invariantOp
__imul__ = invariantOp
__idiv__ = invariantOp
__radd__ = invariantOp
__rsub__ = invariantOp
__rmul__ = invariantOp
__rdiv__ = invariantOp
def __str__(self):
return self.name

import sys
Top = StubbornInt(sys.maxint)
Top.name = "Top"
Bottom = StubbornInt(-sys.maxint-1)
Bottom.name = "Bottom"

Well one problem I have with your solution is the following:
... print x
...
2147483645
2147483646
The idea with a Top (and Bottom) variable is that you once and for
all have a value that is bigger (smaller) than any other value you are
going to use. AFAICS using a subclass of int will always
leave an opportunity open where an other value will be treated
as bigger (smaller), because functions like xrange will just
look at the int value of the object.

 

[Steven D'Aprano]

Here's a little more refined version of my previous post:

class StubbornInt(int):
def invariantOp(self,other):
return self
__add__ = invariantOp
__sub__ = invariantOp

[snip rest of code]

import sys
Top = StubbornInt(sys.maxint)

Top shouldn't need to depend on a specific integer value, but then it is
easy enough to subclass StubbornInt.

However, there is a more serious problem with your definition of Top:
False

But Top is supposed to bigger than everything, right?

Okay, let's fix it:

class FixedTop(StubbornInt):
def __init__(self):
super(FixedTop, self).__init__(sys.maxint)
self.name = 'Top'
def _bigger(self, other):
# Top is bigger than everything
return True
def _smaller(self, other):
# Top is smaller than nothing
return False
__lt__ = __le__ = _smaller
__gt__ = __ge__ = _bigger


Let's test it out:
True


So far so good.

However, the downside of rolling your own Top is the risk that one day
you'll have something like this:

class Funny:
def __init__(self, name):
self.name = name
def __str__(self):
return self.name
def __gt__(self, other):
return len(str(self)) > len(str(other))

and then you'll have problems:
True

As far as I can see, the only way to avoid edge cases like this is for Top
(and Bottom) to be built-ins that are special-cased by Python.

 

[George Sakkis]

What bothers me a bit about the rejection of PEP 326 is that one of the
reasons stated is:

http://mail.python.org/pipermail/python-dev/2004-January/042306.html

- it is easily implemented when you really need it

Well I thought it would simplify some things for me, so I tried an
implementation and then found that some of the things that I would
want to do with it wont work. So the "is easily implemented" bit
seems not to be correct.

IIRC, the PEP proposed the Smallest and Largest singletons with the
sole purpose of being used in comparisons. No numeric behavior was
implied, i.e. Smallest and Largest are not negative and positive
infinity in the math sense of the word. So I guess the "easily
implemented" refers to this case alone.

George

 

[OKB (not okblacke)]

To begin with this already fails:

... print i

What do you expect this to do? Loop forever?

--
--OKB (not okblacke)
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is
no path, and leave a trail."
--author unknown

 

[Tim Chase]

To begin with this already fails:

What do you expect this to do? Loop forever?

Perhaps the infinite loop should take half as long as

Though then the values when the loop first starts are kinda
ambiguous...

Given the limits of xrange:
xrange(10000000000000000000000000,10000000000000000000000009):
.... print i

Traceback (most recent call last):
File "<stdin>", line 1, in ?
OverflowError: long int too large to convert to int

I suspect one would have an overflow condition to help break you
out of the loop...
.... for i in range(Top):
.... print i
.... except OverflowError:
.... print 'done!'



....and then you time your instruction cycles and your disk reads
so they fall under the read-head at just the right time... [*]

-tkc



[*] http://www.catb.org/jargon/html/story-of-mel.html

 

[Antoon Pardon]

IIRC, the PEP proposed the Smallest and Largest singletons with the
sole purpose of being used in comparisons. No numeric behavior was
implied, i.e. Smallest and Largest are not negative and positive
infinity in the math sense of the word.

That is true.

So I guess the "easily implemented" refers to this case alone.

This doesn't follow. Take the example were I got stuck.

lst = range(10)
lst[:Top]

This doesn't need arithmetics done with Top. The only fact that
you need is: Top >= len(lst). In a way this isn't that difficult
in itself, it becomes difficult because python doesn't allow
ducktyping for a lot of its builtins. I could write my own
function:

def leftslice(lst, num):
return [ tp[1] for tp in enumerate(lst) if tp[0] < num ]


This function works as expected if you substituted Top for num
and as you can see, no arithmetic is done on num, only comparisons.

 

[Antoon Pardon]

What do you expect this to do? Loop forever?

Yes that is what I would expect. If someone would ask me
to implement a function like xrange it would look something
like the following (*)

def xrange(start=1, stop, step=1):

while start < stop:
yield start
start += step

Since Top is supposed to be bigger than any other value this would
indeed loop forever. The only reason that this doesn't work with
the builtin, is because the builtin xrange insist on its arguments
being ints instead of allowing duck typing.

(*) Yes I know this isn't legal python. I just wrote it like this
to make the intention clear instead of putting in the code that
would actually behave like xrange with regards to its defaults.
IMO that would just detract from the actual code.

 

[Antoon Pardon]

Perhaps the infinite loop should take half as long as


Though then the values when the loop first starts are kinda
ambiguous...

Given the limits of xrange:

xrange(10000000000000000000000000,10000000000000000000000009):
... print i

Traceback (most recent call last):
File "<stdin>", line 1, in ?
OverflowError: long int too large to convert to int

I suspect one would have an overflow condition to help break you
out of the loop...

Sometime people want an infinite loop.

Personnaly I don't see why I should get an overflow error while
doing this:

import sys
for i in xrange(sys.maxint - 4, sys.maxint + 5):
pass

I remember in the days when longs and ints were being unified,
that some people complained, because without this unification,
a loop in which an int counter was incremented would necessarily
end when the counter overflowed and this was a good indication
there was a bug somewhere. The response to that was, that you
could never foresee how your algorithm would be used and that
maybe someday someone found a good way to use your algoritm
where the counter would go beyond sys.maxint.

Following this logic there is no reason why xrange should be
limited to ints.

... for i in range(Top):
... print i
... except OverflowError:
... print 'done!'

...and then you time your instruction cycles and your disk reads
so they fall under the read-head at just the right time... [*]

[*] http://www.catb.org/jargon/html/story-of-mel.html

There is no reason to compare what I propose to the story of melvin.
All behaviour of the objects would be well defined and could be
easily understood to those who would read the documentation.

 

[Antoon Pardon]

Here's a little more refined version of my previous post:

class StubbornInt(int):
def invariantOp(self,other):
return self
__add__ = invariantOp
__sub__ = invariantOp

[snip rest of code]

import sys
Top = StubbornInt(sys.maxint)

Top shouldn't need to depend on a specific integer value, but then it is
easy enough to subclass StubbornInt.

However, there is a more serious problem with your definition of Top:
False

But Top is supposed to bigger than everything, right?

Okay, let's fix it:

class FixedTop(StubbornInt):
def __init__(self):
super(FixedTop, self).__init__(sys.maxint)
self.name = 'Top'
def _bigger(self, other):
# Top is bigger than everything
return True
def _smaller(self, other):
# Top is smaller than nothing
return False
__lt__ = __le__ = _smaller
__gt__ = __ge__ = _bigger


Let's test it out:
True


So far so good.

However, the downside of rolling your own Top is the risk that one day
you'll have something like this:

class Funny:
def __init__(self, name):
self.name = name
def __str__(self):
return self.name
def __gt__(self, other):
return len(str(self)) > len(str(other))

and then you'll have problems:
True

As far as I can see, the only way to avoid edge cases like this is for Top
(and Bottom) to be built-ins that are special-cased by Python.

An other problem is that if you roll your own, they wont be accepted in
a lot of buitins where they would make sense. Take itertools.repeat as
it is now and I understand correctly there are two possibilities.

1) Your extreme values are not a subclass of int, then you won't be able
to use them as a times arguments.

2) Your extreme values are a subclass of int, then they will somehow
have an int value and this int value will be used, so that the repeat
won't be an infinite loop.

 

[Georg Brandl]

IIRC, the PEP proposed the Smallest and Largest singletons with the
sole purpose of being used in comparisons. No numeric behavior was
implied, i.e. Smallest and Largest are not negative and positive
infinity in the math sense of the word.

That is true.

So I guess the "easily implemented" refers to this case alone.

This doesn't follow. Take the example were I got stuck.

lst = range(10)
lst[:Top]

FWIW, this works with 2.5 and the __index__ slot:
.... def __index__(self):
.... return sys.maxint
....

>>> a=range(5)
>>> a[:Top()] [0, 1, 2, 3, 4]
>>>

Georg

 

[Georg Brandl]

Yes that is what I would expect.

For unterminating loops, use while 1:, and if you need the counter,
itertools.count().

If someone would ask me
to implement a function like xrange it would look something
like the following (*)

def xrange(start=1, stop, step=1):

while start < stop:
yield start
start += step

Since Top is supposed to be bigger than any other value this would
indeed loop forever. The only reason that this doesn't work with
the builtin, is because the builtin xrange insist on its arguments
being ints instead of allowing duck typing.

xrange() *could* be implemented as shown above, but do you realize
that it would be a severe performance hit compared to the current
implementation, which doesn't give almost all users a benefit at all?

Georg

 

[Antoon Pardon]

For unterminating loops, use while 1:, and if you need the counter,
itertools.count().

Neither of these will work if you want an easy way to choose
between repeating a specific number of times and an infinite
loop.

Of course you can include special tests in your code, but the
purpose of values like Top is to eliminate special case code
in a number of situations.

xrange() *could* be implemented as shown above, but do you realize
that it would be a severe performance hit compared to the current
implementation, which doesn't give almost all users a benefit at all?

That code was just meant to illustrate behaviour, one could always
code as follows:

def _xrange(start=1, stop, step=1):

# see implemantation above

def xrange(start=1, stop, step=1):

if (type(start), type(stop), type(step)) == (int, int, int):
return builtin.xrange(start, stop, step)
else:
return _xrange(start, stop, step)

Just to show that with some thinking one could come up with
an implementation that was fast in most cases and yet would
allow for other types than ints if that would make sense.