How to avoid overflow errors

S

Steven D'Aprano

I thought that overflow errors would be a thing of the past now that
Python automatically converts ints to longs as needed. Unfortunately,
that is not the case.
.... pass
....Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: long int too large to convert to int


How do I subclass int and/or long so that my class also auto-converts
only when needed?
 
E

Eduardo O. Padoan

I thought that overflow errors would be a thing of the past now that
Python automatically converts ints to longs as needed. Unfortunately,
that is not the case.

... pass
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: long int too large to convert to int


Not totally unrelated, but in Py3k, as it seems, overflows are really
things of the past:


Python 3.0a1 (py3k:58061, Sep 9 2007, 13:18:37)
[GCC 4.1.3 20070831 (prerelease) (Ubuntu 4.1.2-16ubuntu1)] on linux2
Type "help", "copyright", "credits" or "license" for more information..... pass
....2147483648


Thanks to your mail, only now I see how important this change
(int/liong unification) really is!

How do I subclass int and/or long so that my class also auto-converts
only when needed?

What about just subclassing long - is this not an option?
 
P

Paul Rubin

Eduardo O. Padoan said:
Not totally unrelated, but in Py3k, as it seems, overflows are really
things of the past:


Python 3.0a1 (py3k:58061, Sep 9 2007, 13:18:37)
[GCC 4.1.3 20070831 (prerelease) (Ubuntu 4.1.2-16ubuntu1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.... pass
...2147483648

I'd be interested in knowing what happens in 3.0a1 with

a = itertools.count(sys.maxint)
print a.next()
print a.next()
 
J

James Stroud

Steven said:
I thought that overflow errors would be a thing of the past now that
Python automatically converts ints to longs as needed. Unfortunately,
that is not the case.

... pass
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: long int too large to convert to int


How do I subclass int and/or long so that my class also auto-converts
only when needed?

Use __new__.

py> import sys
py>
py> class MyLong(long):
.... pass
....
py> class MyInt(int):
.... def __new__(cls, *args, **kwargs):
.... try:
.... return int.__new__(cls, *args, **kwargs)
.... except OverflowError:
.... return MyLong(*args, **kwargs)
....
py> MyInt(sys.maxint**2)
4611686014132420609L
py> type(_)
<class '__main__.MyLong'>


James
 
E

Eduardo O. Padoan

Eduardo O. Padoan said:
Not totally unrelated, but in Py3k, as it seems, overflows are really
things of the past:


Python 3.0a1 (py3k:58061, Sep 9 2007, 13:18:37)
[GCC 4.1.3 20070831 (prerelease) (Ubuntu 4.1.2-16ubuntu1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
class MyInt(int):
... pass
...
import sys
MyInt(sys.maxint) 2147483647
MyInt(sys.maxint+1)
2147483648

I'd be interested in knowing what happens in 3.0a1 with

a = itertools.count(sys.maxint)
print a.next()
print a.next()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: cannot count beyond PY_SSIZE_T_MAX

Hum, you've got me there. it is the same as in 2.x. Maybe the message
should be less crypt, at least - nothing that googling for
PY_SSIZE_T_MAX cant help.
 
C

Carl Banks

Eduardo O. Padoan said:
Not totally unrelated, but in Py3k, as it seems, overflows are really
things of the past:


Python 3.0a1 (py3k:58061, Sep 9 2007, 13:18:37) [GCC 4.1.3 20070831
(prerelease) (Ubuntu 4.1.2-16ubuntu1)] on linux2 Type "help",
"copyright", "credits" or "license" for more information.
class MyInt(int):
... pass
...
import sys
MyInt(sys.maxint)
2147483647
MyInt(sys.maxint+1)
2147483648

I'd be interested in knowing what happens in 3.0a1 with

a = itertools.count(sys.maxint)
print a.next()
print a.next()
print(next(a))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: cannot count beyond PY_SSIZE_T_MAX

Hum, you've got me there. it is the same as in 2.x. Maybe the message
should be less crypt, at least - nothing that googling for
PY_SSIZE_T_MAX cant help.

This should demonstrate that OverflowError will not disappear entirely
even in Python 3.

Even if we were to get rid of all OverflowErrors resulting from integer
operations in Python, there are going to be some C extensions--including
some in the Python standard library--that will store 32-bit integers.
Modules such as array and struct will still raise it, and probably other
modules were a long integer sometimes doesn't make sense (socket, for
instance). itertools.count() should work for arbitrary integers, though.


Carl Banks
 
S

Steven D'Aprano

What about just subclassing long - is this not an option?

Of course it's an option. As it turned out, that was the easiest way for
me to proceed. I was a little concerned that the overhead of using longs
might have been excessive, but for values close to zero (and by close I
mean < millions) there's no significant time difference between
arithmetic with ints and longints.
 
S

Steven D'Aprano

Use __new__.

The disadvantage of that is that your example code requires me to
duplicate my methods in the long version and the int version. It's easy
enough to work around that (class factory function) but it just seems all
rather untidy...
 
J

James Stroud

Steven said:
The disadvantage of that is that your example code requires me to
duplicate my methods in the long version and the int version. It's easy
enough to work around that (class factory function) but it just seems all
rather untidy...


The __new__ method is where you implement such decisions. There is no
"tidier" way for one class "auto-convert" to another class, which is
what you asked for (it says so right up there ^). So, if you don't like
the answer, don't ask the question.

Despite its unseemly appearance because of all the underscores, it is
the natural way in python (to do what you have asked and not something
which you didn't ask).

The way to avoid duplicating methods (which is what you are asking for,
no?) while still allowing your classes to maintain their identities is
by using mixins:


class MyModulatingMixin(object):
def modulate(self, sum):
if self.modulus is None:
return self.__class__(sum)
else:
return self.__class__(sum % self.modulus)
def __add__(self, other):
sum = long.__add__(long(self), long(other))
return self.modulate(sum)
def __mul__(self, other):
sum = long.__mul__(long(self), long(other))
return self.modulate(sum)

class MyLong(long):
def __new__(cls, v, m=None):
obj = long.__new__(cls, v)
obj.modulus = m
return obj

class MyInt(int):
def __new__(cls, v, m=None):
try:
obj = int.__new__(cls, v)
obj.modulus = m
except OverflowError:
obj = MyLong(v, m)
return obj

for cls in MyLong, MyInt:
cls.__bases__ = (MyModulatingMixin,) + cls.__bases__


py> i = MyInt(51)
py> j = MyInt(40, 7)
py> k = MyLong(2, 13)
py> i + j
91
py> j + k
0
py> k + j
3L
py> i * k
102
py> type(i * k)
<class '__main__.MyInt'>
py> k * i
11L
py> type(k * i)
<class '__main__.MyLong'>


James
 
E

Eduardo O. Padoan

Not totally unrelated, but in Py3k, as it seems, overflows are really
things of the past:


Python 3.0a1 (py3k:58061, Sep 9 2007, 13:18:37) [GCC 4.1.3 20070831
(prerelease) (Ubuntu 4.1.2-16ubuntu1)] on linux2 Type "help",
"copyright", "credits" or "license" for more information.
class MyInt(int):
... pass
...
import sys
MyInt(sys.maxint)
2147483647
MyInt(sys.maxint+1)
2147483648

I'd be interested in knowing what happens in 3.0a1 with

a = itertools.count(sys.maxint)
print a.next()
print a.next()

print(next(a))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: cannot count beyond PY_SSIZE_T_MAX

Hum, you've got me there. it is the same as in 2.x. Maybe the message
should be less crypt, at least - nothing that googling for
PY_SSIZE_T_MAX cant help.

This should demonstrate that OverflowError will not disappear entirely
even in Python 3.

No one is denying that by now :)
Even if we were to get rid of all OverflowErrors resulting from integer
operations in Python, there are going to be some C extensions--including
some in the Python standard library--that will store 32-bit integers.
Modules such as array and struct will still raise it, and probably other
modules were a long integer sometimes doesn't make sense (socket, for
instance). itertools.count() should work for arbitrary integers, though.

Agreed. I was lookind at itertoolsmodule.c, and even for my very
limited C knowlegment, it does not seem to be too hard.
Anyway: http://bugs.python.org/issue1165
 
S

Steven D'Aprano

The __new__ method is where you implement such decisions. There is no
"tidier" way for one class "auto-convert" to another class, which is
what you asked for (it says so right up there ^). So, if you don't like
the answer, don't ask the question.

How am I supposed to know if I will like the answer until I ask the
question? If I knew the answer I wouldn't need to ask the question.

Anyway, don't be so cranky and defensive. I was just pointing out that
having to duplicate code between two different classes was untidy, not
that __new__ is untidy. I thought that maybe I could inherit from both
long and int, but Python doesn't like that:

TypeError: Error when calling the metaclass bases
multiple bases have instance lay-out conflict


The way to avoid duplicating methods (which is what you are asking for,
no?) while still allowing your classes to maintain their identities is
by using mixins:

[snip code]

I've never come to grips with mixins. It's probably high time I do.

Thanks for the example.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
474,432
Messages
2,571,680
Members
48,796
Latest member
Greg L.

Latest Threads

Top