Warnings killing my performance

[Kylotan]

I have the following code:

def IntToRandFloat(x):
"""Given a 32-bit integer, return a float in [-1.0, 1.0]"""
x = int(x)
x = int(x << 13) ^ x
return (1.0-((x*(x*x*15731+789221)+1376312589)&0x7fffffff)/1073741824.0)


Basically it's a function directly copied from a C implementation. Now
it appears that the line with the left-shift causes the "losing bits
or changing sign will return a long in Python 2.4 and up" warning. All
well and good - I explicitly cast it back to an int so that the code
won't break when I upgrade Python. I also disable the warning using
warnings.filterwarnings(action = 'ignore', etc).

However when the time comes to profile my app, I see lines like these:

11266617 function calls in 488.717 CPU seconds
ncalls tottime percall cumtime percall
filename:lineno(function)
3145728 129.744 0.000 364.845 0.000
terraingen.py:22(IntToRandFloat)
3142872 150.484 0.000 235.101 0.000 warnings.py:24(warn)
3142872 84.617 0.000 84.617 0.000
warnings.py:59(warn_explicit)

Now I obviously can't afford to have almost half my program time
consumed by warnings that I don't want to see.

It gets stranger. If I change the shift line to this:
x = int(x * (2 ** 13)) ^ x
x = int(x | 0xFFFF)

....it's still calling 'warn' once for each time IntToRandFloat is
called, but I see no warning appear, even when I don't import warnings
and disable any. I have no other imports (that i can see) which might
be disabling a warning behind the scenes.

So I have these problems: warnings are slow (even when disabled),
sometimes warnings are being issued and I never see them, and given
that I never see the warnings I don't know how to get around them.

So, my first question is to ask if there is a more efficient way of
disabling warnings?

And my second is, is there a quick way of taking an integer, shifting
it left 13 bits (or multiplying by 2 ** 13, whatever), discarding any
excess bits, and which won't cause a warning?

Lastly, a suggestion; if 2.4 will introduce an automatic promotion to
a long as a result of this shift operation, will the standard library
provide a C implementation of the lossy shift operator for those of us
that would benefit from a quick version? I'm guessing that promoting
it to a long and then getting it back to a normal int is not exactly
the speediest operation.

 

[Skip Montanaro]

Ben> So I have these problems: warnings are slow (even when disabled),
Ben> sometimes warnings are being issued and I never see them, and given
Ben> that I never see the warnings I don't know how to get around them.

Ben> So, my first question is to ask if there is a more efficient way of
Ben> disabling warnings?

This may seem silly, but switching to the CVS version of Python (aka 2.4a0)
should help immensely, simply because that is gone:

% python2.3 ~/local/bin/timeit.py -s 'import sys ; sys.path.append("/Users/skip/tmp"); from warn import IntToRandFloat' 'IntToRandFloat(789221)'
/Users/skip/tmp/warn.py:4: FutureWarning: x<<y losing bits or changing sign will return a long in Python 2.4 and up
x = int(x << 13) ^ x
10000 loops, best of 3: 114 usec per loop
% python2.4 ~/local/bin/timeit.py -s 'import sys ; sys.path.append("/Users/skip/tmp"); from warn import IntToRandFloat' 'IntToRandFloat(789221)'
100000 loops, best of 3: 16.7 usec per loop

Ben> Lastly, a suggestion; if 2.4 will introduce an automatic promotion
Ben> to a long as a result of this shift operation, will the standard
Ben> library provide a C implementation of the lossy shift operator for
Ben> those of us that would benefit from a quick version? I'm guessing
Ben> that promoting it to a long and then getting it back to a normal
Ben> int is not exactly the speediest operation.

Apparently it's quite a bit faster than navigating all the warning
machinery. ;-)

Skip

 

[Kylotan]

The 'or' in "x = int(x | 0xFFFF)" was obviously meant to be an 'and',
before anybody chooses to point it out.

 

[Terry Reedy]

And my second is, is there a quick way of taking an integer, shifting
it left 13 bits (or multiplying by 2 ** 13, whatever), discarding any
excess bits, and which won't cause a warning?

Have you tried masking off the upper 13 bits *before* the shift? So that
there is no overflow to warn about? Does this make any sense?

TJR

 

[Kylotan]

Have you tried masking off the upper 13 bits *before* the shift? So that
there is no overflow to warn about? Does this make any sense?

Yes, it makes perfect sense... but unfortunately getting rid of that
specific warning isn't enough. As noted in my first post, something is
going through the warnings mechanism silently. I changed the code to
this:

def IntToRandFloat(x):
"""Given a 32-bit integer, return a float in [-1.0, 1.0]"""
x = int(x) & 0x3FFFF # Preserve lowest 18 bits, to remove
overflow.
x = int(x << 13) ^ x # Shift left 13 bits.
return (1.0-((x*(x*x*15731+789221)+1376312589)&0x7fffffff)/1073741824.0)

And the profiler is still showing a call to warn() and warn_explicit()
for each call to IntToRandFloat. I'm also having to preserve 18 bits
instead of the desired 19 because the warning displays when I use 18,
presumably because it includes the sign bit. However now I think I am
damaging the distribution of the function (which is supposed to
generate deterministic noise).

I probably should have mentioned earlier that I'm using "Python 2.3.3
(#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32" for
what it's worth.

If I knew what warning it was, maybe I could turn it into an error to
find out what's causing it. But right now, it's a silent warning that
I can't do much about. And which is slowing down my code a lot!

I'm starting to think that I might need to write a 1-function C++
extension here, which would be a shame.

 

[Andrew Bennetts]

specific warning isn't enough. As noted in my first post, something is
going through the warnings mechanism silently. I changed the code to

Have you tried "python -Wall"? It shows PendingDeprecationWarnings, which
are usually not displayed.

-Andrew.

 

[Peter Otten]

I have the following code:

def IntToRandFloat(x):
"""Given a 32-bit integer, return a float in [-1.0, 1.0]"""
x = int(x)
x = int(x << 13) ^ x
return
(1.0-((x*(x*x*15731+789221)+1376312589)&0x7fffffff)/1073741824.0)


Basically it's a function directly copied from a C implementation. Now
it appears that the line with the left-shift causes the "losing bits
or changing sign will return a long in Python 2.4 and up" warning. All
well and good - I explicitly cast it back to an int so that the code
won't break when I upgrade Python. I also disable the warning using
warnings.filterwarnings(action = 'ignore', etc).

However when the time comes to profile my app, I see lines like these:

11266617 function calls in 488.717 CPU seconds
ncalls tottime percall cumtime percall
filename:lineno(function)
3145728 129.744 0.000 364.845 0.000
terraingen.py:22(IntToRandFloat)
3142872 150.484 0.000 235.101 0.000 warnings.py:24(warn)
3142872 84.617 0.000 84.617 0.000
warnings.py:59(warn_explicit)

Now I obviously can't afford to have almost half my program time
consumed by warnings that I don't want to see.

It gets stranger. If I change the shift line to this:
x = int(x * (2 ** 13)) ^ x
x = int(x | 0xFFFF)

...it's still calling 'warn' once for each time IntToRandFloat is
called, but I see no warning appear, even when I don't import warnings
and disable any. I have no other imports (that i can see) which might
be disabling a warning behind the scenes.

So I have these problems: warnings are slow (even when disabled),
sometimes warnings are being issued and I never see them, and given
that I never see the warnings I don't know how to get around them.

So, my first question is to ask if there is a more efficient way of
disabling warnings?

Have you tried brute force?

....> timeit.py -s"import profwarn" "profwarn.IntToRandFloat(99)"
10000 loops, best of 3: 27.3 usec per loop

....> timeit.py -s"import profwarn;profwarn.disableWarnings()"
"profwarn.IntToRandFloat(99)"
100000 loops, best of 3: 7.8 usec per loop

with IntToRandFloat() copied* from your post and the following to entirely
disable warnings:

def disableWarnings():
def _theevilunwarner(*args):
pass
import warnings
warnings.warn = _theevilunwarner
warnings.warn_explicit = _theevilunwarner

I think the last line isn't necessary, but it won't do any (additional)
harm.

Peter

(*) Your email client seems to replace normal space with evil lookalikes, so
I had to delete and reinsert the entire whitespace.

 

[Mel Wilson]

Have you tried masking off the upper 13 bits *before* the shift? So that
there is no overflow to warn about? Does this make any sense?

Yes, it makes perfect sense... but unfortunately getting rid of that
specific warning isn't enough. As noted in my first post, something is
going through the warnings mechanism silently. I changed the code to
this:

def IntToRandFloat(x):
"""Given a 32-bit integer, return a float in [-1.0, 1.0]"""
x = int(x) & 0x3FFFF # Preserve lowest 18 bits, to remove
overflow.
x = int(x << 13) ^ x # Shift left 13 bits.
return (1.0-((x*(x*x*15731+789221)+1376312589)&0x7fffffff)/1073741824.0)

And the profiler is still showing a call to warn() and warn_explicit()
for each call to IntToRandFloat. I'm also having to preserve 18 bits
instead of the desired 19 because the warning displays when I use 18,
presumably because it includes the sign bit. However now I think I am
damaging the distribution of the function (which is supposed to
generate deterministic noise).

I notice that you're effectively cubing x, so you could
only keep 10 bits of it and be sure of avoiding 32-bit
overflow. Maybe a linear pseudo-random formula would save
some trouble. But it appears that random.randint also
generates invisible warnings.

The warnings go away with
x = long(x << 13) ^ x


Regards. Mel.

 

[Kylotan]

def disableWarnings():
def _theevilunwarner(*args):
pass
import warnings
warnings.warn = _theevilunwarner
warnings.warn_explicit = _theevilunwarner

Heh, that's great. However, I don't think it really scales well to a
full application, does it?

(*) Your email client seems to replace normal space with evil lookalikes, so
I had to delete and reinsert the entire whitespace.

I'm posting through Google Groups; my ISP doesn't carry more than the
last 6 threads of comp.lang.python, or indeed many other newsgroups. I
have no idea why.

 

[Kylotan]

I notice that you're effectively cubing x, so you could
only keep 10 bits of it and be sure of avoiding 32-bit
overflow.

I'm not sure why you say that. If I feed '10' into the formula in
question I'll get something much larger than 1000 out!

Maybe a linear pseudo-random formula would save
some trouble. But it appears that random.randint also
generates invisible warnings.

Well, I changed the whole function to the following:

rnd.seed(x)
return rnd.uniform(-1.0, 1.0)

Where rnd is an instance of Random(). This is a little quicker, and
doesn't throw any warnings (visible or otherwise). I just have to hope
the random.uniform implementation doesn't change any time soon.

I would also hope that someone will find a way to remove that hidden
warning from randint because it seems to be an unnecessary performance
hit on what is otherwise a fairly fundamental function.

The warnings go away with
x = long(x << 13) ^ x

So do the results I'm trying to get

 

[Peter Otten]

Heh, that's great. However, I don't think it really scales well to a
full application, does it?

Well, you could disable warnings in critical parts only, but that wouldn't
play well with threads. Enter the not so evil unwarner:

def disableWarnings():
originalWarn = warnings.warn
disabled = sets.Set([
id(exceptions.OverflowWarning),
id(exceptions.FutureWarning)])

def _thenotsoevilunwarner(message, category=None, stacklevel=1):
if id(category) in disabled:
#print "DISMISS", message, category
pass
else:
#print "PROPAGATE", message, category
originalWarn(message, category, stacklevel)
warnings.warn = _thenotsoevilunwarner

It's still pretty fast (10.8 vs 27.1 usec on my machine), but the duplicate
test will slow down warnings that are actually propagated.

And if you are really ambitious, you could devise a patch to speed up warn()
in the library, maybe using a similar technique as shown above.

Peter