Warnings killing my performance

Discussion in 'Python' started by Kylotan, Feb 6, 2004.

  1. Kylotan

    Kylotan Guest

    I have the following code:

    def IntToRandFloat(x):
    """Given a 32-bit integer, return a float in [-1.0, 1.0]"""
    x = int(x)
    x = int(x << 13) ^ x
    return (1.0-((x*(x*x*15731+789221)+1376312589)&0x7fffffff)/1073741824.0)


    Basically it's a function directly copied from a C implementation. Now
    it appears that the line with the left-shift causes the "losing bits
    or changing sign will return a long in Python 2.4 and up" warning. All
    well and good - I explicitly cast it back to an int so that the code
    won't break when I upgrade Python. I also disable the warning using
    warnings.filterwarnings(action = 'ignore', etc).

    However when the time comes to profile my app, I see lines like these:

    11266617 function calls in 488.717 CPU seconds
    ncalls tottime percall cumtime percall
    filename:lineno(function)
    3145728 129.744 0.000 364.845 0.000
    terraingen.py:22(IntToRandFloat)
    3142872 150.484 0.000 235.101 0.000 warnings.py:24(warn)
    3142872 84.617 0.000 84.617 0.000
    warnings.py:59(warn_explicit)

    Now I obviously can't afford to have almost half my program time
    consumed by warnings that I don't want to see.

    It gets stranger. If I change the shift line to this:
    x = int(x * (2 ** 13)) ^ x
    x = int(x | 0xFFFF)

    ....it's still calling 'warn' once for each time IntToRandFloat is
    called, but I see no warning appear, even when I don't import warnings
    and disable any. I have no other imports (that i can see) which might
    be disabling a warning behind the scenes.

    So I have these problems: warnings are slow (even when disabled),
    sometimes warnings are being issued and I never see them, and given
    that I never see the warnings I don't know how to get around them.

    So, my first question is to ask if there is a more efficient way of
    disabling warnings?

    And my second is, is there a quick way of taking an integer, shifting
    it left 13 bits (or multiplying by 2 ** 13, whatever), discarding any
    excess bits, and which won't cause a warning?

    Lastly, a suggestion; if 2.4 will introduce an automatic promotion to
    a long as a result of this shift operation, will the standard library
    provide a C implementation of the lossy shift operator for those of us
    that would benefit from a quick version? I'm guessing that promoting
    it to a long and then getting it back to a normal int is not exactly
    the speediest operation.

    --
    Ben Sizer
     
    Kylotan, Feb 6, 2004
    #1
    1. Advertising

  2. Ben> So I have these problems: warnings are slow (even when disabled),
    Ben> sometimes warnings are being issued and I never see them, and given
    Ben> that I never see the warnings I don't know how to get around them.

    Ben> So, my first question is to ask if there is a more efficient way of
    Ben> disabling warnings?

    This may seem silly, but switching to the CVS version of Python (aka 2.4a0)
    should help immensely, simply because that is gone:

    % python2.3 ~/local/bin/timeit.py -s 'import sys ; sys.path.append("/Users/skip/tmp"); from warn import IntToRandFloat' 'IntToRandFloat(789221)'
    /Users/skip/tmp/warn.py:4: FutureWarning: x<<y losing bits or changing sign will return a long in Python 2.4 and up
    x = int(x << 13) ^ x
    10000 loops, best of 3: 114 usec per loop
    % python2.4 ~/local/bin/timeit.py -s 'import sys ; sys.path.append("/Users/skip/tmp"); from warn import IntToRandFloat' 'IntToRandFloat(789221)'
    100000 loops, best of 3: 16.7 usec per loop

    Ben> Lastly, a suggestion; if 2.4 will introduce an automatic promotion
    Ben> to a long as a result of this shift operation, will the standard
    Ben> library provide a C implementation of the lossy shift operator for
    Ben> those of us that would benefit from a quick version? I'm guessing
    Ben> that promoting it to a long and then getting it back to a normal
    Ben> int is not exactly the speediest operation.

    Apparently it's quite a bit faster than navigating all the warning
    machinery. ;-)

    Skip
     
    Skip Montanaro, Feb 6, 2004
    #2
    1. Advertising

  3. Kylotan

    Kylotan Guest

    The 'or' in "x = int(x | 0xFFFF)" was obviously meant to be an 'and',
    before anybody chooses to point it out. :)

    --
    Kylotan
     
    Kylotan, Feb 6, 2004
    #3
  4. Kylotan

    Terry Reedy Guest

    "Kylotan" <> wrote in message
    news:...
    > And my second is, is there a quick way of taking an integer, shifting
    > it left 13 bits (or multiplying by 2 ** 13, whatever), discarding any
    > excess bits, and which won't cause a warning?


    Have you tried masking off the upper 13 bits *before* the shift? So that
    there is no overflow to warn about? Does this make any sense?

    TJR
     
    Terry Reedy, Feb 7, 2004
    #4
  5. Kylotan

    Kylotan Guest

    "Terry Reedy" <> wrote in message news:<>...
    > "Kylotan" <> wrote in message
    > news:...
    > > And my second is, is there a quick way of taking an integer, shifting
    > > it left 13 bits (or multiplying by 2 ** 13, whatever), discarding any
    > > excess bits, and which won't cause a warning?

    >
    > Have you tried masking off the upper 13 bits *before* the shift? So that
    > there is no overflow to warn about? Does this make any sense?


    Yes, it makes perfect sense... but unfortunately getting rid of that
    specific warning isn't enough. As noted in my first post, something is
    going through the warnings mechanism silently. I changed the code to
    this:

    def IntToRandFloat(x):
    """Given a 32-bit integer, return a float in [-1.0, 1.0]"""
    x = int(x) & 0x3FFFF # Preserve lowest 18 bits, to remove
    overflow.
    x = int(x << 13) ^ x # Shift left 13 bits.
    return (1.0-((x*(x*x*15731+789221)+1376312589)&0x7fffffff)/1073741824.0)

    And the profiler is still showing a call to warn() and warn_explicit()
    for each call to IntToRandFloat. I'm also having to preserve 18 bits
    instead of the desired 19 because the warning displays when I use 18,
    presumably because it includes the sign bit. However now I think I am
    damaging the distribution of the function (which is supposed to
    generate deterministic noise).

    I probably should have mentioned earlier that I'm using "Python 2.3.3
    (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32" for
    what it's worth.

    If I knew what warning it was, maybe I could turn it into an error to
    find out what's causing it. But right now, it's a silent warning that
    I can't do much about. And which is slowing down my code a lot!

    I'm starting to think that I might need to write a 1-function C++
    extension here, which would be a shame.

    --
    Ben Sizer
     
    Kylotan, Feb 7, 2004
    #5
  6. On Sat, Feb 07, 2004 at 04:45:12AM -0800, Kylotan wrote:
    > specific warning isn't enough. As noted in my first post, something is
    > going through the warnings mechanism silently. I changed the code to


    Have you tried "python -Wall"? It shows PendingDeprecationWarnings, which
    are usually not displayed.

    -Andrew.
     
    Andrew Bennetts, Feb 7, 2004
    #6
  7. Kylotan

    Peter Otten Guest

    Kylotan wrote:

    > I have the following code:
    >
    > def IntToRandFloat(x):
    > """Given a 32-bit integer, return a float in [-1.0, 1.0]"""
    > x = int(x)
    > x = int(x << 13) ^ x
    > return
    > (1.0-((x*(x*x*15731+789221)+1376312589)&0x7fffffff)/1073741824.0)
    >
    >
    > Basically it's a function directly copied from a C implementation. Now
    > it appears that the line with the left-shift causes the "losing bits
    > or changing sign will return a long in Python 2.4 and up" warning. All
    > well and good - I explicitly cast it back to an int so that the code
    > won't break when I upgrade Python. I also disable the warning using
    > warnings.filterwarnings(action = 'ignore', etc).
    >
    > However when the time comes to profile my app, I see lines like these:
    >
    > 11266617 function calls in 488.717 CPU seconds
    > ncalls tottime percall cumtime percall
    > filename:lineno(function)
    > 3145728 129.744 0.000 364.845 0.000
    > terraingen.py:22(IntToRandFloat)
    > 3142872 150.484 0.000 235.101 0.000 warnings.py:24(warn)
    > 3142872 84.617 0.000 84.617 0.000
    > warnings.py:59(warn_explicit)
    >
    > Now I obviously can't afford to have almost half my program time
    > consumed by warnings that I don't want to see.
    >
    > It gets stranger. If I change the shift line to this:
    > x = int(x * (2 ** 13)) ^ x
    > x = int(x | 0xFFFF)
    >
    > ...it's still calling 'warn' once for each time IntToRandFloat is
    > called, but I see no warning appear, even when I don't import warnings
    > and disable any. I have no other imports (that i can see) which might
    > be disabling a warning behind the scenes.
    >
    > So I have these problems: warnings are slow (even when disabled),
    > sometimes warnings are being issued and I never see them, and given
    > that I never see the warnings I don't know how to get around them.
    >
    > So, my first question is to ask if there is a more efficient way of
    > disabling warnings?


    Have you tried brute force?

    ....> timeit.py -s"import profwarn" "profwarn.IntToRandFloat(99)"
    10000 loops, best of 3: 27.3 usec per loop

    ....> timeit.py -s"import profwarn;profwarn.disableWarnings()"
    "profwarn.IntToRandFloat(99)"
    100000 loops, best of 3: 7.8 usec per loop

    with IntToRandFloat() copied* from your post and the following to entirely
    disable warnings:

    def disableWarnings():
    def _theevilunwarner(*args):
    pass
    import warnings
    warnings.warn = _theevilunwarner
    warnings.warn_explicit = _theevilunwarner

    I think the last line isn't necessary, but it won't do any (additional)
    harm.

    Peter

    (*) Your email client seems to replace normal space with evil lookalikes, so
    I had to delete and reinsert the entire whitespace.
     
    Peter Otten, Feb 7, 2004
    #7
  8. Kylotan

    Mel Wilson Guest

    In article <>,
    (Kylotan) wrote:
    >"Terry Reedy" <> wrote in message news:<>...
    >> "Kylotan" <> wrote in message
    >> news:...
    >> > And my second is, is there a quick way of taking an integer, shifting
    >> > it left 13 bits (or multiplying by 2 ** 13, whatever), discarding any
    >> > excess bits, and which won't cause a warning?

    >>
    >> Have you tried masking off the upper 13 bits *before* the shift? So that
    >> there is no overflow to warn about? Does this make any sense?

    >
    >Yes, it makes perfect sense... but unfortunately getting rid of that
    >specific warning isn't enough. As noted in my first post, something is
    >going through the warnings mechanism silently. I changed the code to
    >this:
    >
    >def IntToRandFloat(x):
    > """Given a 32-bit integer, return a float in [-1.0, 1.0]"""
    > x = int(x) & 0x3FFFF # Preserve lowest 18 bits, to remove
    >overflow.
    > x = int(x << 13) ^ x # Shift left 13 bits.
    > return (1.0-((x*(x*x*15731+789221)+1376312589)&0x7fffffff)/1073741824.0)
    >
    >And the profiler is still showing a call to warn() and warn_explicit()
    >for each call to IntToRandFloat. I'm also having to preserve 18 bits
    >instead of the desired 19 because the warning displays when I use 18,
    >presumably because it includes the sign bit. However now I think I am
    >damaging the distribution of the function (which is supposed to
    >generate deterministic noise).


    I notice that you're effectively cubing x, so you could
    only keep 10 bits of it and be sure of avoiding 32-bit
    overflow. Maybe a linear pseudo-random formula would save
    some trouble. But it appears that random.randint also
    generates invisible warnings.

    The warnings go away with
    x = long(x << 13) ^ x


    Regards. Mel.
     
    Mel Wilson, Feb 7, 2004
    #8
  9. Kylotan

    Kylotan Guest

    Peter Otten <> wrote in message news:<c02ubl$qqt$07$-online.com>...

    > def disableWarnings():
    > def _theevilunwarner(*args):
    > pass
    > import warnings
    > warnings.warn = _theevilunwarner
    > warnings.warn_explicit = _theevilunwarner


    Heh, that's great. However, I don't think it really scales well to a
    full application, does it? :)

    > (*) Your email client seems to replace normal space with evil lookalikes, so
    > I had to delete and reinsert the entire whitespace.


    I'm posting through Google Groups; my ISP doesn't carry more than the
    last 6 threads of comp.lang.python, or indeed many other newsgroups. I
    have no idea why.

    --
    Ben Sizer
     
    Kylotan, Feb 7, 2004
    #9
  10. Kylotan

    Kylotan Guest

    (Mel Wilson) wrote in message news:<cFQJAls/>...

    > I notice that you're effectively cubing x, so you could
    > only keep 10 bits of it and be sure of avoiding 32-bit
    > overflow.


    I'm not sure why you say that. If I feed '10' into the formula in
    question I'll get something much larger than 1000 out!

    > Maybe a linear pseudo-random formula would save
    > some trouble. But it appears that random.randint also
    > generates invisible warnings.


    Well, I changed the whole function to the following:

    rnd.seed(x)
    return rnd.uniform(-1.0, 1.0)

    Where rnd is an instance of Random(). This is a little quicker, and
    doesn't throw any warnings (visible or otherwise). I just have to hope
    the random.uniform implementation doesn't change any time soon.

    I would also hope that someone will find a way to remove that hidden
    warning from randint because it seems to be an unnecessary performance
    hit on what is otherwise a fairly fundamental function.

    > The warnings go away with
    > x = long(x << 13) ^ x


    So do the results I'm trying to get ;)

    --
    Ben Sizer
     
    Kylotan, Feb 7, 2004
    #10
  11. Kylotan

    Peter Otten Guest

    Kylotan wrote:

    > Peter Otten <> wrote in message
    > news:<c02ubl$qqt$07$-online.com>...
    >
    >> def disableWarnings():
    >> def _theevilunwarner(*args):
    >> pass
    >> import warnings
    >> warnings.warn = _theevilunwarner
    >> warnings.warn_explicit = _theevilunwarner

    >
    > Heh, that's great. However, I don't think it really scales well to a
    > full application, does it? :)


    Well, you could disable warnings in critical parts only, but that wouldn't
    play well with threads. Enter the not so evil unwarner:

    def disableWarnings():
    originalWarn = warnings.warn
    disabled = sets.Set([
    id(exceptions.OverflowWarning),
    id(exceptions.FutureWarning)])

    def _thenotsoevilunwarner(message, category=None, stacklevel=1):
    if id(category) in disabled:
    #print "DISMISS", message, category
    pass
    else:
    #print "PROPAGATE", message, category
    originalWarn(message, category, stacklevel)
    warnings.warn = _thenotsoevilunwarner

    It's still pretty fast (10.8 vs 27.1 usec on my machine), but the duplicate
    test will slow down warnings that are actually propagated.

    And if you are really ambitious, you could devise a patch to speed up warn()
    in the library, maybe using a similar technique as shown above.

    Peter
     
    Peter Otten, Feb 8, 2004
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. jm
    Replies:
    1
    Views:
    516
    alien2_51
    Dec 12, 2003
  2. Rob Meade

    Killing off my sessions

    Rob Meade, Feb 2, 2004, in forum: ASP .Net
    Replies:
    14
    Views:
    2,047
    Rob Meade
    Feb 4, 2004
  3. Michael Johnson Sr.

    Process killing

    Michael Johnson Sr., Feb 10, 2004, in forum: ASP .Net
    Replies:
    0
    Views:
    345
    Michael Johnson Sr.
    Feb 10, 2004
  4. Ted Sung
    Replies:
    1
    Views:
    320
    Sherm Pendley
    Aug 30, 2004
  5. Software Engineer
    Replies:
    0
    Views:
    338
    Software Engineer
    Jun 10, 2011
Loading...

Share This Page