Re: bit count or bit set && Python3

Discussion in 'Python' started by Steven D'Aprano, Oct 25, 2012.

  1. On Fri, 26 Oct 2012 02:31:53 +1100, Chris Angelico wrote:

    > On Fri, Oct 26, 2012 at 2:25 AM, Christian Heimes <>
    > wrote:
    >> Simple, easy, faster than a Python loop but not very elegant:
    >>
    >> bin(number).count("1")

    >
    > Unlikely to be fast.


    Oh I don't know about that. Here's some timing results using Python 2.7:

    py> from timeit import Timer
    py> t = Timer('bin(number).count("1")', setup='number=2**10001-1')
    py> min(t.repeat(number=10000, repeat=7))
    0.6819710731506348

    Compare to MRAB's suggestion:

    def count_set_bits(number):
    count = 0
    while number:
    count += 1
    number &= number - 1
    return count

    py> t = Timer('count_set_bits(number)',
    .... setup='from __main__ import count_set_bits; number=2**10001-1')
    py> min(t.repeat(number=100, repeat=7))
    4.141788959503174


    That makes the "inelegant" solution using bin() and count() about 600
    times faster than the mathematically clever solution using bitwise
    operations.

    On the other hand, I'm guessing that PyPy would speed up MRAB's version
    significantly.



    --
    Steven
    Steven D'Aprano, Oct 25, 2012
    #1
    1. Advertising

  2. Steven D'Aprano

    rusi Guest

    On Oct 25, 8:57 pm, Steven D'Aprano <steve
    > wrote:
    > On Fri, 26 Oct 2012 02:31:53 +1100, Chris Angelico wrote:
    > > On Fri, Oct 26, 2012 at 2:25 AM, Christian Heimes <>
    > > wrote:
    > >> Simple, easy, faster than a Python loop but not very elegant:

    >
    > >>    bin(number).count("1")

    >
    > > Unlikely to be fast.

    >
    > Oh I don't know about that. Here's some timing results using Python 2.7:
    >
    > py> from timeit import Timer
    > py> t = Timer('bin(number).count("1")', setup='number=2**10001-1')
    > py> min(t.repeat(number=10000, repeat=7))
    > 0.6819710731506348
    >
    > Compare to MRAB's suggestion:
    >
    > def count_set_bits(number):
    >      count = 0
    >      while number:
    >          count += 1
    >          number &= number - 1
    >      return count
    >
    > py> t = Timer('count_set_bits(number)',
    > ...     setup='from __main__ import count_set_bits; number=2**10001-1')
    > py> min(t.repeat(number=100, repeat=7))
    > 4.141788959503174
    >
    > That makes the "inelegant" solution using bin() and count() about 600
    > times faster than the mathematically clever solution using bitwise
    > operations.


    You meant 600% I think?
    rusi, Oct 25, 2012
    #2
    1. Advertising

  3. On Fri, Oct 26, 2012 at 3:17 AM, rusi <> wrote:
    > On Oct 25, 8:57 pm, Steven D'Aprano <steve
    > > wrote:
    >> py> min(t.repeat(number=10000, repeat=7))
    >> 0.6819710731506348
    >> py> min(t.repeat(number=100, repeat=7))
    >> 4.141788959503174
    >>
    >> That makes the "inelegant" solution using bin() and count() about 600
    >> times faster than the mathematically clever solution using bitwise
    >> operations.

    >
    > You meant 600% I think?


    It took six times longer to do one hundredth the iterations.

    ChrisA
    Chris Angelico, Oct 25, 2012
    #3
  4. Steven D'Aprano

    rusi Guest

    On Oct 25, 9:30 pm, Chris Angelico <> wrote:
    > On Fri, Oct 26, 2012 at 3:17 AM, rusi <> wrote:
    > > On Oct 25, 8:57 pm, Steven D'Aprano <steve
    > > > wrote:
    > >> py> min(t.repeat(number=10000, repeat=7))
    > >> 0.6819710731506348
    > >> py> min(t.repeat(number=100, repeat=7))
    > >> 4.141788959503174

    >
    > >> That makes the "inelegant" solution using bin() and count() about 600
    > >> times faster than the mathematically clever solution using bitwise
    > >> operations.

    >
    > > You meant 600% I think?

    >
    > It took six times longer to do one hundredth the iterations.
    >
    > ChrisA


    Oh! Missed the number=
    rusi, Oct 25, 2012
    #4
  5. On 25/10/2012 17:29, Chris Angelico wrote:
    > On Fri, Oct 26, 2012 at 3:17 AM, rusi <> wrote:
    >> On Oct 25, 8:57 pm, Steven D'Aprano <steve
    >> > wrote:
    >>> py> min(t.repeat(number=10000, repeat=7))
    >>> 0.6819710731506348
    >>> py> min(t.repeat(number=100, repeat=7))
    >>> 4.141788959503174
    >>>
    >>> That makes the "inelegant" solution using bin() and count() about 600
    >>> times faster than the mathematically clever solution using bitwise
    >>> operations.

    >>
    >> You meant 600% I think?

    >
    > It took six times longer to do one hundredth the iterations.
    >
    > ChrisA
    >


    Oh no, not another PEP 393 foul up :)

    --
    Cheers.

    Mark Lawrence.
    Mark Lawrence, Oct 25, 2012
    #5
  6. On Thu, 25 Oct 2012 09:17:40 -0700, rusi wrote:

    > On Oct 25, 8:57 pm, Steven D'Aprano <steve
    > > wrote:
    >> On Fri, 26 Oct 2012 02:31:53 +1100, Chris Angelico wrote:
    >> > On Fri, Oct 26, 2012 at 2:25 AM, Christian Heimes
    >> > <> wrote:
    >> >> Simple, easy, faster than a Python loop but not very elegant:

    >>
    >> >>    bin(number).count("1")

    >>
    >> > Unlikely to be fast.

    >>
    >> Oh I don't know about that. Here's some timing results using Python
    >> 2.7:
    >>
    >> py> from timeit import Timer
    >> py> t = Timer('bin(number).count("1")', setup='number=2**10001-1')
    >> py> min(t.repeat(number=10000, repeat=7))
    >> 0.6819710731506348
    >>
    >> Compare to MRAB's suggestion:
    >>
    >> def count_set_bits(number):
    >>      count = 0
    >>      while number:
    >>          count += 1
    >>          number &= number - 1
    >>      return count
    >>
    >> py> t = Timer('count_set_bits(number)',
    >> ...     setup='from __main__ import count_set_bits;
    >> ... number=2**10001-1')
    >> py> min(t.repeat(number=100, repeat=7))
    >> 4.141788959503174
    >>
    >> That makes the "inelegant" solution using bin() and count() about 600
    >> times faster than the mathematically clever solution using bitwise
    >> operations.

    >
    > You meant 600% I think?


    No, I mean a factor of 600 times faster. Look at the number of iterations
    used in each test: 10000 versus 100.

    Using bin() and count() takes 0.6819710731506348/10000 = 6.8e-5 seconds
    on my computer; using MRAB's neat trick takes 4.141788959503174/100 =
    0.041 seconds. 0.041/6.8e-5 is slightly over 600.


    --
    Steven
    Steven D'Aprano, Oct 25, 2012
    #6
  7. Chris Angelico's suggestion:

    >>> bitcount = bytes(bin(i).count("1") for i in range(256))
    >>> t = Timer('sum(number.to_bytes((number.bit_length() + 7) // 8,'

    .... '"little").translate(bitcount))',
    .... setup='from __main__ import bitcount; number=2**10001-1')
    >>> min(t.repeat(number=10000, repeat=7))
    Serhiy Storchaka, Oct 25, 2012
    #7
  8. Steven D'Aprano

    Neil Cerutti Guest

    On 2012-10-25, Steven D'Aprano <> wrote:
    > On Fri, 26 Oct 2012 02:31:53 +1100, Chris Angelico wrote:
    >> On Fri, Oct 26, 2012 at 2:25 AM, Christian Heimes
    >> <>
    >> wrote:
    >>> Simple, easy, faster than a Python loop but not very elegant:
    >>>
    >>> bin(number).count("1")

    >>
    >> Unlikely to be fast.

    >
    > Oh I don't know about that.


    Yes indeed! Python string operations are fast enough and its
    arithmetic slow enough that I no longer assume I can beat a neat
    lexicographical solution. Try defeating the following with
    arithmetic:

    def is_palindrom(n):
    s = str(n)
    return s = s[::-1]

    > Here's some timing results using Python 2.7:


    Excellent work.

    You can of course drop to C for arithmetic and likely triumph
    over Python strings. That's never been applicable for me, though.

    --
    Neil Cerutti
    Neil Cerutti, Oct 25, 2012
    #8
  9. Steven D'Aprano

    Neil Cerutti Guest

    On 2012-10-25, Neil Cerutti <> wrote:
    > Try defeating the following with arithmetic:
    >
    > def is_palindrom(n):
    > s = str(n)
    > return s = s[::-1]


    Sorry for the typos. It should've been:

    def is_palindrome(n):
    s = str(n)
    return s == s[::-1]

    --
    Neil Cerutti
    Neil Cerutti, Oct 25, 2012
    #9
  10. Steven D'Aprano

    Ian Kelly Guest

    On Thu, Oct 25, 2012 at 2:00 PM, Neil Cerutti <> wrote:
    > Yes indeed! Python string operations are fast enough and its
    > arithmetic slow enough that I no longer assume I can beat a neat
    > lexicographical solution. Try defeating the following with
    > arithmetic:
    >
    > def is_palindrom(n):
    > s = str(n)
    > return s = s[::-1]


    Problems like these are fundamentally string problems, not math
    problems. The question being asked isn't about some essential
    property of the number, but about its digital representation.
    Certainly they can be reasoned about mathematically, but the fact
    remains that the math being done is about the properties of strings.
    Ian Kelly, Oct 25, 2012
    #10
  11. On 25/10/2012 17:08, Charles Hixson wrote:
    > On 10/25/2012 08:57 AM, Steven D'Aprano wrote:
    >> On Fri, 26 Oct 2012 02:31:53 +1100, Chris Angelico wrote:
    >>
    >>> On Fri, Oct 26, 2012 at 2:25 AM, Christian Heimes<>
    >>> wrote:
    >>>> Simple, easy, faster than a Python loop but not very elegant:
    >>>>
    >>>> bin(number).count("1")
    >>> Unlikely to be fast.

    >> Oh I don't know about that. Here's some timing results using Python 2.7:
    >>
    >> py> from timeit import Timer
    >> py> t = Timer('bin(number).count("1")', setup='number=2**10001-1')
    >> py> min(t.repeat(number=10000, repeat=7))
    >> 0.6819710731506348
    >>
    >> Compare to MRAB's suggestion:
    >>
    >> def count_set_bits(number):
    >> count = 0
    >> while number:
    >> count += 1
    >> number&= number - 1
    >> return count
    >>
    >> py> t = Timer('count_set_bits(number)',
    >> ... setup='from __main__ import count_set_bits; number=2**10001-1')
    >> py> min(t.repeat(number=100, repeat=7))
    >> 4.141788959503174
    >>
    >>
    >> That makes the "inelegant" solution using bin() and count() about 600
    >> times faster than the mathematically clever solution using bitwise
    >> operations.
    >>
    >> On the other hand, I'm guessing that PyPy would speed up MRAB's version
    >> significantly.
    >>
    >>
    >>

    > Really nice and good to know. I had guessed the other way. (As you
    > point out this is compiler dependent, and I'll be using Python3,
    > but...conversion from an int to a bit string must be a *lot* faster than
    > I had thought.)
    >


    The simple rule for Python performance is never guess anything as you'll
    invariably be wrong, time it and/or profile it, then change your code if
    and only if you have to.

    --
    Cheers.

    Mark Lawrence.
    Mark Lawrence, Oct 25, 2012
    #11
  12. On Thu, 25 Oct 2012 14:20:00 -0600, Ian Kelly wrote:

    > On Thu, Oct 25, 2012 at 2:00 PM, Neil Cerutti <> wrote:
    >> Yes indeed! Python string operations are fast enough and its arithmetic
    >> slow enough that I no longer assume I can beat a neat lexicographical
    >> solution. Try defeating the following with arithmetic:
    >>
    >> def is_palindrom(n):
    >> s = str(n)
    >> return s = s[::-1]

    >
    > Problems like these are fundamentally string problems, not math
    > problems. The question being asked isn't about some essential property
    > of the number, but about its digital representation.


    Speaking as somebody who sometimes pretends to be a mathematician, I
    think you are wrong there. The property of being a palindrome may have
    been invented in reference to strings, but it is also a mathematical
    property of a number. Properties of the digits of numbers are properties
    of the relationship between the number and some sequence of divisors (the
    powers of some base), which makes them numeric properties.

    It's interesting to consider properties of numbers which hold for *every*
    base. For example, I understand that the digits of pi are uniformly
    distributed regardless of which base you use.

    Certainly mathematicians frequently ask questions about the digits of
    numbers, generally in base ten but also in other bases. Since the digits
    of a number are based on the numeric properties of the number, they
    certainly are essential to the number (modulo some base).

    For example, apart from two itself[1], every prime number ends in a 1 bit
    in base two. In decimal, every factorial greater than 4! ends with a
    zero. A number is divisible by 9 if the sum of the (decimal) digits
    reduces to 9. A number is divisible by 5 if the last digit is 0 or 5.
    These are not accidents, they depend on the numeric properties.


    > Certainly they can
    > be reasoned about mathematically, but the fact remains that the math
    > being done is about the properties of strings.


    Strings of digits, which are numerically equal to the remainders when you
    divide the number by successively larger powers of some base.:

    123 = 1*10**2 + 2*10**1 + 3*10**0

    Hence, mathematical.

    Of course, none of this challenges the fact that, at least in Python,
    reasoning about digits is often best done on the string representation
    rather than the number itself.



    [1] Which makes 2 the oddest prime of all.


    --
    Steven
    Steven D'Aprano, Oct 26, 2012
    #12
  13. Steven D'Aprano

    Neil Cerutti Guest

    On 2012-10-25, Ian Kelly <> wrote:
    > On Thu, Oct 25, 2012 at 2:00 PM, Neil Cerutti
    > <> wrote:
    >> Yes indeed! Python string operations are fast enough and its
    >> arithmetic slow enough that I no longer assume I can beat a
    >> neat lexicographical solution. Try defeating the following
    >> with arithmetic:
    >>
    >> def is_palindrom(n):
    >> s = str(n)
    >> return s = s[::-1]

    >
    > Problems like these are fundamentally string problems, not math
    > problems. The question being asked isn't about some essential
    > property of the number, but about its digital representation.
    > Certainly they can be reasoned about mathematically, but the
    > fact remains that the math being done is about the properties
    > of strings.


    The "unexpected" part, to me, is that an optimal arithmetic based
    solution conceptually is more efficient. You need to compute just
    half the digits of the number and then perform a contant compare
    operation.

    --
    Neil Cerutti
    Neil Cerutti, Oct 26, 2012
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Charles Hixson

    bit count or bit set && Python3

    Charles Hixson, Oct 25, 2012, in forum: Python
    Replies:
    5
    Views:
    200
    Charles Hixson
    Oct 26, 2012
  2. MRAB
    Replies:
    0
    Views:
    143
  3. Mark Lawrence

    Re: bit count or bit set && Python3

    Mark Lawrence, Oct 25, 2012, in forum: Python
    Replies:
    0
    Views:
    149
    Mark Lawrence
    Oct 25, 2012
  4. Charles Hixson

    bit count or bit set && Python3

    Charles Hixson, Oct 25, 2012, in forum: Python
    Replies:
    0
    Views:
    97
    Charles Hixson
    Oct 25, 2012
  5. Charles Hixson

    bit count or bit set && Python3

    Charles Hixson, Oct 25, 2012, in forum: Python
    Replies:
    0
    Views:
    97
    Charles Hixson
    Oct 25, 2012
Loading...

Share This Page