# Re: bit count or bit set && Python3

Discussion in 'Python' started by Steven D'Aprano, Oct 25, 2012.

1. ### Steven D'ApranoGuest

On Fri, 26 Oct 2012 02:31:53 +1100, Chris Angelico wrote:

> On Fri, Oct 26, 2012 at 2:25 AM, Christian Heimes <>
> wrote:
>> Simple, easy, faster than a Python loop but not very elegant:
>>
>> bin(number).count("1")

>
> Unlikely to be fast.

Oh I don't know about that. Here's some timing results using Python 2.7:

py> from timeit import Timer
py> t = Timer('bin(number).count("1")', setup='number=2**10001-1')
py> min(t.repeat(number=10000, repeat=7))
0.6819710731506348

Compare to MRAB's suggestion:

def count_set_bits(number):
count = 0
while number:
count += 1
number &= number - 1
return count

py> t = Timer('count_set_bits(number)',
.... setup='from __main__ import count_set_bits; number=2**10001-1')
py> min(t.repeat(number=100, repeat=7))
4.141788959503174

That makes the "inelegant" solution using bin() and count() about 600
times faster than the mathematically clever solution using bitwise
operations.

On the other hand, I'm guessing that PyPy would speed up MRAB's version
significantly.

--
Steven

Steven D'Aprano, Oct 25, 2012

2. ### rusiGuest

On Oct 25, 8:57 pm, Steven D'Aprano <steve
> wrote:
> On Fri, 26 Oct 2012 02:31:53 +1100, Chris Angelico wrote:
> > On Fri, Oct 26, 2012 at 2:25 AM, Christian Heimes <>
> > wrote:
> >> Simple, easy, faster than a Python loop but not very elegant:

>
> >>    bin(number).count("1")

>
> > Unlikely to be fast.

>
> Oh I don't know about that. Here's some timing results using Python 2.7:
>
> py> from timeit import Timer
> py> t = Timer('bin(number).count("1")', setup='number=2**10001-1')
> py> min(t.repeat(number=10000, repeat=7))
> 0.6819710731506348
>
> Compare to MRAB's suggestion:
>
> def count_set_bits(number):
>      count = 0
>      while number:
>          count += 1
>          number &= number - 1
>      return count
>
> py> t = Timer('count_set_bits(number)',
> ...     setup='from __main__ import count_set_bits; number=2**10001-1')
> py> min(t.repeat(number=100, repeat=7))
> 4.141788959503174
>
> That makes the "inelegant" solution using bin() and count() about 600
> times faster than the mathematically clever solution using bitwise
> operations.

You meant 600% I think?

rusi, Oct 25, 2012

3. ### Chris AngelicoGuest

On Fri, Oct 26, 2012 at 3:17 AM, rusi <> wrote:
> On Oct 25, 8:57 pm, Steven D'Aprano <steve
> > wrote:
>> py> min(t.repeat(number=10000, repeat=7))
>> 0.6819710731506348
>> py> min(t.repeat(number=100, repeat=7))
>> 4.141788959503174
>>
>> That makes the "inelegant" solution using bin() and count() about 600
>> times faster than the mathematically clever solution using bitwise
>> operations.

>
> You meant 600% I think?

It took six times longer to do one hundredth the iterations.

ChrisA

Chris Angelico, Oct 25, 2012
4. ### rusiGuest

On Oct 25, 9:30 pm, Chris Angelico <> wrote:
> On Fri, Oct 26, 2012 at 3:17 AM, rusi <> wrote:
> > On Oct 25, 8:57 pm, Steven D'Aprano <steve
> > > wrote:
> >> py> min(t.repeat(number=10000, repeat=7))
> >> 0.6819710731506348
> >> py> min(t.repeat(number=100, repeat=7))
> >> 4.141788959503174

>
> >> That makes the "inelegant" solution using bin() and count() about 600
> >> times faster than the mathematically clever solution using bitwise
> >> operations.

>
> > You meant 600% I think?

>
> It took six times longer to do one hundredth the iterations.
>
> ChrisA

Oh! Missed the number=

rusi, Oct 25, 2012
5. ### Mark LawrenceGuest

On 25/10/2012 17:29, Chris Angelico wrote:
> On Fri, Oct 26, 2012 at 3:17 AM, rusi <> wrote:
>> On Oct 25, 8:57 pm, Steven D'Aprano <steve
>> > wrote:
>>> py> min(t.repeat(number=10000, repeat=7))
>>> 0.6819710731506348
>>> py> min(t.repeat(number=100, repeat=7))
>>> 4.141788959503174
>>>
>>> That makes the "inelegant" solution using bin() and count() about 600
>>> times faster than the mathematically clever solution using bitwise
>>> operations.

>>
>> You meant 600% I think?

>
> It took six times longer to do one hundredth the iterations.
>
> ChrisA
>

Oh no, not another PEP 393 foul up

--
Cheers.

Mark Lawrence.

Mark Lawrence, Oct 25, 2012
6. ### Steven D'ApranoGuest

On Thu, 25 Oct 2012 09:17:40 -0700, rusi wrote:

> On Oct 25, 8:57Â pm, Steven D'Aprano <steve
> > wrote:
>> On Fri, 26 Oct 2012 02:31:53 +1100, Chris Angelico wrote:
>> > On Fri, Oct 26, 2012 at 2:25 AM, Christian Heimes
>> > <> wrote:
>> >> Simple, easy, faster than a Python loop but not very elegant:

>>
>> >> Â  Â bin(number).count("1")

>>
>> > Unlikely to be fast.

>>
>> Oh I don't know about that. Here's some timing results using Python
>> 2.7:
>>
>> py> from timeit import Timer
>> py> t = Timer('bin(number).count("1")', setup='number=2**10001-1')
>> py> min(t.repeat(number=10000, repeat=7))
>> 0.6819710731506348
>>
>> Compare to MRAB's suggestion:
>>
>> def count_set_bits(number):
>> Â  Â  Â count = 0
>> Â  Â  Â while number:
>> Â  Â  Â  Â  Â count += 1
>> Â  Â  Â  Â  Â number &= number - 1
>> Â  Â  Â return count
>>
>> py> t = Timer('count_set_bits(number)',
>> ... Â  Â  setup='from __main__ import count_set_bits;
>> ... number=2**10001-1')
>> py> min(t.repeat(number=100, repeat=7))
>> 4.141788959503174
>>
>> That makes the "inelegant" solution using bin() and count() about 600
>> times faster than the mathematically clever solution using bitwise
>> operations.

>
> You meant 600% I think?

No, I mean a factor of 600 times faster. Look at the number of iterations
used in each test: 10000 versus 100.

Using bin() and count() takes 0.6819710731506348/10000 = 6.8e-5 seconds
on my computer; using MRAB's neat trick takes 4.141788959503174/100 =
0.041 seconds. 0.041/6.8e-5 is slightly over 600.

--
Steven

Steven D'Aprano, Oct 25, 2012
7. ### Serhiy StorchakaGuest

Chris Angelico's suggestion:

>>> bitcount = bytes(bin(i).count("1") for i in range(256))
>>> t = Timer('sum(number.to_bytes((number.bit_length() + 7) // 8,'

.... '"little").translate(bitcount))',
.... setup='from __main__ import bitcount; number=2**10001-1')
>>> min(t.repeat(number=10000, repeat=7))

Serhiy Storchaka, Oct 25, 2012
8. ### Neil CeruttiGuest

On 2012-10-25, Steven D'Aprano <> wrote:
> On Fri, 26 Oct 2012 02:31:53 +1100, Chris Angelico wrote:
>> On Fri, Oct 26, 2012 at 2:25 AM, Christian Heimes
>> <>
>> wrote:
>>> Simple, easy, faster than a Python loop but not very elegant:
>>>
>>> bin(number).count("1")

>>
>> Unlikely to be fast.

>
> Oh I don't know about that.

Yes indeed! Python string operations are fast enough and its
arithmetic slow enough that I no longer assume I can beat a neat
lexicographical solution. Try defeating the following with
arithmetic:

def is_palindrom(n):
s = str(n)
return s = s[::-1]

> Here's some timing results using Python 2.7:

Excellent work.

You can of course drop to C for arithmetic and likely triumph
over Python strings. That's never been applicable for me, though.

--
Neil Cerutti

Neil Cerutti, Oct 25, 2012
9. ### Neil CeruttiGuest

On 2012-10-25, Neil Cerutti <> wrote:
> Try defeating the following with arithmetic:
>
> def is_palindrom(n):
> s = str(n)
> return s = s[::-1]

Sorry for the typos. It should've been:

def is_palindrome(n):
s = str(n)
return s == s[::-1]

--
Neil Cerutti

Neil Cerutti, Oct 25, 2012
10. ### Ian KellyGuest

On Thu, Oct 25, 2012 at 2:00 PM, Neil Cerutti <> wrote:
> Yes indeed! Python string operations are fast enough and its
> arithmetic slow enough that I no longer assume I can beat a neat
> lexicographical solution. Try defeating the following with
> arithmetic:
>
> def is_palindrom(n):
> s = str(n)
> return s = s[::-1]

Problems like these are fundamentally string problems, not math
property of the number, but about its digital representation.
Certainly they can be reasoned about mathematically, but the fact
remains that the math being done is about the properties of strings.

Ian Kelly, Oct 25, 2012
11. ### Mark LawrenceGuest

On 25/10/2012 17:08, Charles Hixson wrote:
> On 10/25/2012 08:57 AM, Steven D'Aprano wrote:
>> On Fri, 26 Oct 2012 02:31:53 +1100, Chris Angelico wrote:
>>
>>> On Fri, Oct 26, 2012 at 2:25 AM, Christian Heimes<>
>>> wrote:
>>>> Simple, easy, faster than a Python loop but not very elegant:
>>>>
>>>> bin(number).count("1")
>>> Unlikely to be fast.

>> Oh I don't know about that. Here's some timing results using Python 2.7:
>>
>> py> from timeit import Timer
>> py> t = Timer('bin(number).count("1")', setup='number=2**10001-1')
>> py> min(t.repeat(number=10000, repeat=7))
>> 0.6819710731506348
>>
>> Compare to MRAB's suggestion:
>>
>> def count_set_bits(number):
>> count = 0
>> while number:
>> count += 1
>> number&= number - 1
>> return count
>>
>> py> t = Timer('count_set_bits(number)',
>> ... setup='from __main__ import count_set_bits; number=2**10001-1')
>> py> min(t.repeat(number=100, repeat=7))
>> 4.141788959503174
>>
>>
>> That makes the "inelegant" solution using bin() and count() about 600
>> times faster than the mathematically clever solution using bitwise
>> operations.
>>
>> On the other hand, I'm guessing that PyPy would speed up MRAB's version
>> significantly.
>>
>>
>>

> Really nice and good to know. I had guessed the other way. (As you
> point out this is compiler dependent, and I'll be using Python3,
> but...conversion from an int to a bit string must be a *lot* faster than
>

The simple rule for Python performance is never guess anything as you'll
invariably be wrong, time it and/or profile it, then change your code if
and only if you have to.

--
Cheers.

Mark Lawrence.

Mark Lawrence, Oct 25, 2012
12. ### Steven D'ApranoGuest

On Thu, 25 Oct 2012 14:20:00 -0600, Ian Kelly wrote:

> On Thu, Oct 25, 2012 at 2:00 PM, Neil Cerutti <> wrote:
>> Yes indeed! Python string operations are fast enough and its arithmetic
>> slow enough that I no longer assume I can beat a neat lexicographical
>> solution. Try defeating the following with arithmetic:
>>
>> def is_palindrom(n):
>> s = str(n)
>> return s = s[::-1]

>
> Problems like these are fundamentally string problems, not math
> of the number, but about its digital representation.

Speaking as somebody who sometimes pretends to be a mathematician, I
think you are wrong there. The property of being a palindrome may have
been invented in reference to strings, but it is also a mathematical
property of a number. Properties of the digits of numbers are properties
of the relationship between the number and some sequence of divisors (the
powers of some base), which makes them numeric properties.

It's interesting to consider properties of numbers which hold for *every*
base. For example, I understand that the digits of pi are uniformly
distributed regardless of which base you use.

numbers, generally in base ten but also in other bases. Since the digits
of a number are based on the numeric properties of the number, they
certainly are essential to the number (modulo some base).

For example, apart from two itself[1], every prime number ends in a 1 bit
in base two. In decimal, every factorial greater than 4! ends with a
zero. A number is divisible by 9 if the sum of the (decimal) digits
reduces to 9. A number is divisible by 5 if the last digit is 0 or 5.
These are not accidents, they depend on the numeric properties.

> Certainly they can
> be reasoned about mathematically, but the fact remains that the math
> being done is about the properties of strings.

Strings of digits, which are numerically equal to the remainders when you
divide the number by successively larger powers of some base.:

123 = 1*10**2 + 2*10**1 + 3*10**0

Hence, mathematical.

Of course, none of this challenges the fact that, at least in Python,
reasoning about digits is often best done on the string representation
rather than the number itself.

[1] Which makes 2 the oddest prime of all.

--
Steven

Steven D'Aprano, Oct 26, 2012
13. ### Neil CeruttiGuest

On 2012-10-25, Ian Kelly <> wrote:
> On Thu, Oct 25, 2012 at 2:00 PM, Neil Cerutti
> <> wrote:
>> Yes indeed! Python string operations are fast enough and its
>> arithmetic slow enough that I no longer assume I can beat a
>> neat lexicographical solution. Try defeating the following
>> with arithmetic:
>>
>> def is_palindrom(n):
>> s = str(n)
>> return s = s[::-1]

>
> Problems like these are fundamentally string problems, not math
> property of the number, but about its digital representation.
> Certainly they can be reasoned about mathematically, but the
> fact remains that the math being done is about the properties
> of strings.

The "unexpected" part, to me, is that an optimal arithmetic based
solution conceptually is more efficient. You need to compute just
half the digits of the number and then perform a contant compare
operation.

--
Neil Cerutti

Neil Cerutti, Oct 26, 2012