all() is slow?

[OKB (not okblacke)]

I noticed this (Python 2.6.5 on Windows XP):
.... for a in x:
.... if a not in (True, False):
.... return False
.... return True

x = [random.choice([True, False]) for a in xrange(0, 5000000)]
timeit.timeit('myAll(x)', 'from __main__ import myAll, x',

number=10)
0: 9.7685158309226452import x', number=10)
1: 12.348196768024984

x = [random.randint(0,100) for a in xrange(0, 5000000)]
def myAll(x):

.... for a in x:
.... if not a <= 100:
.... return False
.... return Truenumber=10)
4: 2.8248207523582209__main__ import x', number=10)
5: 4.6433557896324942

What is the point of the all() function being a builtin if it's
slower than writing a function to do the check myself?

--
--OKB (not okblacke)
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is
no path, and leave a trail."
--author unknown

 

[Chris Rebert]

On Mon, Nov 7, 2011 at 1:00 PM, OKB (not okblacke)

       What is the point of the all() function being a builtin if it's
slower than writing a function to do the check myself?

Regardless of whether it's slower (which I expect someone will be
along to debunk or explain shortly), do you really want to have to
write an additional boilerplate function or block of code /every
single time/ you want to do such a check? The runtime speed difference
is unlikely to be worth your time as a developer in many cases. And by
Murphy's Law, you *will* make errors writing these repetitive code
blocks (e.g. forget to negate the conditional), whereas reusing all()
makes that much less likely.

The trade-off is run-time speed for developer
productivity/convenience; Python tends to lean towards the latter (to
varying degrees).

Cheers,
Chris

 

[david vierra]

        What is the point of the all() function being a builtin if it's
slower than writing a function to do the check myself?

But, you didn't write an all() function. You wrote a more specialized
allBoolean() function. I think this comparison is more fair to the
builtin all():
.... for a in x:
.... if not a: return False
.... return True
....12.209779342432576

 

[Chris Angelico]

But, you didn't write an all() function.  You wrote a more specialized
allBoolean() function. I think this comparison is more fair to the
builtin all():

So really, it's not "all() is slow" but "function calls are slow".
Maybe it'd be worthwhile making an all-factory:

def my_all(code,lst):
exec("""def tmp_all(x):
for a in x:
if not ("""+code+"""): return False
return True
""")
return tmp_all(lst)
timeit.timeit('my_all("a in (True, False)",x)','from __main__ import
my_all,x',number=10)

Bad code imho, but it _is_ faster than both the original and the builtin.

ChrisA

 

[Henrik Faber]

So really, it's not "all() is slow" but "function calls are slow".
Maybe it'd be worthwhile making an all-factory:

PLEASE say you're joking. If I saw code like that on any of our project,
this would definitely qualify for a DailyWTF.

Regards, Henrik

 

[Chris Angelico]

PLEASE say you're joking. If I saw code like that on any of our project,
this would definitely qualify for a DailyWTF.

For the benefit of anyone who was actually in doubt: YES, I was
joking. Do *not* put code like this into any production project.

But it's still fun to write bad code once in a while. It's like
Mythbusters getting to crash cars. Fun partly _because_ it's something
you normally don't want to do.

ChrisA

 

[John Posner]

For the benefit of anyone who was actually in doubt: YES, I was
joking. Do *not* put code like this into any production project.

Because an all-factory would produce code smell?

 

[Tim Chase]

Because an all-factory would produce code smell?

Groan...that word-play is the pits!

-tkc

 

[Devin Jeanpierre]

Clearly what we need is a modern hygienic macro system to avoid this exec mess!

defmacro defall(name, cond):
def name(lst):
for a in lst:
if not cond:
return False
return True

invoke defall(all, cond)

Only slightly tongue in cheek.

We have that stupid exec trick in the Python stdlib. It has to die.

http://hg.python.org/cpython/file/6bf07db23445/Lib/collections/__init__.py#l240

Devin

 

[Chris Angelico]

Clearly what we need is a modern hygienic macro system to avoid this execmess!

defmacro defall(name, cond):
   def name(lst):
       for a in lst:
           if not cond:
               return False
       return True

#define defall(name,cond) def name(lst): \
for a in lst: \
if not cond: return False \
return True

gcc -E myprog.pyi -o myprog.py

There's no code you can't make more opaque using the C preprocessor.

Python doesn't have inline functions? Ha! In your FACE, evil Python
development cabal! You can't tell ME what I can't do!

ChrisA
PS. Don't try this at home. We have years of experience with bad code.
Don't write code like this unless you have a life insurance policy
that covers angry mobs.

 

[Chris Rebert]

#define defall(name,cond) def name(lst): \
   for a in lst: \
       if not cond: return False \
   return True

gcc -E myprog.pyi -o myprog.py

There's no code you can't make more opaque using the C preprocessor.

Python doesn't have inline functions? Ha! In your FACE, evil Python
development cabal! You can't tell ME what I can't do!

ChrisA
PS. Don't try this at home. We have years of experience with bad code.
Don't write code like this unless you have a life insurance policy
that covers angry mobs.

Burn him! Witch! Witch! Burn him!
His code turned me into a newt!

 

[Steven D'Aprano]

We have that stupid exec trick in the Python stdlib. It has to die.

http://hg.python.org/cpython/file/6bf07db23445/Lib/collections/__init__.py#l240

http://bugs.python.org/issue3974
http://blog.ccpgames.com/kristjan/2011/05/28/namedtuple-and-exec/

If it were someone other than Raymond Hettinger responsible for the use
of exec in namedtuple, I'd be a lot more suspicious of it.

 

[Hans Mulder]

Burn him! Witch! Witch! Burn him!
His code turned me into a newt!

Is that a European swallow or an African swallow?

-- HansM

 

[Dennis Lee Bieber]

Feeling perverse..

Is that a European swallow or an African swallow?

Both... International coconut carrying rules require the swallows to
hand over messages at the Strait of Gibraltar...

 

[John Nagle]

I noticed this (Python 2.6.5 on Windows XP):

CPython is slow. It's a naive interpreter. There's
almost no optimization during compilation. Try PyPy
or Shed Skin.

John Nagle

 

[Devin Jeanpierre]

If it were someone other than Raymond Hettinger responsible for the use

of exec in namedtuple, I'd be a lot more suspicious of it.

I'm not going to be less suspicious based on a name. It reads like
insanity, and the justification was terrible.

Devin

 

[Steven D'Aprano]

I'm not going to be less suspicious based on a name.

Neither am I. I am less suspicious based on a reputation. Raymond is a
well-known, trusted senior Python developer who knows what he is doing.

It reads like
insanity, and the justification was terrible.

It reads fine, and the justification is perfectly valid.

You're right to be cautious of exec. You're wrong to be phobic about it.
What do you think is going to happen? The exec call inside namedtuple is
going to creep out of the module in the wee hours of the night,
contaminating other functions and modules while you sleep? Be serious. If
you have an actual concrete security vulnerability caused by the use of
exec inside namedtuple, or some other bug, then say so. Otherwise, your
paranoia is unjustified.

 

[Chris Angelico]

I'm not going to be less suspicious based on a name. It reads like
insanity, and the justification was terrible.

It's said that code exists foremost for humans to read, and only
incidentally for computers to execute. I believe that this is inverted
for library code; as long as the _interface_ is clean, you can get
away with some messy internals, because it's going to be executed far
more often than actually read. Python programmers can use namedtuples
happily without knowing that the implementation uses exec.

The justification is, if I understand correctly, that the alternative
is worse. That's plenty of justification imho.

ChrisA

 

[Devin Jeanpierre]

Neither am I. I am less suspicious based on a reputation. Raymond is a

well-known, trusted senior Python developer who knows what he is doing.

I don't really know anything about him or why people respect him, so I
have no reason to share your faith.

It reads fine, and the justification is perfectly valid.

Well. It reads fine in a certain sense, in that I can figure out
what's going on (although I have some troubles figuring out why the
heck certain things are in the code). The issue is that what's going
on is otherworldly: this is not a Python pattern, this is not a normal
approach. To me, that means it does not read fine.

The use of exec also results in (seemingly) arbitrary constraints on
the input. Like, why can't "--" be a name? Because exec? Is there some
other reason?

I don't like the use of exec, and I don't like the justification (it
seems handwavy). I pointed this out in a thread full of people saying
"never EVER use exec this way", so it's obviously not just me that
thinks this is awful.

You're right to be cautious of exec. You're wrong to be phobic about it.
What do you think is going to happen?

I think somebody will read it and think this is a good idea.

Devin

 

[Devin Jeanpierre]

Well. It reads fine in a certain sense, in that I can figure out

what's going on (although I have some troubles figuring out why the
heck certain things are in the code). The issue is that what's going
on is otherworldly: this is not a Python pattern, this is not a normal
approach. To me, that means it does not read fine.

Sorry for the double post, but I think I damaged my statement a
little in the edit process here. I have a hard time
reading the code, and the implications of using exec on how
initialization works is not obvious. I do believe that this solution
is significantly less readable than it would be in a language with
"real" macros, or using an alternate dict-based approach. I don't mean
to just say "it's different, therefore it's wrong" -- it takes more
than just being unusual (not that the unusualness helps).

Devin