Should I use "if" or "try" (as a matter of speed)?

[Steve Juranich]

I know that this topic has the potential for blowing up in my face,
but I can't help asking. I've been using Python since 1.5.1, so I'm
not what you'd call a "n00b". I dutifully evangelize on the goodness
of Python whenever I talk with fellow developers, but I always hit a
snag when it comes to discussing the finer points of the execution
model (specifically, exceptions).

Without fail, when I start talking with some of the "old-timers"
(people who have written code in ADA or Fortran), I hear the same
arguments that using "if" is "better" than using "try". I think that
the argument goes something like, "When you set up a 'try' block, you
have to set up a lot of extra machinery than is necessary just
executing a simple conditional."

I was wondering how true this holds for Python, where exceptions are
such an integral part of the execution model. It seems to me, that if
I'm executing a loop over a bunch of items, and I expect some
condition to hold for a majority of the cases, then a "try" block
would be in order, since I could eliminate a bunch of potentially
costly comparisons for each item. But in cases where I'm only trying
a single getattr (for example), using "if" might be a cheaper way to
go.

What do I mean by "cheaper"? I'm basically talking about the number
of instructions that are necessary to set up and execute a try block
as opposed to an if block.

Could you please tell me if I'm even remotely close to understanding
this correctly?

 

[ncf]

Honestly, I'm rather new to python, but my best bet would be to create
some test code and time it.

 

[wittempj]

My shot would be to test it like this on your platform like this:

#!/usr/bin/env python
import datetime, time
t1 = datetime.datetime.now()
for i in [str(x) for x in range(100)]:
if int(i) == i:
i + 1
t2 = datetime.datetime.now()
print t2 - t1
for i in [str(x) for x in range(100)]:
try:
int(i) +1
except:
pass
t3 = datetime.datetime.now()
print t3 - t2

for me (on python 2.4.1 on Linux on a AMD Sempron 2200+) it gives:
0:00:00.000637
0:00:00.000823

 

[Roy Smith]

Without fail, when I start talking with some of the "old-timers"
(people who have written code in ADA or Fortran), I hear the same
arguments that using "if" is "better" than using "try".

Well, you've now got a failure. I used to write Fortran on punch cards, so
I guess that makes me an "old-timer", and I don't agree with that argument.

I think that the argument goes something like, "When you set up a 'try'
block, you have to set up a lot of extra machinery than is necessary
just executing a simple conditional."

That sounds like a very C++ kind of attitude, where efficiency is prized
above all else, and exception handling is relatively heavy-weight compared
to a simple conditional.

What do I mean by "cheaper"? I'm basically talking about the number
of instructions that are necessary to set up and execute a try block
as opposed to an if block.

Don't worry about crap like that until the whole application is done and
it's not running fast enough, and you've exhausted all efforts to identify
algorithmic improvements that could be made, and careful performance
measurements have shown that the use of try blocks is the problem.

Exceptions are better than returning an error code for several reasons:

1) They cannot be silently ignored by accident. If you don't catch an
exception, it bubbles up until something does catch it, or nothing does and
your program dies with a stack trace. You can ignore them if you want, but
you have to explicitly write some code to do that.

2) It separates the normal flow of control from the error processing. In
many cases, this makes it easier to understand the program logic.

3) In some cases, they can lead to faster code. A classic example is
counting occurances of items using a dictionary:

count = {}
for key in whatever:
try:
count[key] += 1
except KeyError:
count[key] = 1

compared to

count = {}
for key in whatever:
if count.hasKey(key):
count[key] += 1
else:
count[key] = 1

if most keys are going to already be in the dictionary, handling the
occasional exception will be faster than calling hasKey() for every one.

 

[Wezzy]

My shot would be to test it like this on your platform like this:

#!/usr/bin/env python
import datetime, time
t1 = datetime.datetime.now()
for i in [str(x) for x in range(100)]:
if int(i) == i:
i + 1
t2 = datetime.datetime.now()
print t2 - t1
for i in [str(x) for x in range(100)]:
try:
int(i) +1
except:
pass
t3 = datetime.datetime.now()
print t3 - t2

for me (on python 2.4.1 on Linux on a AMD Sempron 2200+) it gives:
0:00:00.000637
0:00:00.000823

PowerBook:~/Desktop wezzy$ python test.py
0:00:00.001206
0:00:00.002092

Python 2.4.1 Pb15 with Tiger

 

[Thorsten Kampe]

* Steve Juranich (2005-07-09 19:21 +0100)

I know that this topic has the potential for blowing up in my face,
but I can't help asking. I've been using Python since 1.5.1, so I'm
not what you'd call a "n00b". I dutifully evangelize on the goodness
of Python whenever I talk with fellow developers, but I always hit a
snag when it comes to discussing the finer points of the execution
model (specifically, exceptions).

Without fail, when I start talking with some of the "old-timers"
(people who have written code in ADA or Fortran), I hear the same
arguments that using "if" is "better" than using "try". I think that
the argument goes something like, "When you set up a 'try' block, you
have to set up a lot of extra machinery than is necessary just
executing a simple conditional."

I was wondering how true this holds for Python, where exceptions are
such an integral part of the execution model. It seems to me, that if
I'm executing a loop over a bunch of items, and I expect some
condition to hold for a majority of the cases, then a "try" block
would be in order, since I could eliminate a bunch of potentially
costly comparisons for each item. But in cases where I'm only trying
a single getattr (for example), using "if" might be a cheaper way to
go.

What do I mean by "cheaper"? I'm basically talking about the number
of instructions that are necessary to set up and execute a try block
as opposed to an if block.

"Catch errors rather than avoiding them to avoid cluttering your code
with special cases. This idiom is called EAFP ('easier to ask
forgiveness than permission'), as opposed to LBYL ('look before you
leap')."

http://jaynes.colorado.edu/PythonIdioms.html

 

[Terry Reedy]

My shot would be to test it like this on your platform like this:

#!/usr/bin/env python
import datetime, time
t1 = datetime.datetime.now()
for i in [str(x) for x in range(100)]:
if int(i) == i:
i + 1
t2 = datetime.datetime.now()
print t2 - t1
for i in [str(x) for x in range(100)]:
try:
int(i) +1
except:
pass
t3 = datetime.datetime.now()
print t3 - t2

This is not a proper test since the if condition always fails and the
addition not done while the try succeeds and the addition is done. To be
equivalent, remove the int call in the try part: try: i+1. This would
still not a proper test since catching exceptions is known to be expensive
and try: except is meant for catching *exceptional* conditions, not
always-bad conditions. Here is a test that I think more useful:

for n in [1,2,3,4,5,10,20,50,100]:
# time this
for i in range(n):
if i != 0: x = 1/i
else: pass
# versus
for i in range(n):
try x = 1/i
except ZeroDivisionError: pass

I expect this will show if faster for small n and try for large n.

Terry J. Reedy

 

[Bruno Desthuilliers]

(e-mail address removed) a écrit :

My shot would be to test it like this on your platform like this:

#!/usr/bin/env python
import datetime, time

Why not use the timeit module instead ?

t1 = datetime.datetime.now()
for i in [str(x) for x in range(100)]:

A bigger range (at least 10/100x more) would probably be better...

if int(i) == i:

This will never be true, so next line...

i + 1

....wont never be executed.

t2 = datetime.datetime.now()
print t2 - t1
for i in [str(x) for x in range(100)]:
try:
int(i) +1
except:
pass

This will never raise, so the addition will always be executed (it never
will be in the previous loop).

t3 = datetime.datetime.now()
print t3 - t2

BTW, you end up including the time spent printing t2 - t1 in the
timing, and IO can be (very) costly.

(snip meaningless results)

The "test-before vs try-expect strategy" is almost a FAQ, and the usual
answer is that it depends on the hit/misses ratio. If the (expected)
ratio is high, try-except is better. If it's low, test-before is better.

HTH

 

[John Roth]

* Steve Juranich (2005-07-09 19:21 +0100)

"Catch errors rather than avoiding them to avoid cluttering your code
with special cases. This idiom is called EAFP ('easier to ask
forgiveness than permission'), as opposed to LBYL ('look before you
leap')."

http://jaynes.colorado.edu/PythonIdioms.html

It depends on what you're doing, and I don't find a "one size fits all"
approach to be all that useful.

If execution speed is paramount and exceptions are relatively rare,
then the try block is the better approach.

If you simply want to throw an exception, then the clearest way
of writing it that I've ever found is to encapsulate the raise statement
together with the condition test in a subroutine with a name that
describes what's being tested for. Even a name as poor as
"HurlOnFalseCondition(<condition>, <exception>, <parms>, <message>)
can be very enlightening. It gets rid of the in-line if and raise
statements,
at the cost of an extra method call.

John Roth



In both approaches, you have some
error handling code that is going to clutter up your program flow.

 

[Thomas Lotze]

I was wondering how true this holds for Python, where exceptions are such
an integral part of the execution model. It seems to me, that if I'm
executing a loop over a bunch of items, and I expect some condition to
hold for a majority of the cases, then a "try" block would be in order,
since I could eliminate a bunch of potentially costly comparisons for each
item.
Exactly.

But in cases where I'm only trying a single getattr (for example),
using "if" might be a cheaper way to go.

Relying on exceptions is faster. In the Python world, this coding style
is called EAFP (easier to ask forgiveness than permission). You can try
it out, just do something 10**n times and measure the time it takes. Do
this twice, once with prior checking and once relying on exceptions.

And JFTR: the very example you chose gives you yet another choice:
getattr can take a default parameter.

What do I mean by "cheaper"? I'm basically talking about the number of
instructions that are necessary to set up and execute a try block as
opposed to an if block.

I don't know about the implementation of exceptions but I suspect most
of what try does doesn't happen at run-time at all, and things get
checked and looked for only if an exception did occur. An I suspect that
it's machine code that does that checking and looking, not byte code.
(Please correct me if I'm wrong, anyone with more insight.)

 

[Dennis Lee Bieber]

Without fail, when I start talking with some of the "old-timers"
(people who have written code in ADA or Fortran), I hear the same

For an "old-timer" the languages are "Ada" and "FORTRAN"; "ADA"
is the American Dental Association, Americans with Disabilities Act,
etc. Or to be more explicit; Ada is a "name", FORTRAN was an
abbreviation/acronym (FORmula TRANslation). "Fortran" was decreed a
"name" with the F90 standard and may be used in general, but FORTRAN is
still proper for F77/F66 (F-IV), etc. which predate the F90 standard.

{Sorry -- but as a longtime fan, with no practical experience, of Ada,
this is a sticking point. Granted, it doesn't help that the Ada language
is, internally, case insensitive <G>}

--

 

[Peter Hansen]

I don't know about the implementation of exceptions but I suspect most
of what try does doesn't happen at run-time at all, and things get
checked and looked for only if an exception did occur. An I suspect that
it's machine code that does that checking and looking, not byte code.
(Please correct me if I'm wrong, anyone with more insight.)

Part right, part confusing. Definitely "try" is something that happens
at run-time, not compile time, at least in the sense of the execution of
the corresponding byte code. At compile time nothing much happens
except a determination of where to jump if an exception is actually
raised in the try block.

Try corresponds to a single bytecode SETUP_EXCEPT, so from the point of
view of Python code it is extremely fast, especially compared to
something like a function call (which some if-tests would do). (There
are also corresponding POP_BLOCK and JUMP_FORWARD instructions at the
end of the try block, and they're even faster, though the corresponding
if-test version would similarly have a jump of some kind involved.)

Exceptions in Python are checked for all the time, so there's little you
can do to avoid part of the cost of that. There is a small additional
cost (in the C code) when the exceptional condition is actually present,
of course, with some resulting work to create the Exception object and
raise it.

Some analysis of this can be done trivially by anyone with a working
interpreter, using the "dis" module.

def f():
try:
func()
except:
print 'ni!'

import dis
dis.dis(f)

Each line of the output represents a single bytecode instruction plus
operands, similar to an assembly code disassembly.

To go further, get the Python source and skim through the ceval.c
module, or do that via CVS
http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Python/ceval.c?rev=2.424&view=auto
, looking for the string "main loop".

And, in any case, remember that readability is almost always more
important than optimization, and you should consider first whether one
or the other approach is clearly more expressive (for future
programmers, including yourself) in the specific case involved.

-Peter

 

[John Machin]

Well, you've now got a failure. I used to write Fortran on punch cards,

which were then fed into an OCR gadget? That's an efficient approach --
where I was, we had to write the FORTRAN [*] on coding sheets; KPOs
would then produce the punched cards.

[snip]

3) In some cases, they can lead to faster code. A classic example is
counting occurances of items using a dictionary:

count = {}
for key in whatever:
try:
count[key] += 1
except KeyError:
count[key] = 1

compared to

count = {}
for key in whatever:
if count.hasKey(key):

Perhaps you mean has_key [*].
Perhaps you might like to try

if key in count:

It's believed to be faster (no attribute lookup, no function call).

[snip]

[*]
humanandcomputerlanguagesshouldnotimhousecaseandwordseparatorsascrutchesbuttheydosogetusedtoit

 

[Steven D'Aprano]

Relying on exceptions is faster. In the Python world, this coding style
is called EAFP (easier to ask forgiveness than permission). You can try
it out, just do something 10**n times and measure the time it takes. Do
this twice, once with prior checking and once relying on exceptions.

True, but only sometimes. It is easy to write a test that gives misleading
results.

In general, setting up a try...except block is cheap, but actually calling
the except clause is expensive. So in a test like this:

for i in range(10000):
try:
x = mydict["missing key"]
except KeyError:
print "Failed!"

will be very slow (especially if you time the print, which is slow).

On the other hand, this will be very fast:

for i in range(10000):
try:
x = mydict["existing key"]
except KeyError:
print "Failed!"

since the except is never called.

On the gripping hand, testing for errors before they happen will be slow
if errors are rare:

for i in range(10000):
if i == 0:
print "Failed!"
else:
x = 1.0/i

This only fails on the very first test, and never again.

When doing your test cases, try to avoid timing things unrelated to the
thing you are actually interested in, if you can help it. Especially I/O,
including print. Do lots of loops, if you can, so as to average away
random delays due to the operating system etc. But most importantly, your
test data must reflect the real data you expect. Are most tests
successful or unsuccessful? How do you know?

However, in general, there are two important points to consider.

- If your code has side effects (eg changing existing objects, writing to
files, etc), then you might want to test for error conditions first.
Otherwise, you can end up with your data in an inconsistent state.

Example:

L = [3, 5, 0, 2, 7, 9]

def invert(L):
"""Changes L in place by inverting each item."""
try:
for i in range(len(L)):
L = 1.0/L
except ZeroDivisionError:
pass

invert(L)
print L

=> [0.333, 0.2, 0, 2, 7, 9]


- Why are you optimizing your code now anyway? Get it working the simplest
way FIRST, then _time_ how long it runs. Then, if and only if it needs to
be faster, should you worry about optimizing. The simplest way will often
be try...except blocks.

 

[Jorey Bump]

I was wondering how true this holds for Python, where exceptions are
such an integral part of the execution model. It seems to me, that if
I'm executing a loop over a bunch of items, and I expect some
condition to hold for a majority of the cases, then a "try" block
would be in order, since I could eliminate a bunch of potentially
costly comparisons for each item. But in cases where I'm only trying
a single getattr (for example), using "if" might be a cheaper way to
go.

What do I mean by "cheaper"? I'm basically talking about the number
of instructions that are necessary to set up and execute a try block
as opposed to an if block.

Could you please tell me if I'm even remotely close to understanding
this correctly?

*If* I'm not doing a lot of things once, I *try* to do one thing a lot.

 

[Dark Cowherd]

http://www.joelonsoftware.com/items/2003/10/13.html

 

[Thomas Lotze]

On the gripping hand, testing for errors before they happen will be slow
if errors are rare:

Hm, might have something to do with why those things intended for
handling errors after they happened are called exceptions ;o)

- If your code has side effects (eg changing existing objects, writing to
files, etc), then you might want to test for error conditions first.
Otherwise, you can end up with your data in an inconsistent state.

BTW: Has the context management stuff from PEP 343 been considered for
implementing transactions?

- Why are you optimizing your code now anyway? Get it working the simplest
way FIRST, then _time_ how long it runs. Then, if and only if it needs to
be faster, should you worry about optimizing. The simplest way will often
be try...except blocks.

Basically, I agree with the "make it run, make it right, make it fast"
attitude. However, FWIW, I sometimes can't resist optimizing routines that
probably don't strictly need it. Not only does the resulting code run
faster, but it is usually also shorter and more readable and expressive.
Plus, I tend to gain further insight into the problem and tools in the
process. YMMV, of course.

 

[Roy Smith]

Basically, I agree with the "make it run, make it right, make it fast"
attitude. However, FWIW, I sometimes can't resist optimizing routines that
probably don't strictly need it. Not only does the resulting code run
faster, but it is usually also shorter and more readable and expressive.

Optimize for readability and maintainability first. Worry about speed
later.

 

[John Roth]

Steven D'Aprano wrote:


Basically, I agree with the "make it run, make it right, make it fast"
attitude. However, FWIW, I sometimes can't resist optimizing routines that
probably don't strictly need it. Not only does the resulting code run
faster, but it is usually also shorter and more readable and expressive.
Plus, I tend to gain further insight into the problem and tools in the
process. YMMV, of course.

Shorter, more readable and expressive are laudable goals in and
of themselves. Most of the "advice" on optimization assumes that
after optimization, routines will be less readable and expressive,
not more.

In other words, I wouldn't call the activity of making a routine
more readable and expressive of intent "optimization." If it runs
faster, that's a bonus. It frequently will, at least if you don't add
method calls to the process.

John Roth

 

[Steven D'Aprano]

http://www.joelonsoftware.com/items/2003/10/13.html

Joel Spolsky might be a great C++ programmer, and his advice on user
interface design is invaluable, but Python is not C++ or Java, and his
arguments about exceptions do not hold in Python.

Joel argues:

"They are invisible in the source code. Looking at a block of code,
including functions which may or may not throw exceptions, there is no way
to see which exceptions might be thrown and from where. This means that
even careful code inspection doesn't reveal potential bugs."

I don't quiet get this argument. In a random piece of source code, there
is no way to tell whether or not it will fail just by inspection. If you
look at:

x = 1
result = myfunction(x)

you can't tell whether or not myfunction will fail at runtime just by
inspection, so why should it matter whether it fails by crashing at
runtime or fails by raising an exception?

Joel's argument that raising exceptions is just a goto in disguise is
partly correct. But so are for loops, while loops, functions and methods!
Like those other constructs, exceptions are gotos tamed and put to work
for you, instead of wild and dangerous. You can't jump *anywhere*, only
highly constrained places.

Joel also writes:

"They create too many possible exit points for a function. To write
correct code, you really have to think about every possible code path
through your function. Every time you call a function that can raise an
exception and don't catch it on the spot, you create opportunities for
surprise bugs caused by functions that terminated abruptly, leaving data
in an inconsistent state, or other code paths that you didn't think about."

This is a better argument for *careful* use of exceptions, not an argument
to avoid them. Or better still, it is an argument for writing code which
doesn't has side-effects and implements data transactions. That's a good
idea regardless of whether you use exceptions or not.

Joel's concern about multiple exit points is good advice, but it can be
taken too far. Consider the following code snippet:

def myfunc(x=None):
result = ""
if x is None:
result = "No argument given"
elif x = 0:
result = "Zero"
elif 0 < x <= 3:
resutl = "x is between 0 and 3"
else:
result = "x is more than 3"
return result

There is no benefit in deferring returning value as myfunc does, just
for the sake of having a single exit point. "Have a single exit point"
is a good heuristic for many functions, but it is pointless make-work for
this one. (In fact, it increases, not decreases, the chances of a bug. If
you look carefully, myfunc above has such a bug.

Used correctly, exceptions in Python have more advantages than
disadvantages. They aren't just for errors either: exceptions can be
triggered for exceptional cases (hence the name) without needing to track
(and debug) multiple special cases.

Lastly, let me argue against one of Joel's comments:

"A better alternative is to have your functions return error values when
things go wrong, and to deal with these explicitly, no matter how verbose
it might be. It is true that what should be a simple 3 line program often
blossoms to 48 lines when you put in good error checking, but that's life,
and papering it over with exceptions does not make your program more
robust."

Maybe that holds true for C++. I don't know the language, and wouldn't
like to guess. But it doesn't hold true for Python. This is how Joel might
write a function as a C programmer:

def joels_function(args):
error_result = 0
good_result = None
process(args)
if error_condition():
error_result = -1 # flag for an error
elif different_error_conditon():
error_result = -2
else:
more_processing()
if another_error_conditon():
error_result = -3
do_more_work()
good_result = "Success!"
if error_result != 0:
return (False, error_result)
else:
return (True, good_result)


and then call it with:

status, msg = joels_function(args)
if status == False:
print msg
# and fail...
else:
print msg
# and now continue...


This is how I would write it in Python:

def my_function(args):
process(args)
if error_condition():
raise SomeError("An error occurred")
elif different_error_conditon():
raise SomeError("A different error occurred")
more_processing()
if another_error_conditon():
raise SomeError("Another error occurred")
do_more_work()
return "Success!"

and call it with:

try:
result = my_function(args)
print "Success!!!"
except SomeError, msg:
print msg
# and fail...
# and now continue safely here...


In the case of Python, calling a function that may raise an exception is
no more difficult or unsafe than calling a function that returns a status
flag and a result, but writing the function itself is much easier, with
fewer places for the programmer to make a mistake.

In effect, exceptions allow the Python programmer to concentrate on his
actual program, rather than be responsible for building error-handling
infrastructure into every function. Python supplies that infrastructure
for you, in the form of exceptions.