Try, except...retry?

[Robert Brewer]

Alex Martelli wrote in another thread:

One sign that somebody has moved from "Python newbie" to "good Python
programmer" is exactly the moment they realize why it's wrong to code:

try:
x = could_raise_an_exception(23)
process(x)
except Exception, err:
deal_with_exception(err)
proceed_here()

and why the one obvious way is, instead:

try:
x = could_raise_an_exception(23)
except Exception, err:
deal_with_exception(err)
else:
process(x)
proceed_here()

I just got there, Alex. However, it got me thinking. I keep running
into cases like:

try:
aPiranha = allPiranhas['Doug']
except KeyError:
aPiranha = Pirhana()
allPiranhas['Doug'] = aPiranha
aPiranha.weapon = u'satire'

which would, in my opinion, be better written (i.e. *clearer*) as:

try:
allPiranhas['Doug'].weapon = u'satire'
except KeyError:
allPiranhas['Doug'] = Pirhana()
retry

Of course, there are other ways of doing it currently, most notably with
1) a while loop and a retry flag, or 2) just repeating the assignment:

try:
allPiranhas['Doug'].weapon = u'satire'
except KeyError:
allPiranhas['Doug'] = Pirhana()
allPiranhas['Doug'].weapon = u'satire'

Yuck to both.

Current docs, 4.2 Exceptions says, "Python uses the ``termination''
model of error handling: an exception handler can find out what happened
and continue execution at an outer level, but it cannot repair the cause
of the error and retry the failing operation (except by re-entering the
offending piece of code from the top)." I'm proposing that 'retry' does
exactly that: reenter the offending piece of code from the top. Given
the aforementioned pressure to reduce try: blocks to one line, this
could become a more viable/common technique.

Quick Googling shows I'm not the first with this concept:
http://mail.zope.org/pipermail/zope-checkins/2001-November/008857.html

Apparently Ruby has this option? Gotta keep up with the Joneses. I'm
not enough of a Pythonista yet to understand all the implications of
such a scheme (which is why this is not a PEP), so I offer it to the
community to discuss.


Robert Brewer
MIS
Amor Ministries
(e-mail address removed)

 

[Isaac To]

Hmmm... I agree only with reservation. The primary advantage of
exceptions, rather than other error reporting mechanisms like a return
value, is that you can group a lot of statements and have a single block of
code handling all of those. This way you can separate the error handling
code and the normal code, to make the whole logic clearer. So if the
exception doesn't happen often enough to warrant the effort, I consider the
one at the begining, i.e.,

try:
x1 = could_raise_an_exception(1)
x2 = could_raise_an_exception(2)
x3 = could_raise_an_exception(3)
x4 = could_raise_an_exception(4)
process(x1)
process(x2)
process(x3)
process(x4)
except Exception, err:
deal_with_exception(err)
proceed_here()

to be a better alternative than the corresponding code when all the excepts
are written out in else parts, just because it is easier to understand.

Regards,
Isaac.

 

[Georgy Pruss]

| <...> The primary advantage of
| exceptions, rather than other error reporting mechanisms like a return
| value, is that you can group a lot of statements and have a single block of
| code handling all of those. This way you can separate the error handling
| code and the normal code, to make the whole logic clearer. <...>
|
| Regards,
| Isaac.

Not only that simple. I would say not only a lot of statements, but a lot of
very complex code, which however has no slightest idea what to do with
error situations inside it and how to proceess the errors or where and how
report them. E.g. it's quite often that you have open(file,mode) at the very
low level of your function hierarchy, but it's only several levels up that you
can analyze the error and decide what to do about it and pop up a message
box with something like "Error in installation; Please reinstall the program"
and log this error, instead of printing "Can't open /usr/bin/blah/blah/config.data"
right across the screen, or having each and every function to return error
code together with some 'useful' data.

That is, I agree with you. Just wanted to add that it's not a simple trick to
process errors for a few statements in one place, but an important design
principle.

Regarding 'retry' - it's good that there's no 'retry' in Python. It would have
introduced another loop-like structure in the language. Anyway now you can
mimic it with try-except within a loop.

 

[Peter Otten]

I just got there, Alex. However, it got me thinking. I keep running
into cases like:

try:
aPiranha = allPiranhas['Doug']
except KeyError:
aPiranha = Pirhana()
allPiranhas['Doug'] = aPiranha
aPiranha.weapon = u'satire'

which would, in my opinion, be better written (i.e. *clearer*) as:

try:
allPiranhas['Doug'].weapon = u'satire'
except KeyError:
allPiranhas['Doug'] = Pirhana()
retry

retry does remind me of VBA's on error goto ... resume [next] a little too
much. You lose track of the actual program flow too fast. To me, the first
variant appears much clearer in that it has only code in the try clause
that is actually supposed to
fail. Also, in

try:
setWeapon("Doug", u"satire")
except KeyError:
allPiranhas["Doug"] = Goldfish()
retry

where would you want to jump? Should setWeapon() be executed in a way
similar to generators, saving its state in case of an exception instead of
a yield? Sometimes you can *not* execute a statement twice in a reliable
manner.

Peter

 

[Peter Otten]

Hmmm... I agree only with reservation. The primary advantage of
exceptions, rather than other error reporting mechanisms like a return
value, is that you can group a lot of statements and have a single block
of
code handling all of those. This way you can separate the error handling
code and the normal code, to make the whole logic clearer. So if the
exception doesn't happen often enough to warrant the effort, I consider
the one at the begining, i.e.,

try:
x1 = could_raise_an_exception(1)
x2 = could_raise_an_exception(2)
x3 = could_raise_an_exception(3)
x4 = could_raise_an_exception(4)
process(x1)
process(x2)
process(x3)
process(x4)
except Exception, err:
deal_with_exception(err)
proceed_here()

to be a better alternative than the corresponding code when all the
excepts are written out in else parts, just because it is easier to
understand.

I would prefer:

x1 = x2 = x3 = x4 = None
try:
x1 = acquireRessource(1)
x2 = acquireRessource(2)
x3 = acquireRessource(3)
x4 = acquireRessource(4)
process(x1, x2, x3, x4)
finally:
if x1: releaseRessource(x1)
if x2: releaseRessource(x2)
if x3: releaseRessource(x3)
if x4: releaseRessource(x4)

where acquireRessource() should try hard to deal with potential exceptions
and only pass the fatal ones to the caller. Exceptions occuring when
processing the ressource should be dealt with in process() if possible. As
for process() working orthoganally on xi, that does not make any sense to
me. In such cases, be generous and provide a try ... finally and/or except
for every ressource.

Now back to else:

try:
x = canRaise()
cannotRaise(x)
except Exception, e:
dealWithException(e)

You must be very confident in your code to assume that cannotRaise() will
never raise an exception that dealWithException() will then accidentally
handle. So after a short phase of accommodation,

try:
x = canRaise()
except Exception, e:
dealWithException(e)
else:
cannotRaise(x) # may still be lying

is both more readable and better code (neglecting for the moment the
redundancy that many python coders will find in these attributes


Peter

 

[Alex Martelli]

Isaac To wrote:
...

try:
x1 = could_raise_an_exception(1)
x2 = could_raise_an_exception(2)
x3 = could_raise_an_exception(3)
x4 = could_raise_an_exception(4)
process(x1)
process(x2)
process(x3)
process(x4)
except Exception, err:
deal_with_exception(err)
proceed_here()

to be a better alternative than the corresponding code when all the
excepts are written out in else parts, just because it is easier to
understand.

So is "a + b" easier to understand than math.hypot(a, b) -- but if
what you need to compute is the square root of a**2 plus b**2, the
"ease of understanding" a very different computation means nothing.

How do you KNOW that function 'process' cannot raise an exception
(e.g., that it has no bugs) or, if it does, deal_with_exception
will be able to uniformly handle any exception coming from EITHER
process (on ANY of the x's, too) OR any of the several calls to
could_raise_an_exception?

It is extremely unlikely (though not impossible) that this structure
is actually best for a given set of needs. When it isn't, what
you're doing is setting up a trap for yourself here.

Let's assume, for example, that deal_with_exception IS able to
uniformly handle ANY exception whatsoever raised by any call to
could_raise_an_exception, but NO exceptions are expected from any
of the calls to process.

If this is the case, then, if you make it a habit to program like
this, it WILL eventually cost you some unpredictable but likely
large amount of debugging time. A bug will be introduced during
a refactoring of function 'process', or an 'x' will be generated
that is inappropriate for the function, etc, etc. The bug will
be swallowed and hidden by the too-wide "except" clause. Unit
tests will fail mysteriously (assuming you have them: people who
botch up program structure this way are unfortunately likely to
think that unit tests are too much trouble, not "easy to understand"
enough, or the like... -- or else, worse!!!, wrong but semi-plausible
results will come out of your program...!). The best wish I can
make is that the resulting utter waste of hours, days, whatever,
will fall on the head of whoever IS responsibile for promoting or
using this wrong structure, and not on some poor hapless maintainer
coming later onto the scene of the crime.


Alex

 

[Alex Martelli]

Robert Brewer wrote:
...

into cases like:

try:
aPiranha = allPiranhas['Doug']
except KeyError:
aPiranha = Pirhana()
allPiranhas['Doug'] = aPiranha
aPiranha.weapon = u'satire'

Yeah, it IS frequent enough that Python has two well-known idioms to
deal with it. If a call to Piranha() has very low cost,

allPiranhas.setdefault('Doug', Piranha()).weapon = u'satire'

is very compact. If calling Piranha() is potentially costly, this
is unfortunately no good (due to Python's "strict" execution order,
all arguments are evaluated before running the method), so:

if 'Doug' not in allPiranhas:
allPiranhas['Doug'] = Piranha()
allPiranhas['Doug'].weapon = u'satire'

or, to avoid indexing twice:

aPiranha = allPiranhas.get('Doug')
if aPiranha is None:
aPiranha = allPiranhas['Doug'] = Piranha()
aPiranha.weapon = u'satire'

In retrospect, it WOULD perhaps be better if setdefault was designed
to take a callable (and optional args for it) and only call it if and
when needed -- that would add a little speed and clarity in the two most
typical use cases, currently:
dictOfDicts.setdefault(mainKey, {})[secondaryKey] = value
and
dictOfLists.setdefault(key, []).append(value)
which would just become:
dictOfDicts.setdefault(mainKey, dict)[secondaryKey] = value
and
dictOfLists.setdefault(key, list).append(value)
respectively; and widen the applicability of .setdefault to other cases,
while costing very little actual use cases (I've never seen setdefault
correctly called with a 2nd argument that wasn't (), {}, or the like).
Ah well, too late, just musing aloud.

Still, it seems to me that the existing idioms are nevertheless
superior to your desideratum:

which would, in my opinion, be better written (i.e. *clearer*) as:

try:
allPiranhas['Doug'].weapon = u'satire'
except KeyError:
allPiranhas['Doug'] = Pirhana()
retry

which does index twice anyway.

Of course, there are other ways of doing it currently, most notably with
1) a while loop and a retry flag, or 2) just repeating the assignment:

try:
allPiranhas['Doug'].weapon = u'satire'
except KeyError:
allPiranhas['Doug'] = Pirhana()
allPiranhas['Doug'].weapon = u'satire'

Yuck to both.

Yes, flags (hiding control flow in data!) and repeated code do suck, but
you need neither to get exactly the same semantics as your desideratum:

while True:
try: allPiranhas['Doug'].weapon = u'satire'
except KeyError: allPiranhas['Doug'] = Pirhana()
else: break

Current docs, 4.2 Exceptions says, "Python uses the ``termination''
model of error handling: an exception handler can find out what happened
and continue execution at an outer level, but it cannot repair the cause
of the error and retry the failing operation (except by re-entering the
offending piece of code from the top)." I'm proposing that 'retry' does
exactly that: reenter the offending piece of code from the top. Given
the aforementioned pressure to reduce try: blocks to one line, this
could become a more viable/common technique.

It does not appear to me that, even assuming that this looping is in
fact the best general approach, there is enough advantage to your
proposed "try/except ... retry" technique, with respect to the
"while/try/except/else: break" one that is already possible today.

I could be wrong, of course: there is nothing that appears to me
to be "outrageously nonPythonic" in your proposal -- it just seems
that new statements need to be more of a win than this in order to
stand a chance. But a PEP on this is surely warranted, if you want
to try one.

Apparently Ruby has this option? Gotta keep up with the Joneses. I'm

Yes, Ruby does allow retry in a begin/rescue/else/end construct (on
the rescue leg only). I don't think the use case in the "Programming
Ruby" book shows it in a good light _at all_, though -- and I quote:

"""
@esmtp = true

begin
# First try an extended login. If it fails because the
# server doesn't support it, fall back to a normal login

if @esmtp then
@command.ehlo(helodom)
else
@command.helo(helodom)
end

rescue ProtocolError
if @esmtp then
@esmtp = false
retry
else
raise
end
end
"""

Eep!-) The dreaded "flag", AND a rather intricated structure
too...!!! It seems to me that a much clearer way is (Python-ish
syntax, same semantics):

try:
self.command.ehlo(helodom)
except ProtocolError:
self.command.helo(helodom)
self.smpt = False
else:
self.smpt = True

(this leaves the self.smpt flag unset if the ProtocolError
exception propagates, rather than ensuring it's false in that
case, but if [unlikely...] this gave any problem, then just
moving the "self.smpt = False" to the top would fix that.

Admittedly the use of else IS "a bit precious" here, since
assigning to self.smpt is VERY unlikely to raise a ProtocolError,
so, maybe,

try:
self.command.ehlo(helodom)
self.smpt = True
except ProtocolError:
self.command.helo(helodom)
self.smpt = False

would also be OK. I just try to foster the HABIT of using
try/except/else to avoid "protecting", with an except, ANY
more code than strictly necessary, on general principles. But
in this case the else-less form has a very pleasing symmetry
so I would nod it through during a code-inspection or the like.

But the point is, the existence of retry has tempted those
_excellent_ (and pragmatic authors & programmers, Thomas and
Hunt, into perverting a clean, simple structure into a little
but definite mess. This sure ain't good recommendation for
adding 'retry' to Python.... Given that the "key missing
in a dict" case is also dealt with quite decently without
looping, I would suggest you look for other use cases.
Perhaps simplest...:

try:
spin = raw_input("Please enter your PIN: ")
pin = int(spin)
except (EOFError, KeyboardInterrupt):
print "Bye bye!"
return 0
except ValueError:
print "PIN must be an integer [just digits!!!], please re-enter"
retry
else:
return validatePIN(pin)

not enough of a Pythonista yet to understand all the implications of
such a scheme (which is why this is not a PEP), so I offer it to the
community to discuss.

No special "implications", as the semantics are just about the
same as the above-indicated (flags-less, duplication-less)
"while True:" loop that is so easy to code explicitly today.

It's just that, partly because of this (and attendant benefits
that writing out "while" DOES clearly indicate to the reader
that the following code may repeat, etc etc), it does not seem
to me that 'retry' is worth adding. But unless somebody does
write a PEP, you'll just have my opinion about this...


Alex

 

[Edvard Majakari]

If this is the case, then, if you make it a habit to program like
this, it WILL eventually cost you some unpredictable but likely
large amount of debugging time. A bug will be introduced during
a refactoring of function 'process', or an 'x' will be generated
that is inappropriate for the function, etc, etc. The bug will
be swallowed and hidden by the too-wide "except" clause. Unit

How about

class xmlParseException(Exception): pass
class xmlSemanticsException(Exception): pass
class fileReadException(Exception): pass

try:

process()
process2()
...lots of code...

except xmlSemanticsException, e:
handle_xml_semantics_errors()
except xmlParseException, e:
handle_xml_parsing_errors()
except fileReadException, e:
handl_file_read_errors()

At least I find it easier to group possibly-exceptions-raising code
separately, inherit specific exceptions from class Exception using the
most simple method (ie. derived class contains only pass as body). Now it
is not that hard to pinpoint where the exception occurred, because there
is custom exception for each type of error situation, and the parameter e
of course contains appropriate error message which even further helps to
pinpoint the problem.

think that unit tests are too much trouble, not "easy to understand"
enough, or the like... -- or else, worse!!!, wrong but semi-plausible
results will come out of your program...!). The best wish I can
make is that the resulting utter waste of hours, days, whatever,
will fall on the head of whoever IS responsibile for promoting or
using this wrong structure, and not on some poor hapless maintainer
coming later onto the scene of the crime.

....but of course, unit tests should be used whenever possible

--
# Edvard Majakari Software Engineer
# PGP PUBLIC KEY available Soli Deo Gloria!

$_ = '456476617264204d616a616b6172692c20612043687269737469616e20'; print
join('',map{chr hex}(split/(\w{2})/)),uc substr(crypt(60281449,'es'),2,4),"\n";