Generators and propagation of exceptions

[r]

I had a problem for which I've already found a "satisfactory"
work-around, but I'd like to ask you if there is a better/nicer
looking solution. Perhaps I'm missing something obvious.

The code looks like this:

stream-of-tokens = token-generator(stream-of-characters)
stream-of-parsed-expressions = parser-generator(stream-of-tokens)
stream-of-results = evaluator-generator(stream-of-parsed-expressions)

each of the above functions consumes and implements a generator:

def evaluator-generator(stream-of-tokens):
for token in stream-of-tokens:
try:
yield token.evaluate() # evaluate() returns a Result
except Exception as exception:
yield ErrorResult(exception) # ErrorResult is a subclass of Result

The problem is that, when I use the above mechanism, the errors
propagate to the output embedded in the data streams. This means, I
have to make them look like real data (in the example above I'm
wrapping the exception with an ErrorExpression object) and raise them
and intercept them again at each level until they finally trickle down
to the output. It feels a bit like writing in C (checking error codes
and propagating them to the caller).

OTOH, if I don't intercept the exception inside the loop, it will
break the for loop and close the generator. So the error no longer
affects a single token/expression but it kills the whole session. I
guess that's because the direction flow of control is sort of
orthogonal to the direction of flow of data.

Any idea for a working and elegant solution?

Thanks,

-r

 

[Raymond Hettinger]

I had a problem for which I've already found a "satisfactory"
work-around, but I'd like to ask you if there is a better/nicer
looking solution. Perhaps I'm missing something obvious.

The code looks like this:

stream-of-tokens = token-generator(stream-of-characters)
stream-of-parsed-expressions = parser-generator(stream-of-tokens)
stream-of-results = evaluator-generator(stream-of-parsed-expressions)

each of the above functions consumes and implements a generator:

def evaluator-generator(stream-of-tokens):
  for token in stream-of-tokens:
    try:
       yield token.evaluate()           # evaluate() returns a Result
    except Exception as exception:
       yield ErrorResult(exception) # ErrorResult is a subclass of Result

The problem is that, when I use the above mechanism, the errors
propagate to the output embedded in the data streams. This means, I
have to make them look like real data (in the example above I'm
wrapping the exception with an ErrorExpression object) and raise them
and intercept them again at each level until they finally trickle down
to the output. It feels a bit like writing in C (checking error codes
and propagating them to the caller).

You're right, checking and propagating at each level doesn't feel
right.

If you need an exception to break out of the loops and close the
generators, then it seems you should just ignore the exception on
every level except for the one where you really want to handle the
exception (possibly in an outer control-loop):

while True:
try:

for result in evaluator-generator(stream-of-parsed-
expressions):


let the exception float up to

 

[Raymond Hettinger]

I had a problem for which I've already found a "satisfactory"
work-around, but I'd like to ask you if there is a better/nicer
looking solution. Perhaps I'm missing something obvious.

The code looks like this:

stream-of-tokens = token-generator(stream-of-characters)
stream-of-parsed-expressions = parser-generator(stream-of-tokens)
stream-of-results = evaluator-generator(stream-of-parsed-expressions)

each of the above functions consumes and implements a generator:

def evaluator-generator(stream-of-tokens):
  for token in stream-of-tokens:
    try:
       yield token.evaluate()           # evaluate() returns a Result
    except Exception as exception:
       yield ErrorResult(exception) # ErrorResult is a subclass of Result

The problem is that, when I use the above mechanism, the errors
propagate to the output embedded in the data streams. This means, I
have to make them look like real data (in the example above I'm
wrapping the exception with an ErrorExpression object) and raise them
and intercept them again at each level until they finally trickle down
to the output. It feels a bit like writing in C (checking error codes
and propagating them to the caller).

You could just let the exception go up to an outermost control-loop
without handling it at all on a lower level. That is what exceptions
for you: terminate all the loops, unwind the stacks, and propagate up
to some level where the exception is caught:

while 1:
try:
results = evaluator-generator(stream-of-parsed-expressions)
for result in results:
print(result)
except Exception as e:
handle_the_exception(e)

OTOH, If you want to catch the exception at the lowest level and wrap
it up as data (the ErrorResult in your example), there is a way to
make it more convenient. Give the ErrorResult object some identify
methods that correspond to the methods being called by upper levels.
This will let the object float through without you cluttering each
level with detect-and-reraise logic.

class ErrorResult:
def __iter__(self):
# pass through an enclosing iterator
yield self

Here's a simple demo of how the pass through works:

from itertools import *
list(chain([1,2,3], ErrorResult(), [4,5,6]))

Any idea for a working and elegant solution?

Hope these ideas have helped.


Raymond

 

[r]

You could just let the exception go up to an outermost control-loop
without handling it at all on a lower level.  That is what exceptions
for you: terminate all the loops, unwind the stacks, and propagate up
to some level where the exception is caught:

 while 1:
    try:
       results = evaluator-generator(stream-of-parsed-expressions)
       for result in results:
           print(result)
    except Exception as e:
       handle_the_exception(e)

I've been thinking about something like this but the problem with
shutting down the generators is that I lose all the state information
associated with them (line numbers, have to reopen files if in batch
mode etc.). It's actually more difficult than my current solution.

OTOH, If you want to catch the exception at the lowest level and wrap
it up as data (the ErrorResult in your example), there is a way to
make it more convenient.  Give the ErrorResult object some identify
methods that correspond to the methods being called by upper levels.
This will let the object float through without you cluttering each
level with detect-and-reraise logic.

I'm already making something like this (that is, if I understand you
correctly). In the example below (an "almost" real code this time, I
made too many mistakes before) all the Expressions (including the
Error one) implement an 'eval' method that gets called by one of the
loops. So I don't have to do anything to detect the error, just have
to catch it and reraise it at each stage so that it propagates to the
next level (what I have to do anyway as the next level generates
errors as well).

class Expression(object):
def eval(self):
pass

class Error(Expression):
def __init__(self, exception):
self.exception = exception

def eval(self):
raise self.exception

def parseTokens(self, tokens):
for token in tokens:
try:
yield Expression.parseToken(token, tokens)
except ExpressionError as e:
# here is where I wrap exceptions raised during parsing and embed them
yield Error(e)

def eval(expressions, frame):
for expression in expressions:
try:
# and here (.eval) is where they get unwrapped and raised again
yield unicode(expression.eval(frame))
except ExpressionError as e:
# here they are embedded again but because it is the last stage
# text representation is fine
yield unicode(e)

  class ErrorResult:
      def __iter__(self):
          # pass through an enclosing iterator
          yield self

Here's a simple demo of how the pass through works:

   >>> from itertools import *
   >>> list(chain([1,2,3], ErrorResult(), [4,5,6]))
   [1, 2, 3, <__main__.ErrorResult object at 0x2250f70>, 4, 5, 6]

I don't really understand what you mean by this example. Why would
making the Error iterable help embedding it into data stream? I'm
currently using yield statement and it seems to work well (?).

Anyway, thank you all for helping me out and bringing some ideas to
the table. I was hoping there might be some pattern specifically
designed for thiskind of job (exception generators anyone?), which
I've overlooked. If not anything else, knowing that this isn't the
case, makes me feel better about the solution I've chosen.

Thanks again,

-r

 

[Raymond Hettinger]

Anyway, thank you all for helping me out and bringing some ideas to
the table. I was hoping there might be some pattern specifically
designed for thiskind of job (exception generators anyone?), which
I've overlooked. If not anything else, knowing that this isn't the
case, makes me feel better about the solution I've chosen.

Sounds like you've gathered a bunch of good ideas and can now be
pretty sure of your design choices.

While it doesn't help your current use case, you might be interested
in the iter_except() recipe in http://docs.python.org/py3k/library/itertools.html#itertools-recipes

Raymond

twitter: @Raymondh

 

[Kent Johnson]

I'm already making something like this (that is, if I understand you
correctly). In the example below (an "almost" real code this time, I
made too many mistakes before) all the Expressions (including the
Error one) implement an 'eval' method that gets called by one of the
loops. So I don't have to do anything to detect the error, just have
to catch it and reraise it at each stage so that it propagates to the
next level (what I have to do anyway as the next level generates
errors as well).

class Expression(object):
    def eval(self):
        pass

class Error(Expression):
    def __init__(self, exception):
        self.exception = exception

    def eval(self):
        raise self.exception

Perhaps, instead of raising exceptions at each level, you could return
an Error object that implements the eval() (or whatever appropriate
protocol) to return an appropriate new Error object. In this case, you
could have
class Error(Expression);
....
def eval(self):
return unicode(self.exception)

Error would itself be created by Expression.parseToken(), instead of
raising an exception.

The idea is that at each level of parsing, instead of raising an
exception you return an object which, when interpreted at the next
outer level, will again return an appropriate error object for that
level. You might be able to have a single Error object which
implements the required methods at each level to just return self.

I don't know if this really works when you start nesting but perhaps
it is worth a try.

Kent