Generators and propagation of exceptions

Discussion in 'Python' started by r, Apr 8, 2011.

  1. r

    r Guest

    I had a problem for which I've already found a "satisfactory"
    work-around, but I'd like to ask you if there is a better/nicer
    looking solution. Perhaps I'm missing something obvious.

    The code looks like this:

    stream-of-tokens = token-generator(stream-of-characters)
    stream-of-parsed-expressions = parser-generator(stream-of-tokens)
    stream-of-results = evaluator-generator(stream-of-parsed-expressions)

    each of the above functions consumes and implements a generator:

    def evaluator-generator(stream-of-tokens):
    for token in stream-of-tokens:
    try:
    yield token.evaluate() # evaluate() returns a Result
    except Exception as exception:
    yield ErrorResult(exception) # ErrorResult is a subclass of Result

    The problem is that, when I use the above mechanism, the errors
    propagate to the output embedded in the data streams. This means, I
    have to make them look like real data (in the example above I'm
    wrapping the exception with an ErrorExpression object) and raise them
    and intercept them again at each level until they finally trickle down
    to the output. It feels a bit like writing in C (checking error codes
    and propagating them to the caller).

    OTOH, if I don't intercept the exception inside the loop, it will
    break the for loop and close the generator. So the error no longer
    affects a single token/expression but it kills the whole session. I
    guess that's because the direction flow of control is sort of
    orthogonal to the direction of flow of data.

    Any idea for a working and elegant solution?

    Thanks,

    -r
    r, Apr 8, 2011
    #1
    1. Advertising

  2. On Apr 8, 8:55 am, r <> wrote:
    > I had a problem for which I've already found a "satisfactory"
    > work-around, but I'd like to ask you if there is a better/nicer
    > looking solution. Perhaps I'm missing something obvious.
    >
    > The code looks like this:
    >
    > stream-of-tokens = token-generator(stream-of-characters)
    > stream-of-parsed-expressions = parser-generator(stream-of-tokens)
    > stream-of-results = evaluator-generator(stream-of-parsed-expressions)
    >
    > each of the above functions consumes and implements a generator:
    >
    > def evaluator-generator(stream-of-tokens):
    >   for token in stream-of-tokens:
    >     try:
    >        yield token.evaluate()           # evaluate() returns a Result
    >     except Exception as exception:
    >        yield ErrorResult(exception) # ErrorResult is a subclass of Result
    >
    > The problem is that, when I use the above mechanism, the errors
    > propagate to the output embedded in the data streams. This means, I
    > have to make them look like real data (in the example above I'm
    > wrapping the exception with an ErrorExpression object) and raise them
    > and intercept them again at each level until they finally trickle down
    > to the output. It feels a bit like writing in C (checking error codes
    > and propagating them to the caller).


    You're right, checking and propagating at each level doesn't feel
    right.

    If you need an exception to break out of the loops and close the
    generators, then it seems you should just ignore the exception on
    every level except for the one where you really want to handle the
    exception (possibly in an outer control-loop):

    while True:
    try:

    for result in evaluator-generator(stream-of-parsed-
    expressions):


    let the exception float up to


    >
    > OTOH, if I don't intercept the exception inside the loop, it will
    > break the for loop and close the generator. So the error no longer
    > affects a single token/expression but it kills the whole session. I
    > guess that's because the direction flow of control is sort of
    > orthogonal to the direction of flow of data.
    >
    > Any idea for a working and elegant solution?
    >
    > Thanks,
    >
    > -r
    Raymond Hettinger, Apr 8, 2011
    #2
    1. Advertising

  3. On Apr 8, 8:55 am, r <> wrote:
    > I had a problem for which I've already found a "satisfactory"
    > work-around, but I'd like to ask you if there is a better/nicer
    > looking solution. Perhaps I'm missing something obvious.
    >
    > The code looks like this:
    >
    > stream-of-tokens = token-generator(stream-of-characters)
    > stream-of-parsed-expressions = parser-generator(stream-of-tokens)
    > stream-of-results = evaluator-generator(stream-of-parsed-expressions)
    >
    > each of the above functions consumes and implements a generator:
    >
    > def evaluator-generator(stream-of-tokens):
    >   for token in stream-of-tokens:
    >     try:
    >        yield token.evaluate()           # evaluate() returns a Result
    >     except Exception as exception:
    >        yield ErrorResult(exception) # ErrorResult is a subclass of Result
    >
    > The problem is that, when I use the above mechanism, the errors
    > propagate to the output embedded in the data streams. This means, I
    > have to make them look like real data (in the example above I'm
    > wrapping the exception with an ErrorExpression object) and raise them
    > and intercept them again at each level until they finally trickle down
    > to the output. It feels a bit like writing in C (checking error codes
    > and propagating them to the caller).


    You could just let the exception go up to an outermost control-loop
    without handling it at all on a lower level. That is what exceptions
    for you: terminate all the loops, unwind the stacks, and propagate up
    to some level where the exception is caught:

    while 1:
    try:
    results = evaluator-generator(stream-of-parsed-expressions)
    for result in results:
    print(result)
    except Exception as e:
    handle_the_exception(e)

    OTOH, If you want to catch the exception at the lowest level and wrap
    it up as data (the ErrorResult in your example), there is a way to
    make it more convenient. Give the ErrorResult object some identify
    methods that correspond to the methods being called by upper levels.
    This will let the object float through without you cluttering each
    level with detect-and-reraise logic.

    class ErrorResult:
    def __iter__(self):
    # pass through an enclosing iterator
    yield self

    Here's a simple demo of how the pass through works:

    >>> from itertools import *
    >>> list(chain([1,2,3], ErrorResult(), [4,5,6]))

    [1, 2, 3, <__main__.ErrorResult object at 0x2250f70>, 4, 5, 6]


    > Any idea for a working and elegant solution?


    Hope these ideas have helped.


    Raymond
    Raymond Hettinger, Apr 8, 2011
    #3
  4. r

    r Guest

    On Sat, Apr 9, 2011 at 3:22 AM, Raymond Hettinger <> wrote:
    >
    > You could just let the exception go up to an outermost control-loop
    > without handling it at all on a lower level.  That is what exceptions
    > for you: terminate all the loops, unwind the stacks, and propagate up
    > to some level where the exception is caught:
    >
    >  while 1:
    >     try:
    >        results = evaluator-generator(stream-of-parsed-expressions)
    >        for result in results:
    >            print(result)
    >     except Exception as e:
    >        handle_the_exception(e)


    I've been thinking about something like this but the problem with
    shutting down the generators is that I lose all the state information
    associated with them (line numbers, have to reopen files if in batch
    mode etc.). It's actually more difficult than my current solution.

    > OTOH, If you want to catch the exception at the lowest level and wrap
    > it up as data (the ErrorResult in your example), there is a way to
    > make it more convenient.  Give the ErrorResult object some identify
    > methods that correspond to the methods being called by upper levels.
    > This will let the object float through without you cluttering each
    > level with detect-and-reraise logic.


    I'm already making something like this (that is, if I understand you
    correctly). In the example below (an "almost" real code this time, I
    made too many mistakes before) all the Expressions (including the
    Error one) implement an 'eval' method that gets called by one of the
    loops. So I don't have to do anything to detect the error, just have
    to catch it and reraise it at each stage so that it propagates to the
    next level (what I have to do anyway as the next level generates
    errors as well).

    class Expression(object):
    def eval(self):
    pass

    class Error(Expression):
    def __init__(self, exception):
    self.exception = exception

    def eval(self):
    raise self.exception

    def parseTokens(self, tokens):
    for token in tokens:
    try:
    yield Expression.parseToken(token, tokens)
    except ExpressionError as e:
    # here is where I wrap exceptions raised during parsing and embed them
    yield Error(e)

    def eval(expressions, frame):
    for expression in expressions:
    try:
    # and here (.eval) is where they get unwrapped and raised again
    yield unicode(expression.eval(frame))
    except ExpressionError as e:
    # here they are embedded again but because it is the last stage
    # text representation is fine
    yield unicode(e)


    >   class ErrorResult:
    >       def __iter__(self):
    >           # pass through an enclosing iterator
    >           yield self
    >
    > Here's a simple demo of how the pass through works:
    >
    >    >>> from itertools import *
    >    >>> list(chain([1,2,3], ErrorResult(), [4,5,6]))
    >    [1, 2, 3, <__main__.ErrorResult object at 0x2250f70>, 4, 5, 6]


    I don't really understand what you mean by this example. Why would
    making the Error iterable help embedding it into data stream? I'm
    currently using yield statement and it seems to work well (?).

    Anyway, thank you all for helping me out and bringing some ideas to
    the table. I was hoping there might be some pattern specifically
    designed for thiskind of job (exception generators anyone?), which
    I've overlooked. If not anything else, knowing that this isn't the
    case, makes me feel better about the solution I've chosen.

    Thanks again,

    -r
    r, Apr 8, 2011
    #4
  5. On Apr 8, 12:47 pm, r <> wrote:
    > Anyway, thank you all for helping me out and bringing some ideas to
    > the table. I was hoping there might be some pattern specifically
    > designed for thiskind of job (exception generators anyone?), which
    > I've overlooked. If not anything else, knowing that this isn't the
    > case, makes me feel better about the solution I've chosen.


    Sounds like you've gathered a bunch of good ideas and can now be
    pretty sure of your design choices.

    While it doesn't help your current use case, you might be interested
    in the iter_except() recipe in http://docs.python.org/py3k/library/itertools.html#itertools-recipes

    Raymond

    twitter: @raymondh
    Raymond Hettinger, Apr 8, 2011
    #5
  6. r

    Kent Johnson Guest

    On Apr 8, 3:47 pm, r <> wrote:
    > I'm already making something like this (that is, if I understand you
    > correctly). In the example below (an "almost" real code this time, I
    > made too many mistakes before) all the Expressions (including the
    > Error one) implement an 'eval' method that gets called by one of the
    > loops. So I don't have to do anything to detect the error, just have
    > to catch it and reraise it at each stage so that it propagates to the
    > next level (what I have to do anyway as the next level generates
    > errors as well).
    >
    > class Expression(object):
    >     def eval(self):
    >         pass
    >
    > class Error(Expression):
    >     def __init__(self, exception):
    >         self.exception = exception
    >
    >     def eval(self):
    >         raise self.exception


    Perhaps, instead of raising exceptions at each level, you could return
    an Error object that implements the eval() (or whatever appropriate
    protocol) to return an appropriate new Error object. In this case, you
    could have
    class Error(Expression);
    ....
    def eval(self):
    return unicode(self.exception)

    Error would itself be created by Expression.parseToken(), instead of
    raising an exception.

    The idea is that at each level of parsing, instead of raising an
    exception you return an object which, when interpreted at the next
    outer level, will again return an appropriate error object for that
    level. You might be able to have a single Error object which
    implements the required methods at each level to just return self.

    I don't know if this really works when you start nesting but perhaps
    it is worth a try.

    Kent
    Kent Johnson, Apr 9, 2011
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    5
    Views:
    387
    E.J. Pitt
    Aug 26, 2005
  2. Dominic
    Replies:
    1
    Views:
    337
    Scott David Daniels
    May 17, 2004
  3. Dhananjay
    Replies:
    1
    Views:
    550
    Owen Jacobson
    Nov 5, 2007
  4. ben mitch
    Replies:
    21
    Views:
    874
    Chris Dollin
    Jun 16, 2008
  5. ben mitch
    Replies:
    22
    Views:
    826
    Chris Dollin
    Jun 16, 2008
Loading...

Share This Page