Re: retry many times decorator

Discussion in 'Python' started by andrea crotti, Jun 28, 2012.

  1. > Returning a boolean isn't very Pythonic. It would be better, IMHO, if
    > it could swallow a specified exception (or specified exceptions?)
    > raised when an attempt failed, up to the maximum permitted number of
    > attempts. If the final attempt fails, propagate the exception.
    > --
    > http://mail.python.org/mailman/listinfo/python-list



    Yes right I also didn't like it.. Now it become something as below,
    so I capture every possible exception (since it must be generic) and
    log what exception was raised. I'm not re-raising because if it fails
    and it's fatal I should just exit, and if it's not fatal it should
    just continue..

    class retry_n_times:
    def __init__(self, ntimes=3, timeout=3, fatal=True):
    self.ntimes = ntimes
    self.timeout = timeout
    self.fatal = fatal

    def __call__(self, func):
    def _retry_n_times(*args, **kwargs):
    attempts = 0
    while True:
    logger.debug("Attempt number %s of %s" % (attempts,
    func.__name__))
    try:
    ret = func(*args, **kwargs)
    except Exception as e:
    logger.error("Got exception %s with error %s" %
    (type(e), str(e)))
    sleep(self.timeout)
    else:
    return ret

    attempts += 1
    if attempts == self.ntimes:
    logger.error("Giving up the attempts while running
    %s" % func.__name__)
    if self.fatal:
    exit(100)

    return _retry_n_times
     
    andrea crotti, Jun 28, 2012
    #1
    1. Advertising

  2. On Thu, 28 Jun 2012 17:26:36 +0100, andrea crotti wrote:

    >> Returning a boolean isn't very Pythonic. It would be better, IMHO, if
    >> it could swallow a specified exception (or specified exceptions?)
    >> raised when an attempt failed, up to the maximum permitted number of
    >> attempts. If the final attempt fails, propagate the exception. --
    >> http://mail.python.org/mailman/listinfo/python-list

    >
    >
    > Yes right I also didn't like it.. Now it become something as below, so
    > I capture every possible exception (since it must be generic)


    I disagree. If you make a coding error in your function, why do you think
    it is useful to retry that buggy code over and over again? It's never
    going to get less buggy unless you see the exception and fix the bug.

    For any operation that you want to retry, identify the *temporary*
    errors, catch them, and retry the request. *Permanent* errors should
    immediately fail, without retrying. *Unexpected* errors should not be
    caught, since they probably represent a bug in your code.

    E.g. when trying to read a HTTP resource, there is no point retrying a
    400 Bad Request repeatedly, and it is rather anti-social to keep
    hammering a website when it's already told you that your syntax is wrong.
    Likewise, if you get 401, there is no point in retrying with the exact
    same request -- you need a *different* request, this time using
    authentication. A 403 is a permanent error and there's no point in
    retrying. A 404 might be temporary (although it probably isn't). A 503 or
    504 is likely to be temporary. And so forth.

    http://en.wikipedia.org/wiki/List_of_HTTP_status_codes

    And if you get an unexpected AttributeError: 'NoneType' object has no
    attribute 'read' (as I just did today, in one of my scripts), that's a
    bug that should be fixed.

    Almost any other resource you might try is likely to have similar
    responses: some errors will indicate temporary failures, others will be
    permanent, and some will indicate a bug in your code. I think it is rude
    and pointless to repeatedly retry the same failed request if it cannot
    possibly succeed.

    I also find that exponential backoff for the delay is best. E.g. wait 1
    second after the first failed attempt, 2 seconds after the second, 4
    seconds after the third, 8 seconds after the fourth, etc.


    --
    Steven
     
    Steven D'Aprano, Jun 28, 2012
    #2
    1. Advertising

  3. On 06/28/2012 06:43 PM, Steven D'Aprano wrote:
    > On Thu, 28 Jun 2012 17:26:36 +0100, andrea crotti wrote:
    >
    >
    > I disagree. If you make a coding error in your function, why do you think
    > it is useful to retry that buggy code over and over again? It's never
    > going to get less buggy unless you see the exception and fix the bug.
    >
    > For any operation that you want to retry, identify the *temporary*
    > errors, catch them, and retry the request. *Permanent* errors should
    > immediately fail, without retrying. *Unexpected* errors should not be
    > caught, since they probably represent a bug in your code.


    Ah well maybe I wasn't clear, but I'm not going to retry random things,
    I will only decorate
    the functions that I know for sure that could go wrong for temporary
    network problems.

    For example they told me that sometimes mysql just doesn't respond in
    time for some reasons,
    but there's nothing permanently wrong, so retrying is the best option..

    It would be good of course, however, to catch the exceptions that are
    known to be permanent problems
    in the function at least, and leave the retry as last resource..

    Thanks for the idea of the exponential backoff, which is also a better
    name than timeout for the variable..
     
    Andrea Crotti, Jun 28, 2012
    #3
  4. On the other hand now that I think again even supposing there is a
    permanent error like MySql completely down, retrying continuosly
    won't do any harm anyway because the machine will not be able to do
    anything else anyway, when someone will fix MySql it would
    restart again without human intervention.

    So I think I could even just let it retry and use maybe a SMTPHanlder
    for the logging errors, to make the notification of problems very
    quick..
     
    andrea crotti, Jun 29, 2012
    #4
  5. On Friday, June 29, 2012 4:53:43 AM UTC-4, andrea crotti wrote:
    > On the other hand now that I think again even supposing there is a
    > permanent error like MySql completely down, retrying continuosly
    > won't do any harm anyway because the machine will not be able to do
    > anything else anyway, when someone will fix MySql it would
    > restart again without human intervention.
    >
    > So I think I could even just let it retry and use maybe a SMTPHanlder
    > for the logging errors, to make the notification of problems very
    > quick..


    Rather then write a decorator, sounds like you should write your own class or method connecting to the database that has the ability to retry. Are you finding you are decorating only db calls or are you decorating all sorts of stuff?
     
    Justin Barber, Jun 29, 2012
    #5
  6. On Friday, June 29, 2012 4:53:43 AM UTC-4, andrea crotti wrote:
    > On the other hand now that I think again even supposing there is a
    > permanent error like MySql completely down, retrying continuosly
    > won't do any harm anyway because the machine will not be able to do
    > anything else anyway, when someone will fix MySql it would
    > restart again without human intervention.
    >
    > So I think I could even just let it retry and use maybe a SMTPHanlder
    > for the logging errors, to make the notification of problems very
    > quick..


    Rather then write a decorator, sounds like you should write your own class or method connecting to the database that has the ability to retry. Are you finding you are decorating only db calls or are you decorating all sorts of stuff?
     
    Justin Barber, Jun 29, 2012
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?bWF2cmlja18xMDE=?=

    SetAuthCookie works some times and fails some times?

    =?Utf-8?B?bWF2cmlja18xMDE=?=, Mar 23, 2006, in forum: ASP .Net
    Replies:
    0
    Views:
    516
    =?Utf-8?B?bWF2cmlja18xMDE=?=
    Mar 23, 2006
  2. =?Utf-8?B?bWF2cmlja18xMDE=?=

    Forms Authentication Fails some times and not some times???

    =?Utf-8?B?bWF2cmlja18xMDE=?=, Mar 28, 2006, in forum: ASP .Net
    Replies:
    0
    Views:
    495
    =?Utf-8?B?bWF2cmlja18xMDE=?=
    Mar 28, 2006
  3. glomde
    Replies:
    5
    Views:
    524
    glomde
    Mar 29, 2007
  4. Dan
    Replies:
    7
    Views:
    159
    Marc Heiler
    Feb 23, 2010
  5. MRAB
    Replies:
    0
    Views:
    170
Loading...

Share This Page