how to handle network failures

Discussion in 'Python' started by harryos, Oct 8, 2010.

  1. harryos

    harryos Guest

    hi
    I am trying to write a DataGrabber which reads some data from given
    url..I made DataGrabber as a Thread and want to wait for some interval
    of time in case there is a network failure that prevents read().
    I am not very sure how to implement this

    class DataGrabber(threading.Thread):
    def __init__(self,url):
    threading.Thread.__init__(self)
    self.url=url
    def run(self):
    data=self.get_page_data()
    process_data(data)

    def get_page_data():
    try:
    f=urllib.urlopen(self.url)
    data=f.read(1024)
    except IOError:
    #wait for some time and try again
    time.sleep(120)
    data=self.get_page_data()
    return data

    Is this the way to implement the part where the thread waits and
    reads the data again? Will this handle network failures?Can somebody
    please help?

    thanks
    harry
    harryos, Oct 8, 2010
    #1
    1. Advertising

  2. harryos <> writes:

    > hi
    > I am trying to write a DataGrabber which reads some data from given
    > url..I made DataGrabber as a Thread and want to wait for some interval
    > of time in case there is a network failure that prevents read().
    > I am not very sure how to implement this
    >
    > class DataGrabber(threading.Thread):
    > def __init__(self,url):
    > threading.Thread.__init__(self)
    > self.url=url
    > def run(self):
    > data=self.get_page_data()
    > process_data(data)
    >
    > def get_page_data():
    > try:
    > f=urllib.urlopen(self.url)
    > data=f.read(1024)
    > except IOError:
    > #wait for some time and try again
    > time.sleep(120)
    > data=self.get_page_data()
    > return data
    >
    > Is this the way to implement the part where the thread waits and
    > reads the data again? Will this handle network failures?Can somebody
    > please help?


    This only works if your page is always 1024 bytes long. Which I
    doubt. So don't pass the 1024 to read.

    Also, you need a loop to re-read the data. Like this:


    for n in xrange(max_number_of_retries):
    try:
    f=urllib.urlopen(self.url)
    data = f.read()
    break # exist the loop if all
    except IOError:
    pass


    self.process_data(data)


    Diez
    Diez B. Roggisch, Oct 9, 2010
    #2
    1. Advertising

  3. In message <>, Diez B. Roggisch wrote:

    > for n in xrange(max_number_of_retries):
    > try:
    > f=urllib.urlopen(self.url)
    > data = f.read()
    > break # exist the loop if all
    > except IOError:
    > pass


    Is it worth delaying before retrying? In case of a transient routing error,
    that kind of thing.
    Lawrence D'Oliveiro, Oct 10, 2010
    #3
  4. harryos

    Jorgen Grahn Guest

    On Fri, 2010-10-08, harryos wrote:
    > hi
    > I am trying to write a DataGrabber which reads some data from given
    > url..I made DataGrabber as a Thread and want to wait for some interval
    > of time in case there is a network failure that prevents read().
    > I am not very sure how to implement this
    >
    > class DataGrabber(threading.Thread):
    > def __init__(self,url):
    > threading.Thread.__init__(self)
    > self.url=url
    > def run(self):
    > data=self.get_page_data()
    > process_data(data)
    >
    > def get_page_data():
    > try:
    > f=urllib.urlopen(self.url)
    > data=f.read(1024)
    > except IOError:
    > #wait for some time and try again
    > time.sleep(120)
    > data=self.get_page_data()
    > return data
    >
    > Is this the way to implement the part where the thread waits and
    > reads the data again? Will this handle network failures?Can somebody
    > please help?


    You are using TCP sockets. When you get an error on one of those, the
    TCP connection is dead (except for a few special cases like EAGAIN,
    EINTR).

    But you also risk *not* getting told and hanging forever, or anyway
    for far longer than your application is likely to want to wait. For
    example if the peer host is suddenly disconnected from the network --
    TCP will keep trying, in case a connection suddenly reappears much
    later.

    Try provoking that situation and see what happens.

    /Jorgen

    --
    // Jorgen Grahn <grahn@ Oo o. . .
    \X/ snipabacken.se> O o .
    Jorgen Grahn, Oct 10, 2010
    #4
  5. harryos

    Aahz Guest

    In article <>,
    harryos <> wrote:
    >
    >class DataGrabber(threading.Thread):
    > def __init__(self,url):
    > threading.Thread.__init__(self)
    > self.url=url
    > def run(self):
    > data=self.get_page_data()
    > process_data(data)
    >
    > def get_page_data():
    > try:
    > f=urllib.urlopen(self.url)
    > data=f.read(1024)
    > except IOError:
    > #wait for some time and try again
    > time.sleep(120)
    > data=self.get_page_data()
    > return data


    Use urllib2 so that you can set a timeout (Python 2.6+).
    --
    Aahz () <*> http://www.pythoncraft.com/

    "If you think it's expensive to hire a professional to do the job, wait
    until you hire an amateur." --Red Adair
    Aahz, Nov 7, 2010
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kevin Spencer

    Re: So many failures early on

    Kevin Spencer, Aug 13, 2003, in forum: ASP .Net
    Replies:
    4
    Views:
    327
    Kevin Spencer
    Aug 27, 2003
  2. Guest
    Replies:
    0
    Views:
    408
    Guest
    Jun 19, 2004
  3. tommy
    Replies:
    0
    Views:
    351
    tommy
    Jul 19, 2004
  4. jlambrecht
    Replies:
    0
    Views:
    551
    jlambrecht
    Dec 22, 2004
  5. MikeB
    Replies:
    4
    Views:
    340
    Owen Jacobson
    Oct 26, 2004
Loading...

Share This Page