how to handle network failures

[harryos]

hi
I am trying to write a DataGrabber which reads some data from given
url..I made DataGrabber as a Thread and want to wait for some interval
of time in case there is a network failure that prevents read().
I am not very sure how to implement this

class DataGrabber(threading.Thread):
def __init__(self,url):
threading.Thread.__init__(self)
self.url=url
def run(self):
data=self.get_page_data()
process_data(data)

def get_page_data():
try:
f=urllib.urlopen(self.url)
data=f.read(1024)
except IOError:
#wait for some time and try again
time.sleep(120)
data=self.get_page_data()
return data

Is this the way to implement the part where the thread waits and
reads the data again? Will this handle network failures?Can somebody
please help?

thanks
harry

 

[Diez B. Roggisch]

hi
I am trying to write a DataGrabber which reads some data from given
url..I made DataGrabber as a Thread and want to wait for some interval
of time in case there is a network failure that prevents read().
I am not very sure how to implement this

class DataGrabber(threading.Thread):
def __init__(self,url):
threading.Thread.__init__(self)
self.url=url
def run(self):
data=self.get_page_data()
process_data(data)

def get_page_data():
try:
f=urllib.urlopen(self.url)
data=f.read(1024)
except IOError:
#wait for some time and try again
time.sleep(120)
data=self.get_page_data()
return data

Is this the way to implement the part where the thread waits and
reads the data again? Will this handle network failures?Can somebody
please help?

This only works if your page is always 1024 bytes long. Which I
doubt. So don't pass the 1024 to read.

Also, you need a loop to re-read the data. Like this:


for n in xrange(max_number_of_retries):
try:
f=urllib.urlopen(self.url)
data = f.read()
break # exist the loop if all
except IOError:
pass


self.process_data(data)


Diez

 

[Lawrence D'Oliveiro]

for n in xrange(max_number_of_retries):
try:
f=urllib.urlopen(self.url)
data = f.read()
break # exist the loop if all
except IOError:
pass

Is it worth delaying before retrying? In case of a transient routing error,
that kind of thing.

 

[Jorgen Grahn]

hi
I am trying to write a DataGrabber which reads some data from given
url..I made DataGrabber as a Thread and want to wait for some interval
of time in case there is a network failure that prevents read().
I am not very sure how to implement this

class DataGrabber(threading.Thread):
def __init__(self,url):
threading.Thread.__init__(self)
self.url=url
def run(self):
data=self.get_page_data()
process_data(data)

def get_page_data():
try:
f=urllib.urlopen(self.url)
data=f.read(1024)
except IOError:
#wait for some time and try again
time.sleep(120)
data=self.get_page_data()
return data

Is this the way to implement the part where the thread waits and
reads the data again? Will this handle network failures?Can somebody
please help?

You are using TCP sockets. When you get an error on one of those, the
TCP connection is dead (except for a few special cases like EAGAIN,
EINTR).

But you also risk *not* getting told and hanging forever, or anyway
for far longer than your application is likely to want to wait. For
example if the peer host is suddenly disconnected from the network --
TCP will keep trying, in case a connection suddenly reappears much
later.

Try provoking that situation and see what happens.

/Jorgen

 

[Aahz]

class DataGrabber(threading.Thread):
def __init__(self,url):
threading.Thread.__init__(self)
self.url=url
def run(self):
data=self.get_page_data()
process_data(data)

def get_page_data():
try:
f=urllib.urlopen(self.url)
data=f.read(1024)
except IOError:
#wait for some time and try again
time.sleep(120)
data=self.get_page_data()
return data

Use urllib2 so that you can set a timeout (Python 2.6+).