Timeout in urllib2

D

Dan M

I'm writing a system monitor script that needs to load web pages. I'm
using urllib2.urlopen to get the pages, and I'm attempting to set the
timeout value using socket.defaulttimeout.

Specifically, I'm calling socket.defaultttimeout(10), then calling
urllib2.urlopen to fetch a web page that never gets delivered. My code
waits about 30 seconds before terminating.

I am about to add threading to my app so that delays on a few servers
won't be a critical issue, but I'd still like to understand why the call
to socket.defaulttimeout doesn't affect the timeout on my urlopen calls.

My code follows.

#!/usr/local/bin/python
import socket, time

socket.setdefaulttimeout(10)
import urllib2

def doHttpTest():
url = "http://url.that.never.returns"
t_start = time.time()
if httptest(url):
print "Error on site ", url
t_end = time.time()
t_diff = t_end - t_start

def httptest(url):
timeout = 10
socket.setdefaulttimeout(timeout)
try:
req = urllib2.Request(url)
urllib2.urlopen(req)
except urllib2.HTTPError, e:
if e.code == 401:
return 1
elif e.code == 404:
return 1
elif e.code == 503:
return 1
else:
return 1
except urllib2.URLError, e:
return 1
else:
return 0



if __name__ == '__main__':
try:
doHttpTest()
except KeyboardInterrupt:
print "Exiting..."
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,905
Latest member
Kristy_Poole

Latest Threads

Top