Non-blocking read() in httplib?

[Marcin Ciura]

I have the following problem with HTTPResponse:

import httplib #, select
....
connection = httplib.HTTPConnection(host)
connection.connect()
connection.request('GET', url)
response = connection.getresponse()
# print response.status, response.reason gives '200 OK'
# ready = select.select([response],[],[], 5.0) # no fileno() method
# signal.alarm(5) # not on Windows
# ...
data=response.read() # this sometimes blocks
connection.close()

Sometimes the read() call blocks forever for no obvious reason
(response.status is OK); it even cannot be interrupted from the keyboard
(on Windows).

I would like to defend against this by throwing an exception when the
read() lasts too long. But I cannot use select.select(), because
HTTPResponse has no fileno() method. Neither can I use signal.alarm(),
as it is for Unixes only.

Is there any other way to break read() or make it non-blocking?

Regards,
Marcin

 

[Benjamin Niemann]

I have the following problem with HTTPResponse:

import httplib #, select
...
connection = httplib.HTTPConnection(host)
connection.connect()
connection.request('GET', url)
response = connection.getresponse()
# print response.status, response.reason gives '200 OK'
# ready = select.select([response],[],[], 5.0) # no fileno() method
# signal.alarm(5) # not on Windows
# ...
data=response.read() # this sometimes blocks
connection.close()

Sometimes the read() call blocks forever for no obvious reason
(response.status is OK); it even cannot be interrupted from the keyboard
(on Windows).

I would like to defend against this by throwing an exception when the
read() lasts too long. But I cannot use select.select(), because
HTTPResponse has no fileno() method. Neither can I use signal.alarm(),
as it is for Unixes only.

Is there any other way to break read() or make it non-blocking?

I'm using timeoutsocket.py which allows you to globally specify a
timeout for all sockets. Used signal.alarm before, which caused more
problems than it solved...

 

[Jarek Zgoda]

Sometimes the read() call blocks forever for no obvious reason
(response.status is OK); it even cannot be interrupted from the keyboard
(on Windows).

I would like to defend against this by throwing an exception when the
read() lasts too long. But I cannot use select.select(), because
HTTPResponse has no fileno() method. Neither can I use signal.alarm(),
as it is for Unixes only.

Is there any other way to break read() or make it non-blocking?

You can try to write your own HTTP client module, that uses
asyncore/asynchat. You may find a minimal implementation in my project's
CVS (URL in signature). It is based on code found in "EffNews Part 1:
Fetching RSS Files" tutorial (http://www.effbot.org/zone/effnews-1.htm).

 

[Paul Rubin]

I would like to defend against this by throwing an exception when the
read() lasts too long. But I cannot use select.select(), because
HTTPResponse has no fileno() method. Neither can I use signal.alarm(),
as it is for Unixes only.

Is there any other way to break read() or make it non-blocking?

I think I'd reach down into the response object and get the fileno.
From httplib.py:

class HTTPResponse:
# ...
def __init__(self, sock, debuglevel=0, strict=0):
self.fp = sock.makefile('rb', 0)
self.debuglevel = debuglevel
self.strict = strict

so maybe you can get the fileno from response.fp.

 

[John J. Lee]

You can try to write your own HTTP client module, that uses
asyncore/asynchat. You may find a minimal implementation in my project's
CVS (URL in signature). It is based on code found in "EffNews Part 1:
Fetching RSS Files" tutorial (http://www.effbot.org/zone/effnews-1.htm).

There used to be one called asynchttp.py. There was also an
asyncurl.py. I think both are likely still around somewhere, but not
maintained.


John