Non-blocking read() in httplib?

Discussion in 'Python' started by Marcin Ciura, Jul 12, 2004.

  1. Marcin Ciura

    Marcin Ciura Guest

    I have the following problem with HTTPResponse:

    import httplib #, select
    ....
    connection = httplib.HTTPConnection(host)
    connection.connect()
    connection.request('GET', url)
    response = connection.getresponse()
    # print response.status, response.reason gives '200 OK'
    # ready = select.select([response],[],[], 5.0) # no fileno() method
    # signal.alarm(5) # not on Windows
    # ...
    data=response.read() # this sometimes blocks
    connection.close()

    Sometimes the read() call blocks forever for no obvious reason
    (response.status is OK); it even cannot be interrupted from the keyboard
    (on Windows).

    I would like to defend against this by throwing an exception when the
    read() lasts too long. But I cannot use select.select(), because
    HTTPResponse has no fileno() method. Neither can I use signal.alarm(),
    as it is for Unixes only.

    Is there any other way to break read() or make it non-blocking?

    Regards,
    Marcin
    Marcin Ciura, Jul 12, 2004
    #1
    1. Advertising

  2. Marcin Ciura wrote:
    > I have the following problem with HTTPResponse:
    >
    > import httplib #, select
    > ...
    > connection = httplib.HTTPConnection(host)
    > connection.connect()
    > connection.request('GET', url)
    > response = connection.getresponse()
    > # print response.status, response.reason gives '200 OK'
    > # ready = select.select([response],[],[], 5.0) # no fileno() method
    > # signal.alarm(5) # not on Windows
    > # ...
    > data=response.read() # this sometimes blocks
    > connection.close()
    >
    > Sometimes the read() call blocks forever for no obvious reason
    > (response.status is OK); it even cannot be interrupted from the keyboard
    > (on Windows).
    >
    > I would like to defend against this by throwing an exception when the
    > read() lasts too long. But I cannot use select.select(), because
    > HTTPResponse has no fileno() method. Neither can I use signal.alarm(),
    > as it is for Unixes only.
    >
    > Is there any other way to break read() or make it non-blocking?

    I'm using timeoutsocket.py which allows you to globally specify a
    timeout for all sockets. Used signal.alarm before, which caused more
    problems than it solved...
    Benjamin Niemann, Jul 12, 2004
    #2
    1. Advertising

  3. Marcin Ciura

    Jarek Zgoda Guest

    Marcin Ciura <> pisze:

    > Sometimes the read() call blocks forever for no obvious reason
    > (response.status is OK); it even cannot be interrupted from the keyboard
    > (on Windows).
    >
    > I would like to defend against this by throwing an exception when the
    > read() lasts too long. But I cannot use select.select(), because
    > HTTPResponse has no fileno() method. Neither can I use signal.alarm(),
    > as it is for Unixes only.
    >
    > Is there any other way to break read() or make it non-blocking?


    You can try to write your own HTTP client module, that uses
    asyncore/asynchat. You may find a minimal implementation in my project's
    CVS (URL in signature). It is based on code found in "EffNews Part 1:
    Fetching RSS Files" tutorial (http://www.effbot.org/zone/effnews-1.htm).

    --
    Jarek Zgoda
    http://jpa.berlios.de/
    Jarek Zgoda, Jul 12, 2004
    #3
  4. Marcin Ciura

    Paul Rubin Guest

    Marcin Ciura <> writes:
    > I would like to defend against this by throwing an exception when the
    > read() lasts too long. But I cannot use select.select(), because
    > HTTPResponse has no fileno() method. Neither can I use signal.alarm(),
    > as it is for Unixes only.
    >
    > Is there any other way to break read() or make it non-blocking?


    I think I'd reach down into the response object and get the fileno.
    From httplib.py:

    class HTTPResponse:
    # ...
    def __init__(self, sock, debuglevel=0, strict=0):
    self.fp = sock.makefile('rb', 0)
    self.debuglevel = debuglevel
    self.strict = strict

    so maybe you can get the fileno from response.fp.
    Paul Rubin, Jul 12, 2004
    #4
  5. Marcin Ciura

    John J. Lee Guest

    Jarek Zgoda <> writes:
    [...]
    > You can try to write your own HTTP client module, that uses
    > asyncore/asynchat. You may find a minimal implementation in my project's
    > CVS (URL in signature). It is based on code found in "EffNews Part 1:
    > Fetching RSS Files" tutorial (http://www.effbot.org/zone/effnews-1.htm).


    There used to be one called asynchttp.py. There was also an
    asyncurl.py. I think both are likely still around somewhere, but not
    maintained.


    John
    John J. Lee, Jul 13, 2004
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hendra Gunawan
    Replies:
    1
    Views:
    12,442
    Allan Herriman
    Apr 8, 2004
  2. Andre Kelmanson

    blocking i/o vs. non blocking i/o (performance)

    Andre Kelmanson, Oct 10, 2003, in forum: C Programming
    Replies:
    3
    Views:
    917
    Valentin Tihomirov
    Oct 12, 2003
  3. nukleus
    Replies:
    14
    Views:
    817
    Chris Uppal
    Jan 22, 2007
  4. Christian
    Replies:
    5
    Views:
    723
    Esmond Pitt
    Dec 2, 2007
  5. Serge Savoie
    Replies:
    4
    Views:
    256
    Serge Savoie
    Oct 1, 2008
Loading...

Share This Page