Asynchronous HTTP client

[Ping]

Hi,

I'm trying to find a way to create an asynchronous HTTP client so I
can get responses from web servers in a way like

async_http_open('http://example.com/', callback_func)
# immediately continues, and callback_func is called with response
as arg when it is ready

It seems twisted can do it, but I hesitate to bring in such a big
package as a dependency because my client should be light. Asyncore
and asynchat are lighter but they don't speak HTTP. The asynchttp
project on sourceforge is a fusion between asynchat and httplib, but
it hasn't been updated since 2001 and is seriously out of sync with
httplib.

I'd appreciate it if anyone can shed some lights on this.

Thanks,
Ping

 

[Lie Ryan]

Hi,

I'm trying to find a way to create an asynchronous HTTP client so I
can get responses from web servers in a way like

async_http_open('http://example.com/', callback_func)
# immediately continues, and callback_func is called with response
as arg when it is ready

It seems twisted can do it, but I hesitate to bring in such a big
package as a dependency because my client should be light. Asyncore
and asynchat are lighter but they don't speak HTTP. The asynchttp
project on sourceforge is a fusion between asynchat and httplib, but
it hasn't been updated since 2001 and is seriously out of sync with
httplib.

I'd appreciate it if anyone can shed some lights on this.

If you want something quite lightweight, you can spawn a thread for the
call:

import threading, urllib
class AsyncOpen(threading.Thread):
def __init__(self, url, callback):
super(AsyncOpen, self).__init__()
self.url = url
self.callback = callback
def run(self):
# of course change urllib to httplib-something-something
content = urllib.urlopen(self.url).read()
self.callback(content)

def asyncopen(url, callback):
AsyncOpen(url, callback).start()

def cb(content):
print content

asyncopen('http://www.google.com', cb)

 

[exarkun]

Hi,

I'm trying to find a way to create an asynchronous HTTP client so I
can get responses from web servers in a way like

async_http_open('http://example.com/', callback_func)
# immediately continues, and callback_func is called with response
as arg when it is ready

It seems twisted can do it, but I hesitate to bring in such a big
package as a dependency because my client should be light. Asyncore
and asynchat are lighter but they don't speak HTTP. The asynchttp
project on sourceforge is a fusion between asynchat and httplib, but
it hasn't been updated since 2001 and is seriously out of sync with
httplib.

Why should it be "light"? In what way would using Twisted cause
problems for you?

Jean-Paul

 

[exarkun]

I'm writing an open source python client for a web service. The client
may
be used in all kinds of environments - Linux, Mac OS X, Windows, web
hosting, etc by others. It is not impossible to have twisted as a
dependency, but that makes deployment a larger job than simply
uploading a
Python file.

Twisted is packaged for many Linux distributions (perhaps most of them).
Many web hosts provide it. It's also shipped with OS X.

Windows may be an issue, but note that there's a binary Windows
installer (as of 10.0, an MSI, so installation can be easily automated).

I'm willing to use twisted, but I'd like to explore lighter
alternatives
first.

Jean-Paul

 

[Antoine Pitrou]

Le Sun, 07 Mar 2010 22:40:36 +0800, pingooo a écrit :

I'm writing an open source python client for a web service. The client
may be used in all kinds of environments - Linux, Mac OS X, Windows, web
hosting, etc by others. It is not impossible to have twisted as a
dependency, but that makes deployment a larger job than simply uploading
a Python file.

If it can be used by non-Python users, you will have to package it using
py2exe or py2app anyway, in which case Twisted will be bundled
automatically and the size overhead won't be very large.