K
Kottiyath
Hi all,
I have to connect to a secure website every second to get the data
and then post to it. I have been investigating on many web clients in
python, but nothing fits the bill properly.
The ones I tried implementing are:
1. httplib based - I created myself. (I cannot use urllib2 since I
have to transfer files, and urllib2 doesnt have multipart content-type
support)
2. Twisted web client.
I also looked at mechanize etc too.
The problems I face are -
1. I liked twisted a lot, but when I implemented it, I found that
client support is there only for twisted.web and not twisted.web2.
Since I connect to the same website every time, I would like to have
persistent connections and since twisted.web is HTTP/1.0, persistent
connection support is not yet there. Without persistent connections, I
would have to have TCP connection handshake everytime and it is taking
too much time.
2. Since I connect to the website every second, I have to have
many connections running at the same time. I am worried that creating
threads for each connection is going to be a big problem (esp if the
server response is slow), since the processor will get swamped -
especially since there are many other activities going on in the
machine.
3. I would also like to pipe line the requests - esp if the
response is slow.
Other requirements:
1. HTTPS Support
2. Connection through proxy.
Is there any good web client which I can use straight up? Or would
I have to implement the whole thing myself? It looks like a big beast
and I was wondering whether python provides it straight up.
Regards
K
I have to connect to a secure website every second to get the data
and then post to it. I have been investigating on many web clients in
python, but nothing fits the bill properly.
The ones I tried implementing are:
1. httplib based - I created myself. (I cannot use urllib2 since I
have to transfer files, and urllib2 doesnt have multipart content-type
support)
2. Twisted web client.
I also looked at mechanize etc too.
The problems I face are -
1. I liked twisted a lot, but when I implemented it, I found that
client support is there only for twisted.web and not twisted.web2.
Since I connect to the same website every time, I would like to have
persistent connections and since twisted.web is HTTP/1.0, persistent
connection support is not yet there. Without persistent connections, I
would have to have TCP connection handshake everytime and it is taking
too much time.
2. Since I connect to the website every second, I have to have
many connections running at the same time. I am worried that creating
threads for each connection is going to be a big problem (esp if the
server response is slow), since the processor will get swamped -
especially since there are many other activities going on in the
machine.
3. I would also like to pipe line the requests - esp if the
response is slow.
Other requirements:
1. HTTPS Support
2. Connection through proxy.
Is there any good web client which I can use straight up? Or would
I have to implement the whole thing myself? It looks like a big beast
and I was wondering whether python provides it straight up.
Regards
K