Making a persistent HTTP connection

D

David Rasmussen

I use urllib2 to do some simple HTTP communication with a web server. In
one "session", I do maybe 10-15 requests. It seems that urllib2 opens op
a connection every time I do a request. Can I somehow make it use _one_
persistent connection where I can do multiple GET->"receive data" passes
before the connection is closed?

/David
 
D

Diez B. Roggisch

David said:
I use urllib2 to do some simple HTTP communication with a web server. In
one "session", I do maybe 10-15 requests. It seems that urllib2 opens op
a connection every time I do a request. Can I somehow make it use _one_
persistent connection where I can do multiple GET->"receive data" passes
before the connection is closed?

Are you sure HTTP supports that? This would be news to me - which
doesn't mean much :)

And even if it works - what is the problem with connections being created?

Regards,

Diez
 
B

Benjamin Niemann

Diez said:
Are you sure HTTP supports that? This would be news to me - which
doesn't mean much :)

It does (HTTP/1.1 at least) and it's called 'keep-alive'.
And even if it works - what is the problem with connections being created?

Performance, network load...
 
A

Alan Kennedy

[David Rasmussen]
[Diez B. Roggisch]
Are you sure HTTP supports that?

Yes, HTTP 1.1 definitely supports multiple requests on the same connection.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1

Some HTTP 1.0 clients supported persistent connections through the use
of the non-standard "keep-alive" header.
And even if it works - what is the problem with connections being created?

The URL above describes the benefits of persistent connections. The
primary problem of the old style of one-request-per-connection is the
creation of more sockets than are necessary.

To the OP: neither urllib nor urllib2 implements persistent connections,
but httplib does. See the httplib documentation page for an example.

http://www.python.org/doc/2.4.2/lib/httplib-examples.html

However, even httplib is "synchronous", in that it cannot pipeline
requests: the response to the first request must be competely read
before a second request can be issued.

HTH,
 
A

Alan Kennedy

[David Rasmussen]
[Diez B. Roggisch]
Are you sure HTTP supports that?

Yes, HTTP 1.1 definitely supports multiple requests on the same
connection.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1

Some HTTP 1.0 clients supported persistent connections through the use
of the non-standard "keep-alive" header.
And even if it works - what is the problem with connections being created?

The URL above describes the benefits of persistent connections. The
primary problem of the old style of one-request-per-connection is the
creation of more sockets than are necessary.

To the OP: neither urllib nor urllib2 implements persistent
connections, but httplib does. See the httplib documentation page for
an example.

http://www.python.org/doc/2.4.2/lib/httplib-examples.html

However, even httplib is "synchronous", in that it cannot pipeline
requests: the response to the first request must be competely read
before a second request can be issued.

HTH,
 
P

Piet van Oostrum

Alan Kennedy said:
AK> Some HTTP 1.0 clients supported persistent connections through the use
AK> of the non-standard "keep-alive" header.
AK> The URL above describes the benefits of persistent connections. The
AK> primary problem of the old style of one-request-per-connection is the
AK> creation of more sockets than are necessary.

Maybe even more important (and just briefly mentioned in the section
referred to above) is the latency introduced by the TCP setup and the slow
startup phase of TCP's congestion control. This calculation is one of the
exercises the students have to make in my networks class.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top