http pipelining

[swq22]

Which python module is capable of pipelining http requests?

(I know httplib can send mulitple requests per tcp connection, but in
a strictly serial way. )

 

[Steve Holden]

Which python module is capable of pipelining http requests?

(I know httplib can send mulitple requests per tcp connection, but in
a strictly serial way. )

There's nothing in the standard library, I believe, that includes both
client and server functionality in the same module. So you would need to
glue them together.

If you want a simple net proxy server I seem to remember the chameleon
system allows you to write one in about twelve lines. If it has to be
HTTP-specific, with header parsing and the like, you might want to think
about Twisted, which supports both client and server functionality and
tries to make it easy to plumb things together in pipelines.

regards
Steve

 

[Steve Holden]

Which python module is capable of pipelining http requests?

(I know httplib can send mulitple requests per tcp connection, but in
a strictly serial way. )

Oops, sorry, you meant sending requests in parallel, right?

You'll need to use either urllib or urllib2 for the web, and the
threading module is one way to run parallel requests. It's fairly easy
to use as long as you keep your tasks properly isolated form each other.

regards
Steve

 

[swq22]

There's nothing in the standard library, I believe, that includes both
client and server functionality in the same module. So you would need to
glue them together.

If you want a simple net proxy server I seem to remember the chameleon
system allows you to write one in about twelve lines. If it has to be
HTTP-specific, with header parsing and the like, you might want to think
about Twisted, which supports both client and server functionality and
tries to make it easy to plumb things together in pipelines.

Sorry for my confused question.

What I'm looking for is to emulate what a modern HTTP 1.1 browser like
Firefox does(when network.http.pipelining is enabled)

1.Open TCP connetion.

send multiple requests without waiting for a reply:

2.GET /url1
3.GET /url2
4.GET /url3
5.read 1st reply
6.read 2st reply
7.read 3st reply

8.Close TCP connection.

I'll be using Apache or IIS as my web server.

 

[John J. Lee]

Oops, sorry, you meant sending requests in parallel, right?

You'll need to use either urllib or urllib2 for the web, and the
threading module is one way to run parallel requests. It's fairly easy
to use as long as you keep your tasks properly isolated form each
other.

No, he means "HTTP pipelining", which means sending multiple requests
down a single TCP connection, without waiting for the first response.

httplib's module-level docstring says (reformatted here):

"""
.... The HTTPResponse class does not enforce this state machine, which
implies sophisticated clients may accelerate the request/response
pipeline. Caution should be taken, though: accelerating the states
beyond the above pattern may imply knowledge of the server's
connection-close behavior for certain requests. For example, it is
impossible to tell whether the server will close the connection UNTIL
the response headers have been read; this means that further requests
cannot be placed into the pipeline until it is known that the server
will NOT be closing the connection.
"""

So, sort-of-yes, if you know what you're doing and you're lucky.

Certainly urllib and urllib2 don't support pipelining. There were
plans for a new HTTP client in Twisted with pipelining support, but I
don't know if that ever came about. AFAIK not many libraries (in any
language) support it -- e.g. none of "Jakarta commons HTTPClient",
libwww-perl, and libcurl currently support it. libwww (without the
"-perl") does claim to support it (I say "claim" merely because I
haven't used it or read the source -- no FUD intended).


Side note: As the OP mentions in a followup, by default firefox does
NOT do pipelining (to the disbelief of the people I told about this
when it came up in a previous job -- but I just tried running tcpdump
and indeed about:config seems to be telling the truth; fortunately, in
response to the limitations imposed by RFC 2616, somebody has
thoughtfully arranged for the speed of light to be fast enough for
this not to be a major problem when using websites on the other side
of the planet). The firefox people say:

http://www.mozilla.org/support/firefox/tips#oth_pipelining

"""
Pipelining is an experimental feature, designed to improve page-load
performance, that is unfortunately not well supported by some web
servers and proxies.
"""

Instead of pipelining, it uses multiple connections (2 when I tried
it, which is the maximum RFC 2616 says SHOULD be used by a
"single-user client"). I didn't try IE, but apparently it has the
same behaviour (2 connections, no pipelining):

http://blogs.msdn.com/ie/archive/2005/04/11/407189.aspx


I wonder if the right economic pressures are there for SCTP ever to
get used for everyday web HTTP stuff...


John