http pipelining

Discussion in 'Python' started by swq22@yahoo.com, Apr 26, 2007.

  1. Guest

    Which python module is capable of pipelining http requests?

    (I know httplib can send mulitple requests per tcp connection, but in
    a strictly serial way. )
     
    , Apr 26, 2007
    #1
    1. Advertising

  2. Steve Holden Guest

    wrote:
    > Which python module is capable of pipelining http requests?
    >
    > (I know httplib can send mulitple requests per tcp connection, but in
    > a strictly serial way. )
    >
    >

    There's nothing in the standard library, I believe, that includes both
    client and server functionality in the same module. So you would need to
    glue them together.

    If you want a simple net proxy server I seem to remember the chameleon
    system allows you to write one in about twelve lines. If it has to be
    HTTP-specific, with header parsing and the like, you might want to think
    about Twisted, which supports both client and server functionality and
    tries to make it easy to plumb things together in pipelines.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC/Ltd http://www.holdenweb.com
    Skype: holdenweb http://del.icio.us/steve.holden
    Recent Ramblings http://holdenweb.blogspot.com
     
    Steve Holden, Apr 27, 2007
    #2
    1. Advertising

  3. Steve Holden Guest

    wrote:
    > Which python module is capable of pipelining http requests?
    >
    > (I know httplib can send mulitple requests per tcp connection, but in
    > a strictly serial way. )
    >

    Oops, sorry, you meant sending requests in parallel, right?

    You'll need to use either urllib or urllib2 for the web, and the
    threading module is one way to run parallel requests. It's fairly easy
    to use as long as you keep your tasks properly isolated form each other.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC/Ltd http://www.holdenweb.com
    Skype: holdenweb http://del.icio.us/steve.holden
    Recent Ramblings http://holdenweb.blogspot.com
     
    Steve Holden, Apr 27, 2007
    #3
  4. Guest

    On Fri, 27 Apr 2007 13:50:21 -0400, Steve Holden <>
    wrote:

    > wrote:
    >> Which python module is capable of pipelining http requests?
    >>
    >> (I know httplib can send mulitple requests per tcp connection, but in
    >> a strictly serial way. )
    >>
    >>

    >There's nothing in the standard library, I believe, that includes both
    >client and server functionality in the same module. So you would need to
    >glue them together.
    >
    >If you want a simple net proxy server I seem to remember the chameleon
    >system allows you to write one in about twelve lines. If it has to be
    >HTTP-specific, with header parsing and the like, you might want to think
    >about Twisted, which supports both client and server functionality and
    >tries to make it easy to plumb things together in pipelines.


    Sorry for my confused question.

    What I'm looking for is to emulate what a modern HTTP 1.1 browser like
    Firefox does(when network.http.pipelining is enabled)

    1.Open TCP connetion.

    send multiple requests without waiting for a reply:

    2.GET /url1
    3.GET /url2
    4.GET /url3
    5.read 1st reply
    6.read 2st reply
    7.read 3st reply

    8.Close TCP connection.

    I'll be using Apache or IIS as my web server.
     
    , Apr 27, 2007
    #4
  5. John J. Lee Guest

    Steve Holden <> writes:

    > wrote:
    > > Which python module is capable of pipelining http requests?
    > > (I know httplib can send mulitple requests per tcp connection, but in
    > > a strictly serial way. )
    > >

    > Oops, sorry, you meant sending requests in parallel, right?
    >
    > You'll need to use either urllib or urllib2 for the web, and the
    > threading module is one way to run parallel requests. It's fairly easy
    > to use as long as you keep your tasks properly isolated form each
    > other.


    No, he means "HTTP pipelining", which means sending multiple requests
    down a single TCP connection, without waiting for the first response.

    httplib's module-level docstring says (reformatted here):

    """
    .... The HTTPResponse class does not enforce this state machine, which
    implies sophisticated clients may accelerate the request/response
    pipeline. Caution should be taken, though: accelerating the states
    beyond the above pattern may imply knowledge of the server's
    connection-close behavior for certain requests. For example, it is
    impossible to tell whether the server will close the connection UNTIL
    the response headers have been read; this means that further requests
    cannot be placed into the pipeline until it is known that the server
    will NOT be closing the connection.
    """

    So, sort-of-yes, if you know what you're doing and you're lucky.

    Certainly urllib and urllib2 don't support pipelining. There were
    plans for a new HTTP client in Twisted with pipelining support, but I
    don't know if that ever came about. AFAIK not many libraries (in any
    language) support it -- e.g. none of "Jakarta commons HTTPClient",
    libwww-perl, and libcurl currently support it. libwww (without the
    "-perl") does claim to support it (I say "claim" merely because I
    haven't used it or read the source -- no FUD intended).


    Side note: As the OP mentions in a followup, by default firefox does
    NOT do pipelining (to the disbelief of the people I told about this
    when it came up in a previous job -- but I just tried running tcpdump
    and indeed about:config seems to be telling the truth; fortunately, in
    response to the limitations imposed by RFC 2616, somebody has
    thoughtfully arranged for the speed of light to be fast enough for
    this not to be a major problem when using websites on the other side
    of the planet). The firefox people say:

    http://www.mozilla.org/support/firefox/tips#oth_pipelining

    """
    Pipelining is an experimental feature, designed to improve page-load
    performance, that is unfortunately not well supported by some web
    servers and proxies.
    """

    Instead of pipelining, it uses multiple connections (2 when I tried
    it, which is the maximum RFC 2616 says SHOULD be used by a
    "single-user client"). I didn't try IE, but apparently it has the
    same behaviour (2 connections, no pipelining):

    http://blogs.msdn.com/ie/archive/2005/04/11/407189.aspx


    I wonder if the right economic pressures are there for SCTP ever to
    get used for everyday web HTTP stuff...


    John
     
    John J. Lee, Apr 30, 2007
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. koko

    Pipelining in VHDL

    koko, Apr 29, 2004, in forum: VHDL
    Replies:
    2
    Views:
    5,731
    mizocom
    Apr 29, 2004
  2. Peggy
    Replies:
    0
    Views:
    648
    Peggy
    Sep 8, 2004
  3. noviceUser

    HTTP 1.1 pipelining

    noviceUser, May 17, 2004, in forum: Python
    Replies:
    3
    Views:
    395
    Ivan Voras
    May 18, 2004
  4. Crouchez

    http pipelining

    Crouchez, Aug 10, 2007, in forum: Java
    Replies:
    4
    Views:
    864
    Crouchez
    Aug 10, 2007
  5. panatha

    Question about http pipelining

    panatha, Aug 17, 2007, in forum: Java
    Replies:
    2
    Views:
    778
Loading...

Share This Page