SimpleHTTPRequestHandler used with HTTP/1.1 hangs after the secondresource on a page.

[Piotr Dobrogost]

Hi!

I'd like to bring your attention to the question titled "Use HTTP/1.1 with SimpleHTTPRequestHandler" at http://stackoverflow.com/q/15839718/95735 which reads; "When I use HTTP/1.1 with SimpleHTTPRequestHandler, loading a pagethat pulls in other resources will hang after the second resource." and there's a tiny test showing the problem.
It's hard to believe that SimpleHTTPServer doesn't handle such a simple task, does it?
If I checked correctly code is stuck in handle_one_request() method at line370 of server.py (http://hg.python.org/cpython/file/d9893d13c628/Lib/http/server.py#l370) but I can't see what's wrong.
Any ideas?


Best regards,
Piotr Dobrogost

 

[Terry Jan Reedy]

Hi!

I'd like to bring your attention to the question titled "Use HTTP/1.1
with SimpleHTTPRequestHandler" at
http://stackoverflow.com/q/15839718/95735 which reads; "When I use

I find the doc slightly confusing. The SO code uses BaseHTTPServer. The
doc says "Usually, this module isn’t used directly," On the other hand,
SimpleHTTPServer only defines a request handler and not a server itself.

HTTP/1.1 with SimpleHTTPRequestHandler, loading a page that pulls in
other resources will hang after the second resource." and there's a

What does 'hang' mean? Does the page get displayed? If so,
server.serve_forever is supposes to 'hang'.

tiny test showing the problem. It's hard to believe that
SimpleHTTPServer doesn't handle such a simple task, does it? If I
checked correctly code is stuck in handle_one_request() method at
line 370 of server.py
(http://hg.python.org/cpython/file/d9893d13c628/Lib/http/server.py#l370)
but I can't see what's wrong. Any ideas?

The SO post says

class MyRequestHandler(SimpleHTTPRequestHandler):
#protocol_version = "HTTP/1.0" # works
protocol_version = "HTTP/1.1" # hangs

so this should be some sort of clue. The code says

569 # The version of the HTTP protocol we support.
570 # Set this to HTTP/1.1 to enable automatic keepalive
571 protocol_version = "HTTP/1.0"

The only substantive code I found using protocol_version (in the CPython
default branch you linked to) is

300 if version_number >= (1, 1) and self.protocol_version >= "HTTP/1.1":
301 self.close_connection = 0

331 elif (conntype.lower() == 'keep-alive' and
332 self.protocol_version >= "HTTP/1.1"):
333 self.close_connection = 0
334 # Examine the headers and look for an Expect directive
335 expect = self.headers.get('Expect', "")
336 if (expect.lower() == "100-continue" and
337 self.protocol_version >= "HTTP/1.1" and
338 self.request_version >= "HTTP/1.1"):
339 if not self.handle_expect_100():
340 return False

I would re-test with the recently released 3.3.1 (and/or 2.7.4) to make
sure nothing has changed

 

[Piotr Dobrogost]

I find the doc slightly confusing. The SO code uses BaseHTTPServer. The
doc says "Usually, this module isn’t used directly," On the other hand,
SimpleHTTPServer only defines a request handler and not a server itself.

Thanks for taking time to reply.

Well, I've seen presentations where BaseHTTPServer is a recommended way of running simple, ad-hoc web server. In addition the documentation of http.server module in Python 3.3 (which contains implementation of the same HTTPServer class as found in BaseHTTPServer module in Python 2.x) does not contain the statement you cited which I take to mean it can safely be used directly.

As Python 3.3.1 exhibits the same behavior let's focus on the following Python 3 version of the code from the original question:

from http.server import SimpleHTTPRequestHandler
from http.server import HTTPServer

class MyRequestHandler(SimpleHTTPRequestHandler):
#protocol_version = "HTTP/1.0" # works
protocol_version = "HTTP/1.1" # hangs

server = HTTPServer(("localhost", 7080), MyRequestHandler)
server.serve_forever()

What does 'hang' mean? Does the page get displayed? If so,
server.serve_forever is supposes to 'hang'.

I increased the number of resources to 7 to be able to detect possible patterns. The page and some of the resources are being displayed almost instantly. However browser waits for the requests to get remaining resources to finish and so those resources naturally are not being displayed. When browserwaits the code is stuck at the line I showed in my original post. What's interesting is that after some time the request to get the second resource (in case there are only 2 on the page) finishes. However the time spent waiting for the second resource is ridiculously long compared to time spend waiting for the first resource. For example in one of my tests Firebug showed waiting time for the first resource to be 4ms whereas the waiting time for the second resource to be 1m 55s.

The SO post says

class MyRequestHandler(SimpleHTTPRequestHandler):
#protocol_version = "HTTP/1.0" # works
protocol_version = "HTTP/1.1" # hangs

so this should be some sort of clue. The code says

569 # The version of the HTTP protocol we support.
570 # Set this to HTTP/1.1 to enable automatic keepalive
571 protocol_version = "HTTP/1.0"

Sure, in version 1.1 of http the connection is not closed by default unless'Connection: Close' header was sent in request. Firefox sends all requestswith 'Connection: keep-alive' header.

I don't really know how to debug this...

 

[Piotr Dobrogost]

I find the doc slightly confusing. The SO code uses BaseHTTPServer. The
doc says "Usually, this module isn’t used directly," On the other hand,
SimpleHTTPServer only defines a request handler and not a server itself.

Thanks for taking time to reply.

Well, I've seen presentations where BaseHTTPServer is a recommended way of running simple, ad-hoc web server. In addition the documentation of http.server module in Python 3.3 (which contains implementation of the same HTTPServer class as found in BaseHTTPServer module in Python 2.x) does not contain the statement you cited which I take to mean it can safely be used directly.

As Python 3.3.1 exhibits the same behavior let's focus on the following Python 3 version of the code from the original question:

from http.server import SimpleHTTPRequestHandler
from http.server import HTTPServer

class MyRequestHandler(SimpleHTTPRequestHandler):
#protocol_version = "HTTP/1.0" # works
protocol_version = "HTTP/1.1" # hangs

server = HTTPServer(("localhost", 7080), MyRequestHandler)
server.serve_forever()

What does 'hang' mean? Does the page get displayed? If so,
server.serve_forever is supposes to 'hang'.

I increased the number of resources to 7 to be able to detect possible patterns. The page and some of the resources are being displayed almost instantly. However browser waits for the requests to get remaining resources to finish and so those resources naturally are not being displayed. When browserwaits the code is stuck at the line I showed in my original post. What's interesting is that after some time the request to get the second resource (in case there are only 2 on the page) finishes. However the time spent waiting for the second resource is ridiculously long compared to time spend waiting for the first resource. For example in one of my tests Firebug showed waiting time for the first resource to be 4ms whereas the waiting time for the second resource to be 1m 55s.

The SO post says

class MyRequestHandler(SimpleHTTPRequestHandler):
#protocol_version = "HTTP/1.0" # works
protocol_version = "HTTP/1.1" # hangs

so this should be some sort of clue. The code says

569 # The version of the HTTP protocol we support.
570 # Set this to HTTP/1.1 to enable automatic keepalive
571 protocol_version = "HTTP/1.0"

Sure, in version 1.1 of http the connection is not closed by default unless'Connection: Close' header was sent in request. Firefox sends all requestswith 'Connection: keep-alive' header.

I don't really know how to debug this...