Select weirdness

[Ron Garret]

Here's my code. It's a teeny weeny little HTTP server. (I'm not really
trying to reinvent the wheel here. What I'm really doing is writing a
dispatching proxy server, but this is the shortest way to illustrate the
problem I'm having):

from SocketServer import *
from socket import *
from select import select

class myHandler(StreamRequestHandler):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[])[0]
if sl:
l = self.rfile.readline()
if len(l)<3: break
print l,
pass
pass
print>>self.wfile, 'HTTP/1.0 200 OK'
print>>self.wfile, 'Content-type: text/plain'
print>>self.wfile
print>>self.wfile, 'foo'
self.rfile.close()
self.wfile.close()
print '<<<<<<<<<<<<'
pass
pass

def main():
server = TCPServer(('',8080), myHandler)
server.serve_forever()
pass

if __name__ == '__main__': main()


If I telnet into this server and type in an HTTP request manually it
works fine. But when I try to access this with a browser (I've tried
Firefox and Safari -- both do the same thing) it hangs immediately after
reading the first line in the request (i.e. before reading the first
header).

When I click the "stop" button in the browser it breaks the logjam and
the server reads the headers (but then of course it dies trying to write
the response to a now-closed socket).

The only difference I can discern is that the browser send \r\n for
end-of-line while telnet just sends \n. But I don't see why that should
make any difference. So I'm stumped. Any clues would be much
appreciated.

Thanks,
rg

 

[Bjoern Schliessmann]

print>>self.wfile, 'HTTP/1.0 200 OK'
print>>self.wfile, 'Content-type: text/plain'
print>>self.wfile
print>>self.wfile, 'foo'
[...]
If I telnet into this server and type in an HTTP request manually
it works fine. But when I try to access this with a browser (I've
tried Firefox and Safari -- both do the same thing) it hangs
immediately after reading the first line in the request (i.e.
before reading the first header).

Could the problem be that print ends lines with "\r\n" not on all
platforms? Try doing it explicitly.

The only difference I can discern is that the browser send \r\n
for end-of-line while telnet just sends \n.

Then, I think, your telnet client is broken. I've seem multiple
applications relying on telnet or similar protocols which
required "\r\n" as end-of-line. E. g. I needed to hack
twisted.protocols.basic.LineReceiverFactory since it showed the
exact behaviour you described for browsers. It showed up because
busybox' telnet only sends \n as EOL.

But I don't see why that should make any difference.

Easy. If you only accept "\r\n" as EOL, you'll wait forever unless
you really receive it.

Regards,


Björn

 

[Irmen de Jong]

Here's my code. It's a teeny weeny little HTTP server. (I'm not really
trying to reinvent the wheel here. What I'm really doing is writing a
dispatching proxy server, but this is the shortest way to illustrate the
problem I'm having):

from SocketServer import *
from socket import *
from select import select

class myHandler(StreamRequestHandler):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[])[0]
if sl:
l = self.rfile.readline()
if len(l)<3: break
print l,
pass
pass
print>>self.wfile, 'HTTP/1.0 200 OK'
print>>self.wfile, 'Content-type: text/plain'
print>>self.wfile
print>>self.wfile, 'foo'
self.rfile.close()
self.wfile.close()
print '<<<<<<<<<<<<'
pass
pass

You shouldn't use a select() of your own inside the handle method.
The socketserver takes care of that for you.

What you need to do is read the input from the socket until you've reached
the end-of-input marker. That marker is an empty line containing just "\r\n"
when dealing with HTTP GET requests.

Any reason don't want to use the BasicHTTPServer that comes with Python?

Anyway, try the following instead:

from SocketServer import *
from socket import *
from select import select

class myHandler(StreamRequestHandler):
def handle(self):
print '>>>>>>>>>>>'
while True:
l = self.rfile.readline()
print repr(l)
if not l or l=='\r\n':
break
print>>self.wfile, 'HTTP/1.0 200 OK'
print>>self.wfile, 'Content-type: text/plain'
print>>self.wfile
print>>self.wfile, 'foo'
self.rfile.close()
self.wfile.close()
print '<<<<<<<<<<<<'

def main():
server = TCPServer(('',8080), myHandler)
server.serve_forever()

if __name__ == '__main__': main()



--Irmen

 

[Ron Garret]

Here's my code. It's a teeny weeny little HTTP server. (I'm not really
trying to reinvent the wheel here. What I'm really doing is writing a
dispatching proxy server, but this is the shortest way to illustrate the
problem I'm having):

from SocketServer import *
from socket import *
from select import select

class myHandler(StreamRequestHandler):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[])[0]
if sl:
l = self.rfile.readline()
if len(l)<3: break
print l,
pass
pass
print>>self.wfile, 'HTTP/1.0 200 OK'
print>>self.wfile, 'Content-type: text/plain'
print>>self.wfile
print>>self.wfile, 'foo'
self.rfile.close()
self.wfile.close()
print '<<<<<<<<<<<<'
pass
pass

You shouldn't use a select() of your own inside the handle method.
The socketserver takes care of that for you.

I don't understand why socketserver calling select should matter. (And
BTW, there are no calls to select in SocketServer.py. I'm using
Python2.5.)

What you need to do is read the input from the socket until you've reached
the end-of-input marker. That marker is an empty line containing just "\r\n"
when dealing with HTTP GET requests.

Any reason don't want to use the BasicHTTPServer that comes with Python?

Yes, but it's a long story. What I'm really doing is writing a
dispatching proxy. It reads the HTTP request and then redirects it to a
number of different servers depending on the URL. The servers are all
local and served off of unix sockets, not TCP sockets (which is why I
can't just configure Apache to do this).

The reason I'm running this weird configuration (in case you're
wondering) is because this is a development machine and multiple copies
of the same website run on it simultaneously. Each copy of the site
runs a customized server to handle AJAX requests efficiently.

Anyway, try the following instead:

That won't work for POST requests.

rg

 

[Ron Garret]

The only difference I can discern is that the browser send \r\n
for end-of-line while telnet just sends \n. ....
But I don't see why that should make any difference.

Easy. If you only accept "\r\n" as EOL, you'll wait forever unless
you really receive it.[/QUOTE]

But it's SELECT that is hanging. Select knows (or ought to know)
nothing of end-of-line markers. Readline (when it is called) is doing
the Right Thing, and my end-of-request test is (currently) reading a
line less than 3 characters long. All that is working. The problem is
that select is saying there is no input (and therefore hanging) when in
fact there is input.

rg

 

[Ron Garret]

Here's my code. It's a teeny weeny little HTTP server. (I'm not really
trying to reinvent the wheel here. What I'm really doing is writing a
dispatching proxy server, but this is the shortest way to illustrate the
problem I'm having):

from SocketServer import *
from socket import *
from select import select

class myHandler(StreamRequestHandler):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[])[0]
if sl:
l = self.rfile.readline()
if len(l)<3: break
print l,
pass
pass
print>>self.wfile, 'HTTP/1.0 200 OK'
print>>self.wfile, 'Content-type: text/plain'
print>>self.wfile
print>>self.wfile, 'foo'
self.rfile.close()
self.wfile.close()
print '<<<<<<<<<<<<'
pass
pass

def main():
server = TCPServer(('',8080), myHandler)
server.serve_forever()
pass

if __name__ == '__main__': main()


If I telnet into this server and type in an HTTP request manually it
works fine. But when I try to access this with a browser (I've tried
Firefox and Safari -- both do the same thing) it hangs immediately after
reading the first line in the request (i.e. before reading the first
header).

When I click the "stop" button in the browser it breaks the logjam and
the server reads the headers (but then of course it dies trying to write
the response to a now-closed socket).

The only difference I can discern is that the browser send \r\n for
end-of-line while telnet just sends \n. But I don't see why that should
make any difference. So I'm stumped. Any clues would be much
appreciated.

I have reproduced the problem using Telnet, so that proves it's not an
EOL issue. The problem seems to be timing-related. If I type the
request in manually it works. If I paste it in all at once, it hangs.
It's actually even weirder than that: if I pipe the request into a
telnet client then it hangs after the request line, just with a browser
(or wget). But if I literally paste the request into a window running a
telnet client, it gets past the request line and four out of seven
headers before it hangs.

rg

 

[Ron Garret]

I think I've figured out what's going on.

First, here's the smoking gun: I changed the code as follows:

class myHandler(StreamRequestHandler):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[],1)[0]
print sl
l = self.rfile.readline()
if len(l)<3: break
print l,
pass

(Rest of code is unchanged.)

In other words, I select on the rfile object and added a timeout.

Here's the result when I cut-and-paste an HTTP request:
[<socket._fileobject object at 0x6b110>]
GET /foo/baz HTTP/1.1
[<socket._fileobject object at 0x6b110>]
Accept: */*
[<socket._fileobject object at 0x6b110>]
Accept-Language: en
[<socket._fileobject object at 0x6b110>]
Accept-Encoding: gzip, deflate
[<socket._fileobject object at 0x6b110>]
Cookie: c1=18:19:55.042196; c2=18:19:55.042508
[]
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en)
AppleWebKit/419 (KHTM
[]
L, like Gecko) Safari/419.3
[]
Connection: keep-alive
[]
Host: localhost:8081
[]
<<<<<<<<<<<<

As you can see, the select call shows input available for a while (five
lines) and then shows no input available despite the fact that there is
manifestly still input available.

The answer is obvious: select is looking only at the underlying socket,
and not at the rfile buffers.

So... is this a bug in select? Or a bug in my code?

rg

 

[Ron Garret]

The answer is obvious: select is looking only at the underlying socket,
and not at the rfile buffers.

Here is conclusive proof that there's a bug in select:

from socket import *
from select import select
s=socket(AF_INET, SOCK_STREAM)
s.bind(('',8080))
s.listen(5)
f = s.accept()[0].makefile()
# Now telnet to port 8080 and enter two lines of random text
f.readline()
select([f],[],[],1)
f.readline()

Here's the sample input:

[ron@mickey:~/devel/sockets]$ telnet localhost 8081
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
123
321


And this is the result:

f.readline() '123\r\n'
select([f],[],[],1)

# After one second...
([], [], [])

So this is clearly a bug, but surely I'm not the first person to have
encountered this? Is there a known workaround?

rg

 

[Erik Max Francis]

So this is clearly a bug, but surely I'm not the first person to have
encountered this? Is there a known workaround?

It's hard to see how this demonstrates a bug in anything, since you're
telnetting to the wrong port in your example.

 

[Ron Garret]

It's hard to see how this demonstrates a bug in anything, since you're
telnetting to the wrong port in your example.

Geez you people are picky. Since I ran this several times I ran into
the TIM_WAIT problem. Here's the actual transcript:

Python 2.5 (r25:51908, Mar 1 2007, 10:09:05)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
File "<stdin>", line 1, in <module>

s.bind(('',8081))
s.listen(5)
f = s.accept()[0].makefile()
f.readline() '123\r\n'
select([f],[],[],1) ([], [], [])
f.readline()

'321\r\n'

 

[Dennis Lee Bieber]

The answer is obvious: select is looking only at the underlying socket,
and not at the rfile buffers.

Here is conclusive proof that there's a bug in select:

from socket import *
from select import select
s=socket(AF_INET, SOCK_STREAM)
s.bind(('',8080))
s.listen(5)
f = s.accept()[0].makefile()
# Now telnet to port 8080 and enter two lines of random text
f.readline()
select([f],[],[],1)
f.readline()

Here's the sample input:

[ron@mickey:~/devel/sockets]$ telnet localhost 8081
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
123
321


And this is the result:

f.readline() '123\r\n'
select([f],[],[],1)

# After one second...
([], [], [])

So this is clearly a bug, but surely I'm not the first person to have
encountered this? Is there a known workaround?

Well, on WinXP, Python 2.4, with

-=-=-=-=-=-
from socket import *
from select import select

s=socket(AF_INET, SOCK_STREAM)
s.bind( ("", 8080) )
s.listen(5)

f = s.accept()[0].makefile()

lcnt = 0
while lcnt < 2:
(si, so, se) = select([f], [], [], 6.0)
if si:
c = None
while c != "\n":
c = f.read(1)
print c,
lcnt += 1
else:
print "."

lcnt = 0
while lcnt < 4:
(si, so, se) = select([f], [], [], 6.0)
if si:
print f.readline()
lcnt += 1
else:
print "."
lcnt += 1
-=-=-=-=-=-=-=-

I get this result -- note: can't capture input, no echo for ProComm
telnet. ProComm seems to send some noise at the start of connection

-=-=-=-=-=-=-

pythonw -u "t.py"

ÿ ý
ÿ û  F i r s t l o o p b l o c k s o n f i r s t d a t a , w
a i t s f o r \ n
..
..
A l o w e d t o t i m e o u t s o m e , t h e n f i r s t k
e
r e t u r n s t o b l c \ o c k i n g r e a d

..
Second loop doing readline, allowed to time
our and looking for \n

..
Time out loop works

Exit code: 0

-=-=-=-=-=-=-

The first loop initially detects the connection static so goes into
the f.read(1) loop, and that loop presumes a \n will eventually arrive;
hitting <enter> won't do it as only \r gets sent -- have to manually
send <ctrl-j>.

The select then was allowed to time out twice before I started
typing more text (which entered the inner loop again). On receipt of the
\n it went to the second loop, which was allowed to time out once, then
I started typing -- nothing was displayed until I sent the <ctrl-j> and
went back to the select, which I allowed to time out again, then started
typing.

Using your code (modified with print statements

print f.readline()
print select([f],[],[],1)
print f.readline()

) and typing fast gave:

pythonw -u "t.py"

ÿý
ÿûthis is line one

Exit code: 0

So the second line got buffered while the first was printing, and
the select recorded data. Typing slower results in...

pythonw -u "t.py"

ÿý
ÿûThis is the first line, will pause after \n

([], [], [])
Timed out properl, now is the second line

Exit code: 0

I'd not entered the second line until the timeout print showed.
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Erik Max Francis]

Geez you people are picky. Since I ran this several times I ran into
the TIM_WAIT problem. Here's the actual transcript:

It's not about being picky, it's about making it clear what your problem
is. You're now describing an entirely different problem, hence why it's
important to be clear about _precisely_ what it is you're doing and
_precisely_ what's happening that you think that's wrong.

Python 2.5 (r25:51908, Mar 1 2007, 10:09:05)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
File "<stdin>", line 1, in <module>

s.bind(('',8081))
s.listen(5)
f = s.accept()[0].makefile()
f.readline() '123\r\n'
select([f],[],[],1) ([], [], [])
f.readline()

'321\r\n'

 

[Ron Garret]

It's not about being picky, it's about making it clear what your problem
is. You're now describing an entirely different problem,

Nope, it's been the same problem all along. If you don't think so then
you have misunderstood something.

rg

 

[Ron Garret]

Well, on WinXP, Python 2.4, with

I should have specified: I'm running 2.5 on unix. (I've reproduced the
problem on both Linux and OS X.)

rg

 

[Irmen de Jong]

I don't understand why socketserver calling select should matter. (And
BTW, there are no calls to select in SocketServer.py. I'm using
Python2.5.)

You don't *need* a select at all.
Socketserver just blocks on accept() and dispatches a handler
on the new connection.

That won't work for POST requests.

Why not?
Just add some more code to deal with the POST request body.
There should be a content-length header to tell you how many
bytes to read after the header section has finished.

--Irmen

 

[Ron Garret]

You don't *need* a select at all.

Yes I do, because what I'm really writing is a dispatching proxy that
has to serve many simultaneous connections.

Here's the full story in case you're interested: We have an application
that is currently fielded as a cgi. We have a development server that
needs to run multiple copies of the application at the same time. This
is so that developers can push changes into their private "sandboxes"
for evaluation before going into the main development branch. Each
sandbox has its own source tree, its own database, and its own URL
namespace.

There are a small number of URLs in the application that are performance
bottlenecks (they are used to serve AJAX updates). In order to
alleviate that bottleneck without having to rewrite the whole
application to run under mod_python or some such thing we've written a
special dedicated server that handles only the AJAX requests.

The tricky part is that each developer needs to have their own copy of
this server running because each developer can have different code that
needs to run to serve those requests. Assigning each developer a
dedicated IP port would be a configuration nightmare, so these servers
serve run on unix sockets rather than TCP sockets.

I have not been able to find a proxy server that can proxy to unix
sockets, so I need to write my own. Conceptually its a very simple
thing: read the first line of an HTTP request, parse it with a regexp to
extract the sandbox name, connect to the appropriate unix server socket,
and then bidirectionally pipe bytes back and forth. But it has to do
this for multiple connections simultaneously, which is why I need select.

Why not?

Because POST requests can be very complicated.

Just add some more code to deal with the POST request body.

I was really hoping to avoid having to write a fully HTTP-aware proxy.

There should be a content-length header to tell you how many
bytes to read after the header section has finished.

Not if the content-transfer-encoding is chunked. Or if there are
multiple file attachments.

Also, even GET requests can become very complicated (from a protocol
point of view) in HTTP 1.1.

rg

 

[Gabriel Genellina]

I have not been able to find a proxy server that can proxy to unix
sockets, so I need to write my own. Conceptually its a very simple
thing: read the first line of an HTTP request, parse it with a regexp to
extract the sandbox name, connect to the appropriate unix server socket,
and then bidirectionally pipe bytes back and forth. But it has to do
this for multiple connections simultaneously, which is why I need select.

No. This is what the *server* should do, not the *handler*. The server
listens on incoming requests, and creates a new handler for each one.
Inside a handler, you don't have to worry about multiple connections.
If you want to process simultaneous connections, inherit your server from
ThreadingMixIn or ForkingMixIn (or use one of the predefined server
classes).

 

[Ron Garret]

No. This is what the *server* should do, not the *handler*. The server
listens on incoming requests, and creates a new handler for each one.
Inside a handler, you don't have to worry about multiple connections.
If you want to process simultaneous connections, inherit your server from
ThreadingMixIn or ForkingMixIn (or use one of the predefined server
classes).

Ah, good point.

But that still leaves the problem of keep-alive connections. I still
need to be able to forward data in both directions without knowing ahead
of time when it will arrive.

(I think I can solve the problem by using send and recv directly BTW.)

rg

 

[Donn Cave]

The answer is obvious: select is looking only at the underlying socket,
and not at the rfile buffers.

So... is this a bug in select? Or a bug in my code?

Yes.

I don't see any specific followup to this point, but it is or
at least should be a well known limitation of C library I/O and
select. select() is an operating system function that has no
way to know the state of your process I/O buffers, nor do the C
stdio buffers don't come with a standard way to inspect that state.
Therefore, if you mix C I/O with select(), you're more or less out
of luck. This applies directly to Python, because it calls the
operating system select() function and it uses C stdio buffers for
its file object (and entices you to make this mistake by supporting
a file object as an input to select().)

This conflict can be relieved after a fashion by eliminating the
buffer. The now unbuffered C fgets / Python readline won't store
extra lines that select can't see, but the semantics of the operation
are still a poor fit with the usual application of select. The
number of system-level I/O calls is significantly greater as you
transfer data one byte at a time, which may add a lot of overhead
if you transfer a lot of data, and your readline() function still
can't return until it gets that '\n', so a half line can block your
application.

It isn't a lot of work to read data with operating system functions
that are compatible with select - os.read(), socket.recv() - and
break it up into lines on your own, and this completely and efficiently
resolves the problem.

I haven't looked at your code.

Donn Cave, (e-mail address removed)

 

[Ron Garret]

Yes.

I don't see any specific followup to this point, but it is or
at least should be a well known limitation of C library I/O and
select. select() is an operating system function that has no
way to know the state of your process I/O buffers, nor do the C
stdio buffers don't come with a standard way to inspect that state.
Therefore, if you mix C I/O with select(), you're more or less out
of luck. This applies directly to Python, because it calls the
operating system select() function and it uses C stdio buffers for
its file object (and entices you to make this mistake by supporting
a file object as an input to select().)

This conflict can be relieved after a fashion by eliminating the
buffer. The now unbuffered C fgets / Python readline won't store
extra lines that select can't see, but the semantics of the operation
are still a poor fit with the usual application of select. The
number of system-level I/O calls is significantly greater as you
transfer data one byte at a time, which may add a lot of overhead
if you transfer a lot of data, and your readline() function still
can't return until it gets that '\n', so a half line can block your
application.

It isn't a lot of work to read data with operating system functions
that are compatible with select - os.read(), socket.recv() - and
break it up into lines on your own, and this completely and efficiently
resolves the problem.

Yep, this is pretty much the conclusion I've come to.

As an editorial comment, I think it's quite confusing for Python's
select to accept file objects without complaint if it's not going to do
the Right Thing with them.

rg