socket.makefile() buggy?

[ahlongxp]

socket.makefile() may lose data when "connection reset by peer".
and socket.recv() will never lose the data.

change the "1" to "0" in the client code to see the difference.

confirmed on both windows and linux.

so I guess there is a problem with makefile().

# Echo server program
import socket

HOST = '' # Symbolic name meaning the local host
PORT = 50007 # Arbitrary non-privileged port
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
while(1):
conn, addr = s.accept()
print 'Connected by', addr
conn.settimeout(1)
toread = 99
retrytime = 0
reads = 0
while reads < toread and retrytime < 10:
try:
data = conn.recv(min(32,toread-reads))
if not data: continue
print data
reads += len(data)
except:
retrytime += 1
print "timeout %d" % retrytime
continue
#conn.shutdown(socket.SHUT_RD)#no more read
if reads == toread:
conn.send("OK")
else:
conn.send("NOT OK")
conn.close()

# Echo client program
import socket

HOST = 'localhost' # The remote host
PORT = 50007 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
for i in range(12):
print "time %d" % i
try:
s.send('0123456789')
except socket.error, e:
print "socket error:", e
break
#data = s.recv(1024)
#print "data %d" %i, data
#s.shutdown(socket.SHUT_WR)#no more write

'''
try changing 1 to 0.
'''
if 1:
data=s.recv(1024)
else:
rf = s.makefile("rb")
data = rf.read()
rf.close()
s.close()
print 'Received', repr(data)

 

[Steve Holden]

That's a pretty pejorative subject line for someone who's been
programming Python [guessing by the date of your first post] for about a
month.

Perhaps "Incomprehensible behavior from socket.makefile()", or "I have
written a buggy network application"? That would at least show that you
are considering the possibility you yourself are the cause of the
problem ;-)

Python has been around for a long time, so you should ask yourself how
likely it is that you would be the first to discover such a fundamental
flaw? I'd be very surprised if someone doesn't point you at an article

socket.makefile() may lose data when "connection reset by peer".
and socket.recv() will never lose the data.

change the "1" to "0" in the client code to see the difference.

confirmed on both windows and linux.

so I guess there is a problem with makefile().

# Echo server program
import socket

HOST = '' # Symbolic name meaning the local host
PORT = 50007 # Arbitrary non-privileged port
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
while(1):
conn, addr = s.accept()
print 'Connected by', addr
conn.settimeout(1)
toread = 99
retrytime = 0
reads = 0
while reads < toread and retrytime < 10:
try:
data = conn.recv(min(32,toread-reads))
if not data: continue
print data
reads += len(data)
except:
retrytime += 1
print "timeout %d" % retrytime
continue
#conn.shutdown(socket.SHUT_RD)#no more read
if reads == toread:
conn.send("OK")
else:
conn.send("NOT OK")
conn.close()

# Echo client program
import socket

HOST = 'localhost' # The remote host
PORT = 50007 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
for i in range(12):
print "time %d" % i
try:
s.send('0123456789')
except socket.error, e:
print "socket error:", e
break
#data = s.recv(1024)
#print "data %d" %i, data
#s.shutdown(socket.SHUT_WR)#no more write

'''
try changing 1 to 0.
'''
if 1:
data=s.recv(1024)
else:
rf = s.makefile("rb")
data = rf.read()
rf.close()
s.close()
print 'Received', repr(data)

The big problem here seems to be that your server is closing the socket
after reading 99 bytes, but the client is actually sending 120. If you
change the "99" in the server to "120" you will see that the client
works when it uses the makefile().read(). I can't be bothered to debug
the exact reason why the 21 bytes of buffered data doesn't cause a
problem with the recv() version, but I wouldn't regard the error you are
getting as a bug in Python, rather as a bug in your application.

The underlying cause of all this appears to be your failure to
understand that network protocols should ideally allow the endpoints to
determine when a complete message has been transmitted, and to consume
all transmitted data.

You can't just ignore some or all of a client request and expect it not
to cause problems. If you look at some of the better-known TCP-based
application protocols you will see they take pains to ensure that
clients and servers can deterministically parse each others' messages.

In summary, a better protocol design will cause you rather less grief
and a little more humility will result in less acidity from crabby old
geeks like me.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

 

[Paul McGuire]

That's a pretty pejorative subject line for someone who's been
programming Python [guessing by the date of your first post] for about a
month.

Perhaps "Incomprehensible behavior from socket.makefile()", or "I have
written a buggy network application"? That would at least show that you
are considering the possibility you yourself are the cause of the
problem ;-)

Python has been around for a long time, so you should ask yourself how
likely it is that you would be the first to discover such a fundamental
flaw? I'd be very surprised if someone doesn't point you at an article

socket.makefile() may lose data when "connection reset by peer".
and socket.recv() will never lose the data.

change the "1" to "0" in the client code to see the difference.

confirmed on both windows and linux.

so I guess there is a problem with makefile().

# Echo server program
import socket

HOST = '' # Symbolic name meaning the local host
PORT = 50007 # Arbitrary non-privileged port
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
while(1):
conn, addr = s.accept()
print 'Connected by', addr
conn.settimeout(1)
toread = 99
retrytime = 0
reads = 0
while reads < toread and retrytime < 10:
try:
data = conn.recv(min(32,toread-reads))
if not data: continue
print data
reads += len(data)
except:
retrytime += 1
print "timeout %d" % retrytime
continue
#conn.shutdown(socket.SHUT_RD)#no more read
if reads == toread:
conn.send("OK")
else:
conn.send("NOT OK")
conn.close()

# Echo client program
import socket

HOST = 'localhost' # The remote host
PORT = 50007 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
for i in range(12):
print "time %d" % i
try:
s.send('0123456789')
except socket.error, e:
print "socket error:", e
break
#data = s.recv(1024)
#print "data %d" %i, data
#s.shutdown(socket.SHUT_WR)#no more write

'''
try changing 1 to 0.
'''
if 1:
data=s.recv(1024)
else:
rf = s.makefile("rb")
data = rf.read()
rf.close()
s.close()
print 'Received', repr(data)

The big problem here seems to be that your server is closing the socket
after reading 99 bytes, but the client is actually sending 120. If you
change the "99" in the server to "120" you will see that the client
works when it uses the makefile().read(). I can't be bothered to debug
the exact reason why the 21 bytes of buffered data doesn't cause a
problem with the recv() version, but I wouldn't regard the error you are
getting as a bug in Python, rather as a bug in your application.

The underlying cause of all this appears to be your failure to
understand that network protocols should ideally allow the endpoints to
determine when a complete message has been transmitted, and to consume
all transmitted data.

You can't just ignore some or all of a client request and expect it not
to cause problems. If you look at some of the better-known TCP-based
application protocols you will see they take pains to ensure that
clients and servers can deterministically parse each others' messages.

In summary, a better protocol design will cause you rather less grief
and a little more humility will result in less acidity from crabby old
geeks like me.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------- Hide quoted text -

- Show quoted text -

I don't think you're crabby, Steve.

-- Paul

 

[ahlongxp]

That's a pretty pejorative subject line for someone who's been
programming Python [guessing by the date of your first post] for about a
month.

I have to admit it that I'm quite a newbie programmer.

Perhaps "Incomprehensible behavior from socket.makefile()", or "I have
written a buggy network application"? That would at least show that you
are considering the possibility you yourself are the cause of the
problem ;-)

Thanks. I'll be more careful about my words.
but I did show some possibility myself is the cause of the problem.
Is there any chance you missed the "?" and "guess" at the same time.

Python has been around for a long time, so you should ask yourself how
likely it is that you would be the first to discover such a fundamental
flaw?

very low but not zero.
Miracle happens.

I'd be very surprised if someone doesn't point you at an article
on "how to ask smart questions", but I will content myself with that
oblique reference.

The big problem is that, I didn't know such a person like you before.
It's my misfortune.

The big problem here seems to be that your server is closing the socket
after reading 99 bytes, but the client is actually sending 120. If you
change the "99" in the server to "120" you will see that the client
works when it uses the makefile().read(). I can't be bothered to debug
the exact reason why the 21 bytes of buffered data doesn't cause a
problem with the recv() version, but I wouldn't regard the error you are
getting as a bug in Python, rather as a bug in your application.

I don't know the underlying details of networking.
But with recv() version I can get things done as expected while
makefile()
version can't.
I guess it's not that unreasonable to make a guess like I did.

The underlying cause of all this appears to be your failure to
understand that network protocols should ideally allow the endpoints to
determine when a complete message has been transmitted, and to consume
all transmitted data.

You can't just ignore some or all of a client request and expect it not
to cause problems.

Back to the problem, I make it working by adding a sleep(0.5) just
before the
server closes the connection.
Actually this is Hendrik van Rooyen's idea.
And luckily I got a another lesson
http://groups.google.com/group/comp...2608f4c80d5/4302772dfe27d8f1#4302772dfe27d8f1

If you look at some of the better-known TCP-based
application protocols you will see they take pains to ensure that
clients and servers can deterministically parse each others' messages.

I found something useful in RFC2616--Hypertext Transfer Protocol --
HTTP/1.1.
[PAGE49]
- If an origin server receives a request that does not include
an
Expect request-header field with the "100-continue"
expectation,
the request includes a request body, and the server responds
with a final status code before reading the entire request
body
from the transport connection, then the server SHOULD NOT
close
the transport connection until it has read the entire request,
or until the client closes the connection. Otherwise, the
client
might not reliably receive the response message. However, this
requirement is not be construed as preventing a server from
defending itself against denial-of-service attacks, or from
badly broken client implementations.

"Otherwise, the client might not reliably receive the response
message".

My problem seems fixed. But it's not "relialbe" though it's working
pretty
good under windows 2k, windowsxp, and linux.
I may reconsider my design and willprobably try dividing request with
body into two steps like HTTP does.

In summary, a better protocol design will cause you rather less grief
and a little more humility will result in less acidity from crabby old
geeks like me.

I do appreciate your response.
I mean it.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

And last but not least, I' here to be helped and help as long as I
can.
But as you noticed I'm not very good at expressing myself in English.
I didn't mean to offense anyone but I might seem to be rude or
offensive.
I hope you can understand.


--
ahlongxp

Software College,Northeastern University,China
(e-mail address removed)
http://www.herofit.cn

 

[Steve Holden]

That's a pretty pejorative subject line for someone who's been
programming Python [guessing by the date of your first post] for about a
month.

[...]
And last but not least, I' here to be helped and help as long as I
can.
But as you noticed I'm not very good at expressing myself in English.
I didn't mean to offense anyone but I might seem to be rude or
offensive.
I hope you can understand.

Sure. I felt it necessary to explain my remarks as my own remarks seemed
potentially rude or offensive. Naturally not everyone has English as a
first language, but your first post was good enough that it wasn't
immediately obvious in your case.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

 

[Frank Swarbrick]

That's a pretty pejorative subject line for someone who's been
programming Python [guessing by the date of your first post] for about a
month.

I have to admit it that I'm quite a newbie programmer.

Perhaps "Incomprehensible behavior from socket.makefile()", or "I have
written a buggy network application"? That would at least show that you
are considering the possibility you yourself are the cause of the
problem ;-)

Thanks. I'll be more careful about my words.
but I did show some possibility myself is the cause of the problem.
Is there any chance you missed the "?" and "guess" at the same time.

Python has been around for a long time, so you should ask yourself how
likely it is that you would be the first to discover such a fundamental
flaw?

very low but not zero.
Miracle happens.

I'd be very surprised if someone doesn't point you at an article
on "how to ask smart questions", but I will content myself with that
oblique reference.

The big problem is that, I didn't know such a person like you before.
It's my misfortune.

The big problem here seems to be that your server is closing the socket
after reading 99 bytes, but the client is actually sending 120. If you
change the "99" in the server to "120" you will see that the client
works when it uses the makefile().read(). I can't be bothered to debug
the exact reason why the 21 bytes of buffered data doesn't cause a
problem with the recv() version, but I wouldn't regard the error you are
getting as a bug in Python, rather as a bug in your application.

I don't know the underlying details of networking.
But with recv() version I can get things done as expected while
makefile()
version can't.
I guess it's not that unreasonable to make a guess like I did.

The underlying cause of all this appears to be your failure to
understand that network protocols should ideally allow the endpoints to
determine when a complete message has been transmitted, and to consume
all transmitted data.

You can't just ignore some or all of a client request and expect it not
to cause problems.

Back to the problem, I make it working by adding a sleep(0.5) just
before the
server closes the connection.
Actually this is Hendrik van Rooyen's idea.
And luckily I got a another lesson
http://groups.google.com/group/comp...2608f4c80d5/4302772dfe27d8f1#4302772dfe27d8f1

It would be nice to know just exactly what you are trying to achieve.
Sure, adding the sleep(0.5) works, but I think that is just a
work-around and doesn't really fix the underlying problem. The
underlying problem, of course, being that the server is closing the
connection while the client is still trying to send data.

Seems to me what you really want to do is either have the send and the
recv in different threads, or use non-blocking sockets and select. Or
maybe you don't even need to use select necessarily. Seems like you
could use non-blocking sockets and then attempt a recv after each send,
in order to see if the response is ready. You want to use a
non-blocking socket so that if there's no response ready the recv will
not block waiting for one. Maybe after your last send you'd want to put
it back in blocking mode prior to doing the final recv. Or something
that that!

Hope that gives you a start, or at least some better questions to ask.

Frank

 

[John J. Lee]

That's a pretty pejorative subject line for someone who's been
programming Python [guessing by the date of your first post] for about a
month.

[...]
And last but not least, I' here to be helped and help as long as I
can.
But as you noticed I'm not very good at expressing myself in English.
I didn't mean to offense anyone but I might seem to be rude or
offensive.
I hope you can understand.

Sure. I felt it necessary to explain my remarks as my own remarks
seemed potentially rude or offensive. Naturally not everyone has
English as a first language, but your first post was good enough that
it wasn't immediately obvious in your case.

This is the danger in getting too good at another language
ahlongxp, maybe you should drop some deliberate mistakes in your
too-good English ;-)

And FWLIW, I'd be less surprised than Steve at your finding a real bug
in Python's socket code...


John

 

[ahlongxp]

That's a pretty pejorative subject line for someone who's been
programming Python [guessing by the date of your first post] for about a
month.

[...]
And last but not least, I' here to be helped and help as long as I
can.
But as you noticed I'm not very good at expressing myself in English.
I didn't mean to offense anyone but I might seem to be rude or
offensive.
I hope you can understand.

Sure. I felt it necessary to explain my remarks as my own remarks
seemed potentially rude or offensive. Naturally not everyone has
English as a first language, but your first post was good enough that
it wasn't immediately obvious in your case.

This is the danger in getting too good at another language
ahlongxp, maybe you should drop some deliberate mistakes in your
too-good English ;-)

And FWLIW, I'd be less surprised than Steve at your finding a real bug
in Python's socket code...

John

Sorry, I don't think I understand what you are trying to say.
But thanks for your response anyway.

Frank Swarbrick, thank your for your advice. I'll think about it.

--
ahlongxp

Software College,Northeastern University,China
(e-mail address removed)
http://www.herofit.cn