strange sockets

[Skink]

Hi,

I'm preparing a python server that sends java classes and resources to
custom java class loader. In order to make it faster I don't want to use
URLClassLoader that uses HTTP protocol 1.0 and for each class/resource
creates own connection.
Instead I'd like to use raw sockets with simple protocol:

- class loader sends a line terminated with \n with resource to get
- python server reads that line, gets the file and sends back an
integer with file length and then the file itself
- class loader reads a lenght integer and then reads the remainig data


The problem is when I try to read several files the first one is read
quite fast, but the rest is read 40 x slower. For example (time is in
seconds):

% python client.py client.py client.py client.py server.py server.py
init 0.00066089630127
client.py 0.000954866409302
client.py 0.0408389568329
client.py 0.0409188270569
server.py 0.0409059524536
server.py 0.0409259796143

what's wrong here?

thanks,
skink

client.py
------------------------------------------------------------------------------------
import socket, sys, struct, time

HOST = 'localhost'
PORT = 8080
t1 = time.time()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
t2 = time.time()
print "init", t2-t1
for arg in sys.argv[1:]:
t1 = time.time()
s.send(arg + "\n")
len, = struct.unpack("!i", s.recv(4))
data = s.recv(len)
t2 = time.time()
print arg, t2-t1
s.close()
------------------------------------------------------------------------------------

server.py
------------------------------------------------------------------------------------
import socket, struct, binascii

HOST = ''
PORT = 8080
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
while 1:
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr
f = conn.makefile()
while 1:
resource = f.readline().rstrip()
print "[%s]" % resource
if not resource:
break
data = open(resource, "rb").read()
conn.sendall(struct.pack("!i", len(data)))
conn.sendall(data)
conn.close()
------------------------------------------------------------------------------------

 

[Sion Arrowsmith]

% python client.py client.py client.py client.py server.py server.py
init 0.00066089630127
client.py 0.000954866409302
client.py 0.0408389568329
client.py 0.0409188270569
server.py 0.0409059524536
server.py 0.0409259796143

what's wrong here?

That smells of a Nagle/delayed ACK problem to me (see, for instance,
http://www.port80software.com/200ok/archive/2005/01/31/317.aspx). 40ms
is the default delayed ACK timeout on Linux, IIRC (pretty much
everything else uses 200ms). I *think* what's happening from the
server's point of view is:

receive request 1
send length (first undersized packet is sent immediately by Nagle)
(client delays ack #1)
send data (larger than 1 packet, send immediately)
(client delays acks #2--#n)
receive request 2 with ack #1
buffer sending length (undersized packet, not received last ack)
-> 40ms passes <-
(client timesout delayed acks and sends)
send length
send data (as before)

although why the undersized packet at the end of the first chunk of
data isn't buffered, I don't know.

Solutions: either change

conn.sendall(struct.pack("!i", len(data)))
conn.sendall(data)

to

conn.sendall(struct.pack("!i", len(data)) + data)

or after creating conn

conn.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)

to disable Nagle.

 

[Jim Segrave]

Hi,

I'm preparing a python server that sends java classes and resources to
custom java class loader. In order to make it faster I don't want to use
URLClassLoader that uses HTTP protocol 1.0 and for each class/resource
creates own connection.
Instead I'd like to use raw sockets with simple protocol:

- class loader sends a line terminated with \n with resource to get
- python server reads that line, gets the file and sends back an
integer with file length and then the file itself
- class loader reads a lenght integer and then reads the remainig data


The problem is when I try to read several files the first one is read
quite fast, but the rest is read 40 x slower. For example (time is in
seconds):

% python client.py client.py client.py client.py server.py server.py
init 0.00066089630127
client.py 0.000954866409302
client.py 0.0408389568329
client.py 0.0409188270569
server.py 0.0409059524536
server.py 0.0409259796143

what's wrong here?

At a guess, what you've measured is how long it takes to transfer the
data to the underlying OS socket buffers. The first transfer fills the
buffers, subsequent ones have to wait until the data has been put on
the wire and acknowledged before there's space for the writes.

 

[Bryan Olson]

Skink wrote:
[...]

what's wrong here?

Sion Arrowsmith is right about what causes the delay.
Just in case your real code looks like this, I'll note:

len, = struct.unpack("!i", s.recv(4))
data = s.recv(len)

First, you almost certainly don't want to use the name 'len'.
Ought not to be allowed. Second, recv can return fewer bytes
than requested, even when the connection is still open for
reading. You might replace the lines above with (untested):

length = struct.unpack("!i", s.recv(4))
data = []
while length:
data.append(s.recv(length))
length -= len(data[-1])
data = ''.join(data)


There's still a robustness problem, but in the absense of errors
and malice, that should work. I think.

 

[Skink]

Sion,

Solutions: either change




to

conn.sendall(struct.pack("!i", len(data)) + data)

or after creating conn

conn.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)

to disable Nagle.

thank yuo so much, both solutions work perfect!

%python client.py client.py client.py client.py
init 0.00101184844971
client.py 0.000586986541748
client.py 0.000448942184448
client.py 0.000470161437988

I think that I'll use the second one


skink

 

[Skink]

Bryan,

Sion Arrowsmith is right about what causes the delay.
Just in case your real code looks like this, I'll note:

yes, my mistake

First, you almost certainly don't want to use the name 'len'.
Ought not to be allowed. Second, recv can return fewer bytes
than requested, even when the connection is still open for
reading. You might replace the lines above with (untested):

length = struct.unpack("!i", s.recv(4))
data = []
while length:
data.append(s.recv(length))
length -= len(data[-1])
data = ''.join(data)

i know, i know, i sent fake python client: the real will be done in java.

 

[Skink]

conn.sendall(struct.pack("!i", len(data)) + data)

or after creating conn

conn.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)

to disable Nagle.

Sion,

thank you for your help,

it works but...
it works when client & server is in python
i tried both solutions and they work when client is client.py
they both don't work when client is java client
when i tried to connect python's server by java client i have the same:

% java Loader server.py server.py server.py
init 29
server.py reading 631 1
server.py reading 631 40
server.py reading 631 41

why?

thanks,
skink.

 

[Steve Holden]

Sion,

thank you for your help,

it works but...
it works when client & server is in python
i tried both solutions and they work when client is client.py
they both don't work when client is java client
when i tried to connect python's server by java client i have the same:

% java Loader server.py server.py server.py
init 29
server.py reading 631 1
server.py reading 631 40
server.py reading 631 41

why?

Seems to me that should probably be a question for comp.lang.java.

regards
Steve

 

[Skink]

Seems to me that should probably be a question for comp.lang.java.

ok, my falt. again...
i forgot to use Buffered[Input|Output]Stream