Buffer size when receiving data through a socket?

[John Salerno]

I wrote some pretty basic socket programming again, but I'm still confused about what's happening with the buffer_size variable. Here are the server and client programs:

--------------

from socket import *

host = ''
port = 51567
address = (host, port)
buffer_size = 1024

server_socket = socket(AF_INET, SOCK_STREAM)
server_socket.bind(address)
server_socket.listen(5)

while True:
print 'waiting for connection...'
client_socket, client_address = server_socket.accept()
print '...connected from:', client_address

while True:
data = client_socket.recv(buffer_size)
if not data:
break
client_socket.send('%s %s' % ('You typed:', data))

client_socket.close()

server_socket.close()

------------

from socket import *

host = 'localhost'
port = 51567
address = (host, port)
buffer_size = 1024

client_socket = socket(AF_INET, SOCK_STREAM)
client_socket.connect(address)

while True:
data = raw_input('> ')
if not data:
break
client_socket.send(data)
data = client_socket.recv(buffer_size)
if not data:
break
print data

client_socket.close()

---------------

I tried changing buffer_size to 10 and I got this output:

john@john-laptop:~$ python myclient.py

hello You typed:
something hello
this is a long string You typed:
why doesn't this work right something

john@john-laptop:~$

My first question is, isn't buffer_size the number of bytes being sent at one time? If so, why doesn't 'hello' get printed after the server returns the data to the client? Isn't 'hello' just 5 bytes?

Secondly, how is it working that once I type in a new string (e.g. 'something') and then the server returns data to the client, it prints the *previous* string, (i.e. 'hello')? Wouldn't the data variable get overwritten with the value, or is the value being stored somewhere else at this point?

Thanks!

 

[Gabriel Genellina]

I wrote some pretty basic socket programming again, but I'm still confused about what's happening with the buffer_size variable. Here are the server and client programs:

--------------

from socket import *

host = ''
port = 51567
address = (host, port)
buffer_size = 1024

server_socket = socket(AF_INET, SOCK_STREAM)
server_socket.bind(address)
server_socket.listen(5)

while True:
print 'waiting for connection...'
client_socket, client_address = server_socket.accept()
print '...connected from:', client_address

while True:
data = client_socket.recv(buffer_size)
if not data:
break
client_socket.send('%s %s' % ('You typed:', data))

client_socket.close()

server_socket.close()

------------

from socket import *

host = 'localhost'
port = 51567
address = (host, port)
buffer_size = 1024

client_socket = socket(AF_INET, SOCK_STREAM)
client_socket.connect(address)

while True:
data = raw_input('> ')
if not data:
break
client_socket.send(data)
data = client_socket.recv(buffer_size)
if not data:
break
print data

client_socket.close()

---------------

I tried changing buffer_size to 10 and I got this output:

john@john-laptop:~$ python myclient.py
john@john-laptop:~$

My first question is, isn't buffer_size the number of bytes being sent at one time? If so, why doesn't 'hello' get printed after the server returns the data to the client? Isn't 'hello' just 5 bytes?

Both programs say recv(buffer_size) - buffer_size is the maximum number of bytes to be RECEIVED, that is, READ. recv will return at most buffer_size bytes. It may return less than that, even if the other side sent the data in a single operation.

Secondly, how is it working that once I type in a new string (e.g. 'something') and then the server returns data to the client, it prints the *previous* string, (i.e. 'hello')? Wouldn't the data variable get overwritten with the value, or is the value being stored somewhere else at this point?

Yes, it is stored in an intermediate buffer until you read it. You typed "hello" and sent it, the server replied with the string "You typed: hello"; the OS stores it. You read only 10 bytes "You typed:", the remaining are still in the buffer. Next round: you type something, the server replies, you read the remaining bytes from the original reply, and so on...

(Note that in this particular configuration, the client will fill its buffer at some time: because the server sends at least 11 bytes each round, but the client reads at most 10 bytes, so the client is always behind the server...)

 

[John Salerno]

Both programs say recv(buffer_size) - buffer_size is the maximum number of
bytes to be RECEIVED, that is, READ. recv will return at most buffer_size
bytes. It may return less than that, even if the other side sent the data
in a single operation.
Note that most of the time you want to use the sendall() method, because
send() doesn't guarantee that all the data was actually sent.
<http://docs.python.org/lib/socket-objects.html>

I was wondering about sendall(). The examples I've read in two different
books are consistent in their use of send() and don't even mention
sendall(), so I thought maybe it was for a more specialized situation.

Yes, it is stored in an intermediate buffer until you read it. You typed
"hello" and sent it, the server replied with the string "You typed:
hello"; the OS stores it. You read only 10 bytes "You typed:", the
remaining are still in the buffer. Next round: you type something, the
server replies, you read the remaining bytes from the original reply, and
so on...

Oh!!!! I didn't even count "You typed:" as part of the 10 bytes! And what a
coincidence that it happens to be exactly 10 characters! That really helped
to hide the problem from me!

(Note that in this particular configuration, the client will fill its
buffer at some time: because the server sends at least 11 bytes each
round, but the client reads at most 10 bytes, so the client is always
behind the server...)

How is the server sending back 11 bytes? Is it because it's sending at least
the 10 characters, plus the extra space?

Thanks!

 

[John Salerno]

from socket import *

host = 'localhost'
port = 51567
address = (host, port)
buffer_size = 1024

client_socket = socket(AF_INET, SOCK_STREAM)
client_socket.connect(address)

while True:
data = raw_input('> ')
if not data:
break
client_socket.send(data)
data = client_socket.recv(buffer_size)
if not data:
break
print data

client_socket.close()

Also, is that second "if not data: break" statement necessary? It seems like
once you get past the first if, you don't need the second one. Of course, I
guses it's possible that the server could return a False value, but even
still, would it make sense to break out of the loop and close the connection
because of that?

It runs fine without the if statement, but I'm wondering if I just haven't
encountered the proper problem situation yet.

 

[John Salerno]

Note that most of the time you want to use the sendall() method, because
send() doesn't guarantee that all the data was actually sent.
<http://docs.python.org/lib/socket-objects.html>

If I use sendall(), am I still recv'ing data with a given buffer size? What
if I send more data than the buffer size. Is my code as written not prepared
to handle that case? It seems like I might need to continue receiving data
until there is no more to receive (in a loop?)...is that right?

 

[Gabriel Genellina]

I was wondering about sendall(). The examples I've read in two different
books are consistent in their use of send() and don't even mention
sendall(), so I thought maybe it was for a more specialized situation.

«Stream sockets (e.g., TCP sockets) exhibit a behavior with the read and write functions that differs from normal file I/O. A read or write on a stream socket might input or output fewer bytes than requested, but this is not an error condition. The reason is that buffer limits might be reached for the socket in the kernel. All that is required to input or output the remaining bytes is for the caller to invoke the read or write function again. Some versions of Unix also exhibit this behavior when writing more than 4,096 bytes to a pipe. This scenario is always a possibility on a stream socket with read, but is normally seen with write only if the socket is nonblocking. Nevertheless, we always call our writen function instead of write, in case the implementation returns a short count.» [1]

The Python `sendall` method is equivalent to the `writen` function they refer to. Rather than analyzing in each case whether sendall is to be required or not, I prefer to always use it and forget about it...

Oh!!!! I didn't even count "You typed:" as part of the 10 bytes! And what a
coincidence that it happens to be exactly 10 characters! That really helped
to hide the problem from me!



How is the server sending back 11 bytes? Is it because it's sending at least
the 10 characters, plus the extra space?

Yes, 10 + whatever you typed in the client side (at least one character - an empty string exits the program).

[1] Richard Stevens et al.: UNIX Network Programming, Volume 1, Third Edition: The Sockets Networking API. Section 3.9

 

[Gabriel Genellina]

If I use sendall(), am I still recv'ing data with a given buffer size? What
if I send more data than the buffer size. Is my code as written not prepared
to handle that case? It seems like I might need to continue receiving data
until there is no more to receive (in a loop?)...is that right?

send and recv are separate calls (they occur usually in different processes, even in different computers). Buffer sizes are separate too. It is posible to send 5K at once from one side, and require three recv calls on the other side to read it completely. On the other hand, if you try to read from a blocking socket (the default state) when no data is available, the read call will block (and the whole program freezes) until some data is received. There are several alternatives to avoid this, and surely they're explained in detail in a later chapter in your book...

 

[Gabriel Genellina]

If I use sendall(), am I still recv'ing data with a given buffer size? What
if I send more data than the buffer size. Is my code as written not prepared
to handle that case? It seems like I might need to continue receiving data
until there is no more to receive (in a loop?)...is that right?

send and recv are separate calls (they occur usually in different processes, even in different computers). Buffer sizes are separate too. It is posible to send 5K at once from one side, and require three recv calls on the other side to read it completely. On the other hand, if you try to read from a blocking socket (the default state) when no data is available, the read call will block (and the whole program freezes) until some data is received. There are several alternatives to avoid this, and surely they're explained in detail in a later chapter in your book...

 

[MRAB]

        The first if is checking for lack of interactive input -- and, as
coded, will never break out as ANY response to the > prompt will have a
newline attached.

[snip]
FYI, I've just checked:

abc

'abc'

raw_input() doesn't put a newline on the end.

 

[John Salerno]

The first if is checking for lack of interactive input -- and, as
coded, will never break out as ANY response to the > prompt will have a
newline attached.

Try with raw_input("> ").strip() instead

Well, I know the first if block works properly. Pressing just ENTER will
exit the loop and close the client socket.

The second if is checking for empty receive block... And since
.recv() blocks until it has something to return (as I recall) it may not
be of use...

Interesting point. I'm not sure if it works that way though. I *think* I
tried sending an empty string from the server back to the client, and as
expected it exited the loop and closed the client, which doesn't make sense
to me, since an empty string could be perfectly valid return data.

I opted to remove the second if statement and see where that takes me.

 

[John Salerno]

Okay... I suppose socket could consider a transmission with 0 data
bytes as valid "data". The main key is that there had to be send of 0
data -- opposed to just not receiving anything..

No, I think you're right. I read elsewhere that when I send() call
returns 0 bytes, the connection is automatically closed.