Buffer size when receiving data through a socket?

Discussion in 'Python' started by John Salerno, Jun 17, 2008.

  1. John Salerno

    John Salerno Guest

    I wrote some pretty basic socket programming again, but I'm still confused about what's happening with the buffer_size variable. Here are the server and client programs:

    --------------

    from socket import *

    host = ''
    port = 51567
    address = (host, port)
    buffer_size = 1024

    server_socket = socket(AF_INET, SOCK_STREAM)
    server_socket.bind(address)
    server_socket.listen(5)

    while True:
    print 'waiting for connection...'
    client_socket, client_address = server_socket.accept()
    print '...connected from:', client_address

    while True:
    data = client_socket.recv(buffer_size)
    if not data:
    break
    client_socket.send('%s %s' % ('You typed:', data))

    client_socket.close()

    server_socket.close()

    ------------

    from socket import *

    host = 'localhost'
    port = 51567
    address = (host, port)
    buffer_size = 1024

    client_socket = socket(AF_INET, SOCK_STREAM)
    client_socket.connect(address)

    while True:
    data = raw_input('> ')
    if not data:
    break
    client_socket.send(data)
    data = client_socket.recv(buffer_size)
    if not data:
    break
    print data

    client_socket.close()

    ---------------

    I tried changing buffer_size to 10 and I got this output:

    john@john-laptop:~$ python myclient.py
    > hello

    You typed:
    > something

    hello
    > this is a long string

    You typed:
    > why doesn't this work right

    something
    >

    john@john-laptop:~$

    My first question is, isn't buffer_size the number of bytes being sent at one time? If so, why doesn't 'hello' get printed after the server returns the data to the client? Isn't 'hello' just 5 bytes?

    Secondly, how is it working that once I type in a new string (e.g. 'something') and then the server returns data to the client, it prints the *previous* string, (i.e. 'hello')? Wouldn't the data variable get overwritten with the value, or is the value being stored somewhere else at this point?

    Thanks!
     
    John Salerno, Jun 17, 2008
    #1
    1. Advertising

  2. En Mon, 16 Jun 2008 21:21:35 -0300, John Salerno <> escribió:

    > I wrote some pretty basic socket programming again, but I'm still confused about what's happening with the buffer_size variable. Here are the server and client programs:
    >
    > --------------
    >
    > from socket import *
    >
    > host = ''
    > port = 51567
    > address = (host, port)
    > buffer_size = 1024
    >
    > server_socket = socket(AF_INET, SOCK_STREAM)
    > server_socket.bind(address)
    > server_socket.listen(5)
    >
    > while True:
    > print 'waiting for connection...'
    > client_socket, client_address = server_socket.accept()
    > print '...connected from:', client_address
    >
    > while True:
    > data = client_socket.recv(buffer_size)
    > if not data:
    > break
    > client_socket.send('%s %s' % ('You typed:', data))
    >
    > client_socket.close()
    >
    > server_socket.close()
    >
    > ------------
    >
    > from socket import *
    >
    > host = 'localhost'
    > port = 51567
    > address = (host, port)
    > buffer_size = 1024
    >
    > client_socket = socket(AF_INET, SOCK_STREAM)
    > client_socket.connect(address)
    >
    > while True:
    > data = raw_input('> ')
    > if not data:
    > break
    > client_socket.send(data)
    > data = client_socket.recv(buffer_size)
    > if not data:
    > break
    > print data
    >
    > client_socket.close()
    >
    > ---------------
    >
    > I tried changing buffer_size to 10 and I got this output:
    >
    > john@john-laptop:~$ python myclient.py
    >> hello

    > You typed:
    >> something

    > hello
    >> this is a long string

    > You typed:
    >> why doesn't this work right

    > something
    >>

    > john@john-laptop:~$
    >
    > My first question is, isn't buffer_size the number of bytes being sent at one time? If so, why doesn't 'hello' get printed after the server returns the data to the client? Isn't 'hello' just 5 bytes?


    Both programs say recv(buffer_size) - buffer_size is the maximum number of bytes to be RECEIVED, that is, READ. recv will return at most buffer_size bytes. It may return less than that, even if the other side sent the data in a single operation.
    Note that most of the time you want to use the sendall() method, because send() doesn't guarantee that all the data was actually sent. <http://docs.python.org/lib/socket-objects.html>

    > Secondly, how is it working that once I type in a new string (e.g. 'something') and then the server returns data to the client, it prints the *previous* string, (i.e. 'hello')? Wouldn't the data variable get overwritten with the value, or is the value being stored somewhere else at this point?


    Yes, it is stored in an intermediate buffer until you read it. You typed "hello" and sent it, the server replied with the string "You typed: hello"; the OS stores it. You read only 10 bytes "You typed:", the remaining are still in the buffer. Next round: you type something, the server replies, you read the remaining bytes from the original reply, and so on...

    (Note that in this particular configuration, the client will fill its buffer at some time: because the server sends at least 11 bytes each round, but the client reads at most 10 bytes, so the client is always behind the server...)

    --
    Gabriel Genellina
     
    Gabriel Genellina, Jun 17, 2008
    #2
    1. Advertising

  3. John Salerno

    John Salerno Guest

    "Gabriel Genellina" <> wrote in message
    news:...
    > Both programs say recv(buffer_size) - buffer_size is the maximum number of
    > bytes to be RECEIVED, that is, READ. recv will return at most buffer_size
    > bytes. It may return less than that, even if the other side sent the data
    > in a single operation.
    > Note that most of the time you want to use the sendall() method, because
    > send() doesn't guarantee that all the data was actually sent.
    > <http://docs.python.org/lib/socket-objects.html>


    I was wondering about sendall(). The examples I've read in two different
    books are consistent in their use of send() and don't even mention
    sendall(), so I thought maybe it was for a more specialized situation.

    > Yes, it is stored in an intermediate buffer until you read it. You typed
    > "hello" and sent it, the server replied with the string "You typed:
    > hello"; the OS stores it. You read only 10 bytes "You typed:", the
    > remaining are still in the buffer. Next round: you type something, the
    > server replies, you read the remaining bytes from the original reply, and
    > so on...


    Oh!!!! I didn't even count "You typed:" as part of the 10 bytes! And what a
    coincidence that it happens to be exactly 10 characters! That really helped
    to hide the problem from me!

    > (Note that in this particular configuration, the client will fill its
    > buffer at some time: because the server sends at least 11 bytes each
    > round, but the client reads at most 10 bytes, so the client is always
    > behind the server...)


    How is the server sending back 11 bytes? Is it because it's sending at least
    the 10 characters, plus the extra space?

    Thanks!
     
    John Salerno, Jun 17, 2008
    #3
  4. John Salerno

    John Salerno Guest

    "John Salerno" <> wrote in message
    news:...
    > from socket import *
    >
    > host = 'localhost'
    > port = 51567
    > address = (host, port)
    > buffer_size = 1024
    >
    > client_socket = socket(AF_INET, SOCK_STREAM)
    > client_socket.connect(address)
    >
    > while True:
    > data = raw_input('> ')
    > if not data:
    > break
    > client_socket.send(data)
    > data = client_socket.recv(buffer_size)
    > if not data:
    > break
    > print data
    >
    > client_socket.close()


    Also, is that second "if not data: break" statement necessary? It seems like
    once you get past the first if, you don't need the second one. Of course, I
    guses it's possible that the server could return a False value, but even
    still, would it make sense to break out of the loop and close the connection
    because of that?

    It runs fine without the if statement, but I'm wondering if I just haven't
    encountered the proper problem situation yet.
     
    John Salerno, Jun 17, 2008
    #4
  5. John Salerno

    John Salerno Guest

    "Gabriel Genellina" <> wrote in message
    news:...
    > Note that most of the time you want to use the sendall() method, because
    > send() doesn't guarantee that all the data was actually sent.
    > <http://docs.python.org/lib/socket-objects.html>


    If I use sendall(), am I still recv'ing data with a given buffer size? What
    if I send more data than the buffer size. Is my code as written not prepared
    to handle that case? It seems like I might need to continue receiving data
    until there is no more to receive (in a loop?)...is that right?
     
    John Salerno, Jun 17, 2008
    #5
  6. En Tue, 17 Jun 2008 10:34:24 -0300, John Salerno <> escribió:

    > I was wondering about sendall(). The examples I've read in two different
    > books are consistent in their use of send() and don't even mention
    > sendall(), so I thought maybe it was for a more specialized situation.


    «Stream sockets (e.g., TCP sockets) exhibit a behavior with the read and write functions that differs from normal file I/O. A read or write on a stream socket might input or output fewer bytes than requested, but this is not an error condition. The reason is that buffer limits might be reached for the socket in the kernel. All that is required to input or output the remaining bytes is for the caller to invoke the read or write function again. Some versions of Unix also exhibit this behavior when writing more than 4,096 bytes to a pipe. This scenario is always a possibility on a stream socket with read, but is normally seen with write only if the socket is nonblocking. Nevertheless, we always call our writen function instead of write, in case the implementation returns a short count.» [1]

    The Python `sendall` method is equivalent to the `writen` function they refer to. Rather than analyzing in each case whether sendall is to be required or not, I prefer to always use it and forget about it...

    > Oh!!!! I didn't even count "You typed:" as part of the 10 bytes! And what a
    > coincidence that it happens to be exactly 10 characters! That really helped
    > to hide the problem from me!


    :)

    >> (Note that in this particular configuration, the client will fill its
    >> buffer at some time: because the server sends at least 11 bytes each
    >> round, but the client reads at most 10 bytes, so the client is always
    >> behind the server...)

    >
    > How is the server sending back 11 bytes? Is it because it's sending at least
    > the 10 characters, plus the extra space?


    Yes, 10 + whatever you typed in the client side (at least one character - an empty string exits the program).

    [1] Richard Stevens et al.: UNIX Network Programming, Volume 1, Third Edition: The Sockets Networking API. Section 3.9

    --
    Gabriel Genellina
     
    Gabriel Genellina, Jun 18, 2008
    #6
  7. En Tue, 17 Jun 2008 14:32:44 -0300, John Salerno <> escribió:

    > "Gabriel Genellina" <> wrote in message
    > news:...
    >> Note that most of the time you want to use the sendall() method, because
    >> send() doesn't guarantee that all the data was actually sent.
    >> <http://docs.python.org/lib/socket-objects.html>

    >
    > If I use sendall(), am I still recv'ing data with a given buffer size? What
    > if I send more data than the buffer size. Is my code as written not prepared
    > to handle that case? It seems like I might need to continue receiving data
    > until there is no more to receive (in a loop?)...is that right?


    send and recv are separate calls (they occur usually in different processes, even in different computers). Buffer sizes are separate too. It is posible to send 5K at once from one side, and require three recv calls on the other side to read it completely. On the other hand, if you try to read from a blocking socket (the default state) when no data is available, the read call will block (and the whole program freezes) until some data is received. There are several alternatives to avoid this, and surely they're explained in detail in a later chapter in your book...

    --
    Gabriel Genellina
     
    Gabriel Genellina, Jun 18, 2008
    #7
  8. En Tue, 17 Jun 2008 14:32:44 -0300, John Salerno <> escribió:

    > "Gabriel Genellina" <> wrote in message
    > news:...
    >> Note that most of the time you want to use the sendall() method, because
    >> send() doesn't guarantee that all the data was actually sent.
    >> <http://docs.python.org/lib/socket-objects.html>

    >
    > If I use sendall(), am I still recv'ing data with a given buffer size? What
    > if I send more data than the buffer size. Is my code as written not prepared
    > to handle that case? It seems like I might need to continue receiving data
    > until there is no more to receive (in a loop?)...is that right?


    send and recv are separate calls (they occur usually in different processes, even in different computers). Buffer sizes are separate too. It is posible to send 5K at once from one side, and require three recv calls on the other side to read it completely. On the other hand, if you try to read from a blocking socket (the default state) when no data is available, the read call will block (and the whole program freezes) until some data is received. There are several alternatives to avoid this, and surely they're explained in detail in a later chapter in your book...

    --
    Gabriel Genellina
     
    Gabriel Genellina, Jun 18, 2008
    #8
  9. John Salerno

    MRAB Guest

    On Jun 18, 7:52 am, Dennis Lee Bieber <> wrote:
    > On Tue, 17 Jun 2008 09:39:07 -0400, "John Salerno"
    > <> declaimed the following in comp.lang.python:
    >
    >
    >
    > > > while True:
    > > >    data = raw_input('> ')
    > > >    if not data:
    > > >        break
    > > >    client_socket.send(data)
    > > >    data = client_socket.recv(buffer_size)
    > > >    if not data:
    > > >        break
    > > >    print data

    >
    > > > client_socket.close()

    >
    > > Also, is that second "if not data: break" statement necessary? It seems like
    > > once you get past the first if, you don't need the second one. Of course, I
    > > guses it's possible that the server could return a False value, but even
    > > still, would it make sense to break out of the loop and close the connection
    > > because of that?

    >
    >         The first if is checking for lack of interactive input -- and, as
    > coded, will never break out as ANY response to the > prompt will have a
    > newline attached.
    >

    [snip]
    FYI, I've just checked:

    >>> raw_input('> ')

    > abc

    'abc'

    raw_input() doesn't put a newline on the end.
     
    MRAB, Jun 18, 2008
    #9
  10. John Salerno

    John Salerno Guest

    "Dennis Lee Bieber" <> wrote in message
    news:...
    > The first if is checking for lack of interactive input -- and, as
    > coded, will never break out as ANY response to the > prompt will have a
    > newline attached.
    >
    > Try with raw_input("> ").strip() instead


    Well, I know the first if block works properly. Pressing just ENTER will
    exit the loop and close the client socket.

    > The second if is checking for empty receive block... And since
    > .recv() blocks until it has something to return (as I recall) it may not
    > be of use...


    Interesting point. I'm not sure if it works that way though. I *think* I
    tried sending an empty string from the server back to the client, and as
    expected it exited the loop and closed the client, which doesn't make sense
    to me, since an empty string could be perfectly valid return data.

    I opted to remove the second if statement and see where that takes me. :)
     
    John Salerno, Jun 18, 2008
    #10
  11. John Salerno

    John Salerno Guest

    Dennis Lee Bieber wrote:
    > On Wed, 18 Jun 2008 09:56:29 -0400, "John Salerno"
    > <> declaimed the following in comp.lang.python:
    >
    >> Interesting point. I'm not sure if it works that way though. I *think* I
    >> tried sending an empty string from the server back to the client, and as
    >> expected it exited the loop and closed the client, which doesn't make sense
    >> to me, since an empty string could be perfectly valid return data.
    >>

    > Okay... I suppose socket could consider a transmission with 0 data
    > bytes as valid "data". The main key is that there had to be send of 0
    > data -- opposed to just not receiving anything..


    No, I think you're right. I read elsewhere that when I send() call
    returns 0 bytes, the connection is automatically closed.
     
    John Salerno, Jun 22, 2008
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    0
    Views:
    514
  2. riloo
    Replies:
    0
    Views:
    817
    riloo
    Apr 27, 2004
  3. Pep
    Replies:
    2
    Views:
    461
  4. Raja
    Replies:
    12
    Views:
    24,715
    John Harrison
    Jun 21, 2004
  5. John Ramsden
    Replies:
    1
    Views:
    145
    Brian McCauley
    Aug 5, 2003
Loading...

Share This Page