Re: sending very large packets over the network

Discussion in 'Python' started by Gary Herron, Aug 2, 2007.

  1. Gary Herron

    Gary Herron Guest

    Walker Lindley wrote:
    > OK, I'm back with another networking question. I'm trying to seend
    > large amounts of information over TCP (the length of data being given
    > to send() is on the order of 16000 characters in length).
    > Unfortunately on the receiving end, the packets appear to be
    > truncated. So I wrote some code that continuously tries to send bigger
    > and bigger packets until it fails and noticed that it never fails at
    > the same length. I'm not even sure these two things are related, but
    > is there some undocumented (or documented and I missed it) maximum
    > size for data you can pass to send()?

    For ethernet connections the size is often about 1500. But the size
    depends on the underlying protocol, and even if you know the underlying
    protocol along the full route, you can't rely on packets that are
    received being the same as those that are sent.

    TCP/IP is a *stream* connection. What you are guaranteed is this: All
    the bytes that are sent from one end will be received eventually on the
    other end, in the proper order. (Or failing that, you will receive an
    error notification.) No guarantee is made about the packet sizes on
    either end, and you can't even rely on the packets being the same in
    number or length on the two ends.

    Your send code can try sending a packet of any size, but it must be
    prepared to examine the number of bytes actually sent, and retry with
    the remainder in a loop until all bytes are sent. Similarly your
    receiving code must loop around the receive accepting whatever sized
    packets makes it through the connection.

    In many TCP/IP connections, it seems that the packets received are
    one-for-one with the packets sent, but relying on this *IS AN ERROR*
    that will bite you. It fails to across the internet (sometimes) and
    when (at least some) wireless cards are involved.

    You may be better off using a package that knows all this and handles it
    properly. Modules asyncore and asynchat are one possibility.


    Gary Herron
    >
    > the sample code is as follows
    > #server
    > import socket
    >
    > s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    > s.bind(("", 2000))
    > s.listen(0)
    > sock, addrinfo = s.accept()
    > for i in range(2 ** 16):
    > length = int(sock.recv(16))
    > print "excpecting data of length:", length
    > data = sock.recv(length)
    > print "received data of length:", len(data)
    > print
    > s.close()
    > sock.close()
    >
    > #client
    > import socket
    > import time
    >
    > s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    > s.bind(("", 2001))
    > s.connect(("localhost", 2000))
    > for i in range(2 ** 16):
    > packet = "h" * i
    > s.send("%16d" % len(packet))
    > print "attempting to send data of length:", len(packet)
    > tmp = s.send(packet)
    > print "actually sent data of length:", tmp
    > print
    > time.sleep(.001)
    > s.close()
    >
    > i just put them in different files and ran them from the command line.
    > Any help or suggestions would be greatly appreciated. Thanks.
    >
    >
    > -Walker
    >
    > --
    > This e-mail is licensed under the Creative Commons
    > Attribution-NoDerivs 2.5 License. To view a copy of this license,
    > visit http://creativecommons.org/licenses/by-nd/2.5/ or send a letter
    > to Creative Commons, 543 Howard Street, 5th Floor, San Francisco,
    > California, 94105, USA.
     
    Gary Herron, Aug 2, 2007
    #1
    1. Advertising

  2. Gary Herron

    Azazello Guest

    On Aug 1, 8:50 pm, Gary Herron <> wrote:
    > Walker Lindley wrote:
    > > OK, I'm back with another networking question. I'm trying to seend
    > > large amounts of information over TCP (the length of data being given
    > > to send() is on the order of 16000 characters in length).
    > > Unfortunately on the receiving end, the packets appear to be
    > > truncated. So I wrote some code that continuously tries to send bigger
    > > and bigger packets until it fails and noticed that it never fails at
    > > the same length. I'm not even sure these two things are related, but
    > > is there some undocumented (or documented and I missed it) maximum
    > > size for data you can pass to send()?

    >
    > For ethernet connections the size is often about 1500. But the size
    > depends on the underlying protocol, and even if you know the underlying
    > protocol along the full route, you can't rely on packets that are
    > received being the same as those that are sent.
    >
    > TCP/IP is a *stream* connection. What you are guaranteed is this: All
    > the bytes that are sent from one end will be received eventually on the
    > other end, in the proper order. (Or failing that, you will receive an
    > error notification.) No guarantee is made about the packet sizes on
    > either end, and you can't even rely on the packets being the same in
    > number or length on the two ends.
    >
    > Your send code can try sending a packet of any size, but it must be
    > prepared to examine the number of bytes actually sent, and retry with
    > the remainder in a loop until all bytes are sent. Similarly your
    > receiving code must loop around the receive accepting whatever sized
    > packets makes it through the connection.
    >
    > In many TCP/IP connections, it seems that the packets received are
    > one-for-one with the packets sent, but relying on this *IS AN ERROR*
    > that will bite you. It fails to across the internet (sometimes) and
    > when (at least some) wireless cards are involved.
    >
    > You may be better off using a package that knows all this and handles it
    > properly. Modules asyncore and asynchat are one possibility.
    >
    > Gary Herron
    >
    >
    >
    >
    >
    > > the sample code is as follows
    > > #server
    > > import socket

    >
    > > s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    > > s.bind(("", 2000))
    > > s.listen(0)
    > > sock, addrinfo = s.accept()
    > > for i in range(2 ** 16):
    > > length = int(sock.recv(16))
    > > print "excpecting data of length:", length
    > > data = sock.recv(length)
    > > print "received data of length:", len(data)
    > > print
    > > s.close()
    > > sock.close()

    >
    > > #client
    > > import socket
    > > import time

    >
    > > s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    > > s.bind(("", 2001))
    > > s.connect(("localhost", 2000))
    > > for i in range(2 ** 16):
    > > packet = "h" * i
    > > s.send("%16d" % len(packet))
    > > print "attempting to send data of length:", len(packet)
    > > tmp = s.send(packet)
    > > print "actually sent data of length:", tmp
    > > print
    > > time.sleep(.001)
    > > s.close()

    >
    > > i just put them in different files and ran them from the command line.
    > > Any help or suggestions would be greatly appreciated. Thanks.

    >
    > > -Walker

    >
    > > --
    > > This e-mail is licensed under the Creative Commons
    > > Attribution-NoDerivs 2.5 License. To view a copy of this license,
    > > visithttp://creativecommons.org/licenses/by-nd/2.5/or send a letter
    > > to Creative Commons, 543 Howard Street, 5th Floor, San Francisco,
    > > California, 94105, USA.- Hide quoted text -

    >
    > - Show quoted text -


    I'm not an expert on networking so take my advice with a grain of
    salt!

    My guess is that you're encountering race-like conditions between the
    send and recv. commmands because you're running your communication
    command 2^16 times, hoping that send and recv. are synchronous on
    every step. This is especially true when you're running other
    commands in your data sending loops! Prints take quite a lot of time.
    Also, the send command in C (i'm not sure if this is true in python)
    does not guarantee that all of your data gets out.

    I'm assuming that you're somewhat new to networking so I would
    recommend shying away from running an Asynchronous polling server.
    Although it is probably a better way to do this.

    My suggestions:

    Write a while loop in the server with some end of stream checking,
    timeouts, et cetera. This will give you a bit of flexibility and
    feedback when things go to pot. Use the sendall() command in your
    client because it throws an exception if data isn't properly sent. and
    will eliminate a unsightly for loop.

    Client

    socket.sendall(data)

    Something akin to this old piece of test code

    datastream = ''

    ## A timeout exception is thrown if the receiver is waiting on the
    line for .25 seconds
    socket.settimeout(.25)

    while 1:
    try:
    buf = socket.recv(1024) ## If data on line put in buffer
    except socket.timeout: ## Data buffer timeout, breaks
    while loop
    break
    if buf == '': ## If an empty buffer, jump to top of loop (beginning
    of file check)
    continue

    datastream = datastream + str(buf)

    if not len(buf) == 1024: ## If the buffer is not full break
    signaling end of file.
    break

    Hope this helps! And like i said i'm pretty new to this so be
    careful! Foundations of Network Programming by John Goerzen is a
    great book if you want some good information on the topic.

    Tyler
     
    Azazello, Aug 2, 2007
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. sreehari
    Replies:
    7
    Views:
    366
  2. Ashoka!
    Replies:
    6
    Views:
    572
    Ashoka!
    Apr 26, 2007
  3. Thomas Jollans
    Replies:
    0
    Views:
    386
    Thomas Jollans
    Aug 2, 2007
  4. Krzysztof Retel

    Network packets processor advice

    Krzysztof Retel, Feb 9, 2009, in forum: Python
    Replies:
    0
    Views:
    286
    Krzysztof Retel
    Feb 9, 2009
  5. Zsdfhdfgasdf Gsfgsdgsdgsd

    Sending packets over TCP server

    Zsdfhdfgasdf Gsfgsdgsdgsd, Jul 15, 2010, in forum: Ruby
    Replies:
    6
    Views:
    367
    Zsdfhdfgasdf Gsfgsdgsdgsd
    Jul 15, 2010
Loading...

Share This Page