Scatter/gather on sockets?

Discussion in 'Python' started by Roy Smith, Apr 1, 2006.

  1. Roy Smith

    Roy Smith Guest

    I've got a bunch of strings in a list:

    vector = []
    vector.append ("foo")
    vector.append ("bar")
    vector.append ("baz")

    I want to send all of them out a socket in a single send() call, so
    they end up in a single packet (assuming the MTU is large enough). I
    can do:

    mySocket.send ("".join (vector))

    but that involves creating an intermediate string. Is there a more
    efficient way, that doesn't involve that extra data copy?
     
    Roy Smith, Apr 1, 2006
    #1
    1. Advertising

  2. Roy Smith

    Peter Hansen Guest

    Roy Smith wrote:
    > I've got a bunch of strings in a list:
    >
    > vector = []
    > vector.append ("foo")
    > vector.append ("bar")
    > vector.append ("baz")
    >
    > I want to send all of them out a socket in a single send() call, so
    > they end up in a single packet (assuming the MTU is large enough). I
    > can do:
    >
    > mySocket.send ("".join (vector))
    >
    > but that involves creating an intermediate string. Is there a more
    > efficient way, that doesn't involve that extra data copy?


    Two possible answers that I can see:

    A. No, the send call is implemented in C and requires a single buffer
    with the entire piece of data that will be sent. Ultimately this gets
    passed down to the NIC hardware in some fashion, so there's certainly no
    hope of using something like a generator to send it in pieces.

    B. Don't bother trying, because even if the MTU is large enough there is
    absolutely no guarantee that the packet will stay intact all the way
    through the network anyway (even if you use sendall() instead of send()).

    So fixing your design not to require this appears to be the only viable
    solution.

    -Peter
     
    Peter Hansen, Apr 1, 2006
    #2
    1. Advertising

  3. On Sat, 01 Apr 2006 14:56:02 -0500, Roy Smith wrote:

    > I've got a bunch of strings in a list:
    >
    > vector = []
    > vector.append ("foo")
    > vector.append ("bar")
    > vector.append ("baz")
    >
    > I want to send all of them out a socket in a single send() call, so
    > they end up in a single packet (assuming the MTU is large enough). I
    > can do:
    >
    > mySocket.send ("".join (vector))
    >
    > but that involves creating an intermediate string. Is there a more
    > efficient way, that doesn't involve that extra data copy?


    Is sendall() what you're looking for?

    --
    A wise man knows he knows nothing.
     
    Anthony Greene, Apr 1, 2006
    #3
  4. Roy Smith

    Roy Smith Guest

    Peter Hansen <> wrote:
    > B. Don't bother trying, because even if the MTU is large enough there is
    > absolutely no guarantee that the packet will stay intact all the way
    > through the network anyway (even if you use sendall() instead of send()).


    This is true, but I'm generating the message being sent in very small
    chunks (often as small as 4 bytes at a time), and typically need to flush a
    packet out onto the network after a few dozen bytes. Maybe at most a few
    hundred. I don't know of any networks with MTU's smaller than that.
    Measurements show a 10-fold improvement in protocol throughput with large
    packets vs. small ones. The only question is what's the most efficient way
    in Python to generate the large packets.

    > So fixing your design not to require this appears to be the only viable
    > solution.


    My design is not broken. I'm writing code to drive a pre-existing binary
    communications protocol. It is what it is. The functionality I seek
    exists at the Unix system call level (writev, sendmsg), but doesn't appear
    to be exposed in the Python socket API.
     
    Roy Smith, Apr 1, 2006
    #4
  5. Roy Smith

    Roy Smith Guest

    In article <>,
    Anthony Greene <> wrote:

    > On Sat, 01 Apr 2006 14:56:02 -0500, Roy Smith wrote:
    >
    > > I've got a bunch of strings in a list:
    > >
    > > vector = []
    > > vector.append ("foo")
    > > vector.append ("bar")
    > > vector.append ("baz")
    > >
    > > I want to send all of them out a socket in a single send() call, so
    > > they end up in a single packet (assuming the MTU is large enough). I
    > > can do:
    > >
    > > mySocket.send ("".join (vector))
    > >
    > > but that involves creating an intermediate string. Is there a more
    > > efficient way, that doesn't involve that extra data copy?

    >
    > Is sendall() what you're looking for?


    No. Sendall() is actually what I'm using now. It handles the other side
    of the issue; issuing repeated send() calls if the system fragments your
    buffer. I'm trying to aggregate lots of small buffers into one large one.
     
    Roy Smith, Apr 1, 2006
    #5
  6. Roy Smith

    Paul Rubin Guest

    Roy Smith <> writes:
    > This is true, but I'm generating the message being sent in very small
    > chunks (often as small as 4 bytes at a time), and typically need to flush a
    > packet out onto the network after a few dozen bytes. Maybe at most a few
    > hundred. I don't know of any networks with MTU's smaller than that.
    > Measurements show a 10-fold improvement in protocol throughput with large
    > packets vs. small ones. The only question is what's the most efficient way
    > in Python to generate the large packets.


    Probably: build up the packet with cStringIO or with the array module
    instead of as a list of small strings. But if you time both versions
    I don't think it'll matter much. Python (at least CPython) simply
    will not be very fast no matter what you do. The overhead of building
    a large string (with ''.join, say) from a bunch of small ones isn't
    that big a deal compared with what you already lose in interpreter
    overhead running the application.
     
    Paul Rubin, Apr 1, 2006
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. rhmd
    Replies:
    9
    Views:
    708
    Pat Bills
    Oct 27, 2003
  2. Dr. Colombes
    Replies:
    10
    Views:
    21,324
    VParekh
    Dec 18, 2009
  3. Derek Basch

    Matplotlib logarithmic scatter plot

    Derek Basch, Feb 27, 2006, in forum: Python
    Replies:
    9
    Views:
    4,986
    evander21
    Nov 3, 2010
  4. Richard Maher
    Replies:
    16
    Views:
    840
    Chris Uppal
    Feb 20, 2007
  5. C Barrington-Leigh
    Replies:
    1
    Views:
    671
    C Barrington-Leigh
    Sep 12, 2010
Loading...

Share This Page