sockets: How to know when your data is sent

[Marc Ederis]

Hello,

I'm having a problem with sending data with sockets over a dial-up
connection. When I use the send function, it will happily send a
buffer of a megabyte and more in one shot. But of course, the data is
still in the network buffer... Meaning you can't disconnect for awhile
(but for how long...). The problem is, how can I know when it's done?
Is there a way to be notified when the data has truly been sent?

I tried using setsockopt and SO_SNDBUF to reduce the send buffer size,
thinking that the data would actually send in smaller increments,
instead of putting the whole thing in a buffer and taking it out of my
hands... No luck. It all works, mind you, but I'm left clueless as to
when the data is really off my computer!

I'm just following the example in the Python sockets How-To. Here's
the send function:

def mysend(msg):
totalsent = 0
while totalsent < MSGLEN:
sent = self.sock.send(msg[totalsent:])
if sent == 0:
raise RuntimeError, \\
"socket connection broken"
totalsent = totalsent + sent

Is there something I'm missing, or is there a better way to do this?

Thanks,
-Marc

 

[Grant Edwards]

I'm having a problem with sending data with sockets over a
dial-up connection. When I use the send function, it will
happily send a buffer of a megabyte and more in one shot. But
of course, the data is still in the network buffer... Meaning
you can't disconnect for awhile (but for how long...). The
problem is, how can I know when it's done? Is there a way to
be notified when the data has truly been sent?

What OS are you using?

Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. I actually wrote some kernel mode code
once upon a time to provide an ioctl() that I could call from
user space to find out when the data I had written to a socket
was actually sent.

Is there something I'm missing, or is there a better way to do
this?

The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.

 

[Sion Arrowsmith]

When I use the send function, it will
happily send a buffer of a megabyte and more in one shot. But
of course, the data is still in the network buffer...
[ ... ] The
problem is, how can I know when it's done?

Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. I actually wrote some kernel mode code
once upon a time to provide an ioctl() that I could call from
user space to find out when the data I had written to a socket
was actually sent.

I may well be missing the point of the problem here, but wouldn't
just turning TCP_NODELAY on guarantee that send() won't return
until the last ACK has returned? And if you're not using TCP:

The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.

applies in spades if you care about reliability.

 

[Grant Edwards]

When I use the send function, it will happily send a buffer of
a megabyte and more in one shot. But of course, the data is
still in the network buffer... [ ... ] The problem is, how
can I know when it's done?

Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. I actually wrote some kernel mode code
once upon a time to provide an ioctl() that I could call from
user space to find out when the data I had written to a socket
was actually sent.

I may well be missing the point of the problem here, but wouldn't
just turning TCP_NODELAY on guarantee that send() won't return
until the last ACK has returned?

Nope. Send returns as soon as the the data is copied to a a
kernel-space buffer. TCP_NODELAY has no effect on that. It
only changes the stack's behavior as it's taking data from that
buffer and putting it on the wire. Eventually, the TCP window
will fill up, and after than the tx buffer will fill up, and
after that, send will block. There's no practical way to
predict the size of either the TCP window or the tx buffer.
There's also no practical way to determine how much data is
still waiting to be acked (unless you want to do kernel
hacking).

And if you're not using TCP:


applies in spades if you care about reliability.

Unfortunately, many people who write network applications (and
protocols) make all sorts of bizarre assumptions, and somebody
ends up jumping through hoops later to try to fix things when
those assumptions turn out to be wrong. :/

 

[Marc Ederis]

When I use the send function, it will happily send a buffer of
a megabyte and more in one shot. But of course, the data is
still in the network buffer... [ ... ] The problem is, how
can I know when it's done?

Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. I actually wrote some kernel mode code
once upon a time to provide an ioctl() that I could call from
user space to find out when the data I had written to a socket
was actually sent.

I may well be missing the point of the problem here, but wouldn't
just turning TCP_NODELAY on guarantee that send() won't return
until the last ACK has returned?

Nope. Send returns as soon as the the data is copied to a a
kernel-space buffer. TCP_NODELAY has no effect on that. It
only changes the stack's behavior as it's taking data from that
buffer and putting it on the wire. Eventually, the TCP window
will fill up, and after than the tx buffer will fill up, and
after that, send will block. There's no practical way to
predict the size of either the TCP window or the tx buffer.
There's also no practical way to determine how much data is
still waiting to be acked (unless you want to do kernel
hacking).

And if you're not using TCP:


applies in spades if you care about reliability.

Unfortunately, many people who write network applications (and
protocols) make all sorts of bizarre assumptions, and somebody
ends up jumping through hoops later to try to fix things when
those assumptions turn out to be wrong. :/

I see, so the right way to do it is to have the server respond on a
successful receipt of the data. Thanks Grant. And thanks for
mentioning TCP_NODELAY, Sion, because I was about to try that too, but
this saves me the trouble.

-Marc

 

[Jaime Wyant]

Unfortunately, many people who write network applications (and
protocols) make all sorts of bizarre assumptions, and somebody
ends up jumping through hoops later to try to fix things when
those assumptions turn out to be wrong. :/

Woah! This is all enlightening news to me. Of course, what little
bit of socket level stuff I've done, I've always had a
"send/acknowledge" type protocol wrapped around. I guess I've just
been lucky so far .

jw

 

[Grant Edwards]

Woah! This is all enlightening news to me. Of course, what little
bit of socket level stuff I've done, I've always had a
"send/acknowledge" type protocol wrapped around. I guess I've just
been lucky so far .

While we're on the topic, another common problem occurs when
people assume that there is a 1:1 correspondance between
write() and read() operations when using TCP. IOW, they assume
that if they write() (or send()) a block of data of size N,
that it will arrive at the other end as a block and be read as
a block of size N. Under some conditions, this will often be
true. I ran into one app that worked under this assumption for
years, but broken when a satellite link was placed in the path.

 

[Richie Hindle]

[Marc]

When I use the send function, it will
happily send a buffer of a megabyte and more in one shot. But
of course, the data is still in the network buffer... Meaning
you can't disconnect for awhile (but for how long...). The
problem is, how can I know when it's done? Is there a way to
be notified when the data has truly been sent?
[Grant]
Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. [...]
The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.

If you want to ensure that all the data has been sent before *closing* the
socket, which sounds like Marc's requirement, you can use
setsockopt(SO_LINGER):

SO_LINGER
Sets or gets the SO_LINGER option. The argument is
a linger structure.

struct linger {
int l_onoff; /* linger active */
int l_linger; /* how many seconds to linger for */
};

When enabled, a close(2) or shutdown(2) will not
return until all queued messages for the socket
have been successfully sent or the linger timeout
has been reached. Otherwise, the call returns imme­
diately and the closing is done in the background.
When the socket is closed as part of exit(2), it
always lingers in the background.

(from the GNU manpages). I don't believe that's true for shutdown() on
all platforms, but I do believe it's true for close() / closesocket().
Calling it from Python is a bit of a chore involving the struct module,
and is left as an exercise for the reader.

 

[Gandalf]

Woah! This is all enlightening news to me. Of course, what little
bit of socket level stuff I've done, I've always had a
"send/acknowledge" type protocol wrapped around. I guess I've just
been lucky so far .

Oh yeah. I did the same and I realized that the response time can be
very long.
I had to reconsider my protocol, and use as few send/ack pairs as possible.
No matter if you have a broadband connection or not - it can be VERY slow
if your protocol uses many send/ack pairs.

 

[Grant Edwards]

Oh yeah. I did the same and I realized that the response time
can be very long.

For a badly designed protocol, it can.

I had to reconsider my protocol, and use as few send/ack pairs
as possible. No matter if you have a broadband connection or
not - it can be VERY slow if your protocol uses many send/ack
pairs.

The answer is to use a sliding window protocol.

 

[Ron Samuel Klatchko]

I'm having a problem with sending data with sockets over a dial-up
connection. When I use the send function, it will happily send a
buffer of a megabyte and more in one shot. But of course, the data is
still in the network buffer... Meaning you can't disconnect for awhile
(but for how long...). The problem is, how can I know when it's done?
Is there a way to be notified when the data has truly been sent?

You can get pretty close if you are willing to close the socket (but
since you want to disconnect, that should be fine).

What you need to do is turn on linger mode with a very large timeout (large
enough so that your slow connection has enough time to send the data).
Once linger is turned on, close() will block until all the data has
been sent.

The one issue that you can run into is if the linger time runs out before
all the data is sent then the rest of the data is discarded. But as long
as you set linger long enough (how about 10 years), you will run out of
patience long before the data is discarded.

samuel

 

[Marc Ederis]

If you want to ensure that all the data has been sent before *closing* the
socket, which sounds like Marc's requirement, you can use
setsockopt(SO_LINGER):

SO_LINGER
Sets or gets the SO_LINGER option. The argument is
a linger structure.

struct linger {
int l_onoff; /* linger active */
int l_linger; /* how many seconds to linger for */
};

When enabled, a close(2) or shutdown(2) will not
return until all queued messages for the socket
have been successfully sent or the linger timeout
has been reached. Otherwise, the call returns imme­
diately and the closing is done in the background.
When the socket is closed as part of exit(2), it
always lingers in the background.

(from the GNU manpages). I don't believe that's true for shutdown() on
all platforms, but I do believe it's true for close() / closesocket().
Calling it from Python is a bit of a chore involving the struct module,
and is left as an exercise for the reader.

SO_LINGER... Interesting, I'll keep that in mind for the future. In
the end, acknowledging receipt of the data, as it was arriving at the
server, turned out to be the best solution for me. I wanted to be able
to show a progress bar (at the client) of the data being sent.

Thanks guys,
-Marc

 

[Alex Martelli]

applies in spades if you care about reliability.

Oh yep:
http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf


Alex