asyncore and OOB data

[billiejoex]

Hi there,
In an asyncore based FTP server I wrote I should be able to receive
OOB data from clients.
A client sending such kind of data should act like this:

According to asyncore documentation I should handle this kind of event
in "handle_expt" method of dispatcher class, that should be called
when OOB is received by the underlying socket. I've tried to override
handle_expt method in such way:

def handle_expt(self):
print "OOB data arrived"
data = self.socket.recv(1024, socket.MSG_OOB)
print data

....but, even if it is called, "data" contains only a "\n" character
instead of the entire string ("hello there\r\n").
Why does this happen? Should I have to use a different approach?

Thanks in advance.

 

[billiejoex]

If it could be useful I attach the complete server application I used
for doing tests:


< code >

import asyncore, asynchat, socket, os

class Handler(asynchat.async_chat):
def __init__(self, sock_obj):
asynchat.async_chat.__init__(self, conn=sock_obj)
self.remote_ip, self.remote_port = self.socket.getpeername()
self.in_buffer = ""
self.out_buffer = ""
self.set_terminator("\r\n")

def handle_expt(self):
print "OOB data arrived"
data = self.socket.recv(1024, socket.MSG_OOB)
print tuple(data)

def collect_incoming_data(self, data):
self.in_buffer += data

def found_terminator(self):
print self.in_buffer
self.in_buffer = ''

def handle_error(self):
raise


class Server(asyncore.dispatcher):

def __init__(self):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
if os.name == 'posix':
self.set_reuse_addr()
self.bind(('', 21))
self.listen(5)

def handle_accept(self):
sock_obj, addr = self.accept()
Handler(sock_obj)

def handle_error(self):
raise


Server()
asyncore.loop(timeout=1)

 

[Douglas Wells]

In an asyncore based FTP server I wrote I should be able to receive
OOB data from clients.
A client sending such kind of data should act like this:


According to asyncore documentation I should handle this kind of event
in "handle_expt" method of dispatcher class, that should be called
when OOB is received by the underlying socket. I've tried to override
handle_expt method in such way:

def handle_expt(self):
print "OOB data arrived"
data = self.socket.recv(1024, socket.MSG_OOB)
print data

...but, even if it is called, "data" contains only a "\n" character
instead of the entire string ("hello there\r\n").
Why does this happen? Should I have to use a different approach?

You should use a different approach, but first you need to read
the relevant documentation.

I've never used asyncore, but when I look at the documentation,
the only reference that I find to OOB says:
"Called when there is out of band (OOB) data for a socket
connection. This will almost never happen, as OOB is
tenuously supported and rarely used."
That should be fair warning.

Second, when I look at the FTP specification, I don't find the
concept of OOB anywhere. So, it's not clear what OOB data would
mean in terms of the defined FTP commands in any case.

Third, the TCP protocol, which you have selected via the SOCK_STREAM
option doesn't support OOB at all, so there's no way that you can
even send OOB data in the manner that you are expecting.

Let's switch to the positive, however. What TCP does have is the
concept of "urgent" data. Urgent data is data that the sender
believes is so important that the receiver should discard intermediate
data, if necessary, in order to process the urgent data -- but the
urgent data is sent *in-band*.

In the case of FTP, the concept of urgent data is used to allow
the user side to interrupt the data transfer, which it does by
sending an ABOR command. The underlying problem is that the server
might be blocked on a receive on the data connection and not
listening for commands on the control connection. So the user
side is required to perform some "special action" that includes
sending urgent data, which the server can then process specially.
This problem arises primarily in systems without either threading
or asynchronous I/O APIs. Your use of asyncore should alleviate
this problem.

The thing that may be confusing you is that for whatever reason
the designers of the Berkeley socket API (which is what you are
using when running under POSIX/Linux/MS Windows), chose to use the
OOB flags (e.g., MSG_OOB) to indicate the *transmission* of OOB
data -- but it's not really OOB data. In fact, when I attempt to
receive using the MSG_OOB flag using the OS interfaces on my system,
I get back an error (EINVAL - Invalid argument). I would expect
that asyncore would report this error somehow or other.

In summary, you almost certainly can't use the concept of a separate
OOB channel to transfer data in an FTP environment. If you really,
really need something like this, you will need to extend the FTP
protocol for your special purposes.

- dmw

 

[Steve Holden]

You should use a different approach, but first you need to read
the relevant documentation.

I've never used asyncore, but when I look at the documentation,
the only reference that I find to OOB says:
"Called when there is out of band (OOB) data for a socket
connection. This will almost never happen, as OOB is
tenuously supported and rarely used."
That should be fair warning.

Second, when I look at the FTP specification, I don't find the
concept of OOB anywhere. So, it's not clear what OOB data would
mean in terms of the defined FTP commands in any case.

Third, the TCP protocol, which you have selected via the SOCK_STREAM
option doesn't support OOB at all, so there's no way that you can
even send OOB data in the manner that you are expecting.

Wrong. The URGENT pointer indicates that the data begins with
out-of-band data. You are correct, however, in stating that the FTP
protocol doesn't support or implement out-of-band data transmissions.

Let's switch to the positive, however. What TCP does have is the
concept of "urgent" data. Urgent data is data that the sender
believes is so important that the receiver should discard intermediate
data, if necessary, in order to process the urgent data -- but the
urgent data is sent *in-band*.

No, technically the urgent data is out-of-band, isn't it? There is no
requirement to ditch the other data in the packet, so urgent data can be
interspersed without loss of regular in-band data if required.

In the case of FTP, the concept of urgent data is used to allow
the user side to interrupt the data transfer, which it does by
sending an ABOR command. The underlying problem is that the server
might be blocked on a receive on the data connection and not
listening for commands on the control connection. So the user
side is required to perform some "special action" that includes
sending urgent data, which the server can then process specially.
This problem arises primarily in systems without either threading
or asynchronous I/O APIs. Your use of asyncore should alleviate
this problem.

The data and control connections are typically dealt with in most
implementations by separate processes, so an ABOR command can be sent
without having to wait even if the data connection is blocked. Of course
if a specific implementation uses a single-threaded single process to
drive both the data and control connections then all bets are off.

The thing that may be confusing you is that for whatever reason
the designers of the Berkeley socket API (which is what you are
using when running under POSIX/Linux/MS Windows), chose to use the
OOB flags (e.g., MSG_OOB) to indicate the *transmission* of OOB
data -- but it's not really OOB data. In fact, when I attempt to
receive using the MSG_OOB flag using the OS interfaces on my system,
I get back an error (EINVAL - Invalid argument). I would expect
that asyncore would report this error somehow or other.

You can, of course, expect what you like, but experience is the best
guide to what will actually happen.

In summary, you almost certainly can't use the concept of a separate
OOB channel to transfer data in an FTP environment. If you really,
really need something like this, you will need to extend the FTP
protocol for your special purposes.

I'd certainly concur with that: anything that's processing FTP streams
will almost certainly barf when urgent data arrives, since this is
outside the scope of the FTP protocol specification.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

 

[billiejoex]

Second, when I look at the FTP specification, I don't find the
concept of OOB anywhere. So, it's not clear what OOB data would
mean in terms of the defined FTP commands in any case.

You are correct, however, in stating that the FTP
protocol doesn't support or implement out-of-band
data transmissions.

Wait a sec. RFC959 [1], on chapter 4.1.3 says:

Certain commands (such as ABOR, STAT, QUIT) may be sent over the
control connection while a data transfer is in progress. Some
servers may not be able to monitor the control and data connections
simultaneously, in which case some special action will be necessary
to get the server's attention. The following ordered format is
tentatively recommended:

1. User system inserts the Telnet "Interrupt Process" (IP) signal
in the Telnet stream.
2. User system sends the Telnet "Synch" signal.
3. User system inserts the command (e.g., ABOR) in the Telnet
stream.
4. Server PI, after receiving "IP", scans the Telnet stream for
EXACTLY ONE FTP command.

I believe that the TCP "urgent" flag, activated by using
socket.MSG_OOB, should be set when client must send the "Sync" signal
(RFC854 [2] talks about it). I think that you do not find references
of OOB in RFC959 (FTP) just because it is specified into RFC854
(Telnet).
According to RFC854 a Sync signal should consist of sending Telnet
DATA MARK (DM) inside a TCP packet with URGent flag set:

The Synch is sent via the TCP send operation with the Urgent
flag set and the DM as the last (or only) data octet.

If we'd want to traduce this in Python we could assume that a fully
RFC-compliant FTP client implementation sending 'ABOR' command should
act like this:

We also could find references about such flag ("URG"ent, alias OOB)
even in Bernstein FTP-related paper (http://cr.yp.to/ftp/
pipelining.html) which says:

The client is required to send the TELNET string \377\364, and
then send the TELNET string \377\362 with the TCP Urgent bit
set, immediately before the ABOR request.

....that, traduced, should appear like this:


I also find interesting the fact that ftplib ABOR command
implementation is not RFC-compliant (and I can't realize how my server
could recognize it):

def abort(self):
'''Abort a file transfer. Uses out-of-band data.
This does not follow the procedure from the RFC to send Telnet
IP and Synch; that doesn't seem to work with the servers I've
tried. Instead, just send the ABOR command as OOB data.'''
line = 'ABOR' + CRLF
if self.debugging > 1: print '*put urgent*',
self.sanitize(line)
self.sock.sendall(line, MSG_OOB)
resp = self.getmultiline()
if resp[:3] not in ('426', '226'):
raise error_proto, resp


[1] http://www.faqs.org/rfcs/rfc959.html
[2] http://www.faqs.org/rfcs/rfc854.html

 

[Douglas Wells]

Wrong. The URGENT pointer indicates that the data begins with
out-of-band data. You are correct, however, in stating that the FTP
protocol doesn't support or implement out-of-band data transmissions.

Well, first I have to admit that I just looked at the wiki article
on out-of-band (<http://en.wikipedia.org/wiki/Out-of-band>) and
it also makes a statement that TCP urgent data is OOB. But I
disagree with that interpretation. So, let me acknowledge that
there are dissenting opinions, and explain my assertion in terms
of the OP's problem of wanting a separate data stream (or at least
that was my interpretation).

First, I fully acknowledge that the concept of TCP's URGENT pointer
itself is out-of-band, but I don't know of any way to associate
data, in the form of an identifiable set of octets, with that
concept. As supporting evidence I offer the following:

- There is no way for the sender and the receiver to agree on
the *start* of an urgent data set. The end point is clearly
identified by the urgent pointer, but the start is time dependent
and is not identified in the TCP communication stream. Thus, the
sender could not express the concept of sending a particular
data value, e.g. "This string", to the receiver reliably.

- If receivers are highly responsive (meaning that the receiver
is always able to process incoming data without having to
discard any of it), there will be no difference in the behavior
of two receivers, one of which properly implements the concept
of the urgent pointer, and the other of which totally ignores
the concept of urgency.

- There appears to be no way to actually read any OOB data from
a Berkeley socket API. (As I noted my system, at least, returns
an error in this case.)

No, technically the urgent data is out-of-band, isn't it? There is no
requirement to ditch the other data in the packet, so urgent data can be
interspersed without loss of regular in-band data if required.

This is the same point that I argued above. I now acknowledge that
some people think of the TCP urgent concept as OOB. I don't; but
this is a matter of definition and almost certainly not an issue
for coherent argument.

You can, of course, expect what you like, but experience is the best
guide to what will actually happen.

Agreed, but having acknowledged that I had never used asyncore,
I was trying to express the hope that the designers would not
purposely discard a error condition without notifying the user,
particularly for a "reliable" transfer method such as TCP.

Steve

- dmw

 

[Douglas Wells]

Second, when I look at the FTP specification, I don't find the
concept of OOB anywhere. So, it's not clear what OOB data would
mean in terms of the defined FTP commands in any case.

You are correct, however, in stating that the FTP
protocol doesn't support or implement out-of-band
data transmissions.

Wait a sec. RFC959 [1], on chapter 4.1.3 says:

[ ... ]

I believe that the TCP "urgent" flag, activated by using
socket.MSG_OOB, should be set when client must send the "Sync" signal
(RFC854 [2] talks about it). I think that you do not find references
of OOB in RFC959 (FTP) just because it is specified into RFC854
(Telnet).

The point that I was trying to make, which Steve made more clearly
that I did in the referenced paragraph, is that FTP doesn't support
OOB data transmission, not that RFP doesn't require the use of OOB
information (in the form of the Telnet Synch signal).

Note that FTP limits the commands that may be issues while a data
transfer in progress. The RFC isn't totally explicit, but the
general notion is that issuing a data transfer command (such as
STOR or RETR) is not allowed while another data transfer is in
progress.

- dmw