[newbie] trying socket as a replacement for nc

[Grant Edwards]

Now, if you want reliability AND datagrams, it's a lot easier to add
boundaries to a TCP stream (sentinel or length prefixes) than to add
reliability to UDP...

It's unfortunate that there's no standardized reliable
connection-oriented datagram protocol. The linux kernel implements
one for Unix domain sockets (SOCK_SEQPACKET), and its really, really
useful.

Adding boundaries to a TCP stream achieves the same goal (and isn't
that hard to do), but since there's no standard for it, people keep
having to reinvent it (often badly and always incompaibly).

 

[Chris Angelico]

Adding boundaries to a TCP stream achieves the same goal (and isn't
that hard to do), but since there's no standard for it, people keep
having to reinvent it (often badly and always incompaibly).

Nearest to a standard would be the way heaps of internet protocols are
line-based - SMTP, POP, IMAP, FTP, and to a lesser extent HTTP as
well. The end-of-line sequence \r\n delimits messages.

ChrisA

 

[Grant Edwards]

Nearest to a standard would be the way heaps of internet protocols are
line-based - SMTP, POP, IMAP, FTP, and to a lesser extent HTTP as
well. The end-of-line sequence \r\n delimits messages.

And that works very nicely for things that transport text. It's easy
to implement, easy to debug, easy to test. But, when you need to
transport binary data, it gets ugly and compatibility problems start
to arise pretty quickly.

One could also borrow standards from the old-school serial world and
us the SYN/STX/ETX framing with byte stuffing used by HDLC et al.

 

[rusi]

Op vrijdag 13 december 2013 09:35:18 UTC+1 schreef Mark Lawrence:

Dear Mark,
I'm sorry for the inconvenience my postings may have caused. I now have
followed
the instructions on the link you mentioned and installed the plugin en
python-script.

Thanks for cooperating

hope it worked (I saw the text light up yellow when pressing the edit-key a second time). A small suggestion from a newbie: it would perhaps be possible
to make the script check itself whether pyhon2 or python3 should be used?

Yes... Half way
The double-spacing problem is cured
However the long-lines remain (see your "hope it worked..." above)
Did you click the edit button both before and after your typing?

The 'before' should remove the double-spaced (old >...) lines
The 'after' should even out the right margins of what you've just typed

thanks for having patience with me

Yes and you too please bear with us as we iron out this little irritant niggle

 

[Chris Angelico]

And that works very nicely for things that transport text. It's easy
to implement, easy to debug, easy to test. But, when you need to
transport binary data, it gets ugly and compatibility problems start
to arise pretty quickly.

One could also borrow standards from the old-school serial world and
us the SYN/STX/ETX framing with byte stuffing used by HDLC et al.

Yeah, or if it's a tight binary protocol with occasional bits of
bigger payload, either MIDI or TELNET could offer ideas. MIDI's SysEx
message can carry whatever is needed of it, and TELNET has
subnegotiation that can be used for the odd thingy or so. But for a
generic binary stream-of-messages pipe (as opposed to the
stream-of-bytes pipe that TCP normally offers), probably the easiest
is to precede each write with an N-byte length... and somehow
negotiate what N should be. (And endianness, but hopefully that's just
network byte order.)

Standards are awesome, there are so many to choose from! http://xkcd.com/927/

ChrisA

 

[rusi]

to make the script check itself whether pyhon2 or python3 should be used?

As far as I know both (2 and 3) worked
Do you have some reason to suspect one works and other not?

 

[Jean Dubois]

Op vrijdag 13 december 2013 18:09:50 UTC+1 schreef rusi:

As far as I know both (2 and 3) worked
Do you have some reason to suspect one works and other not?

The reason I suggested this is that the script has #!/usr/bin/env python3
as the first line. When you have python2 installed and not python3 a
newbie will
probably not understand why it doesn't work.

kind regards,
jean

 

[Jean Dubois]

Op vrijdag 13 december 2013 09:35:18 UTC+1 schreef Mark Lawrence:

Would you please read and action this
https://wiki.python.org/moin/GoogleGroupsPython to prevent us seeing the
double line spacing that accompanied the above, thanks.

I think I got it right this time. Thanks for helping me through it.

kind regards,
jean

 

[Mark Lawrence]

Op vrijdag 13 december 2013 09:35:18 UTC+1 schreef Mark Lawrence:
I think I got it right this time. Thanks for helping me through it.

kind regards,
jean

Yep, I've seen a couple of your posts that are just fine. Thanks for
taking these steps, I greatly appreciate your efforts.

 

[Jean Dubois]

Op vrijdag 13 december 2013 16:35:31 UTC+1 schreef Jean-Michel Pichavant:

----- Original Message -----
Such equipment often implements a telnet protocol. Have use try using the telnetlib module ?
http://docs.python.org/2/library/telnetlib.html
t = Telnet(host, port)
t.write('*IDN?')
print t.read_until('Whateverprompt')
# you can use read_very_eager also
JM

Thanks for the suggestion, I'll first wait for a response from Dan Stromberg concerning how to install his module, if he doesn't answer or if I'm still unsuccessfull then I'll try out your suggestion

kind regards,
jean

 

[Dan Stromberg]

Op vrijdag 13 december 2013 16:35:31 UTC+1 schreef Jean-Michel Pichavant:
Thanks for the suggestion, I'll first wait for a response from Dan Stromberg concerning how to install his module, if he doesn't answer or if I'm still unsuccessfull then I'll try out your suggestion

kind regards,
jean

You can "svn checkout <url>". You might try Sliksvn if you're on
Windows, or if you're on Linux it's in synaptic or yum or whatever.

You can "wget <url>".

You can bring up the URL in a web browser and cut and paste.

 

[Dan Stromberg]

Sockets reserve the right to split one socket.send() into multiple
socket.recv()'s on the other end of the communication, or to aggregate
multiple socket.send()'s into a single socket.recv() - pretty much any way
the relevant IP stacks and communications equipment feel like for the sake
of performance or reliability.

Just to be pedantic: _TCP_ sockets reserver that right. UDP sockets
do not, and do in fact guarantee that each message is discrete. [It
appears that the OP is undoubtedly using TCP sockets.]

I haven't done a lot of UDP, but are you pretty sure UDP can't at
least fragment large packets? What's a router or switch to do if the
Path MTU isn't large enough for an original packet?

http://www.gamedev.net/topic/343577-fragmented-udp-packets/

You're conflating IP datagrams and Ethernet packets. The IP stack can
fragment an IP datagram into multiple Ethernet packets which are then
reassembled by the receiving IP stack into a single datagram before
being passed up to the next layer (in this case, UDP).

As long as you're saying this of UDP, I have no problem with it.

I've seen TCP fragment and not be reassembled though, which suggests
to me that the reassembly's happening in UDP rather than IP. If it's
done by IP the same way for UDP and TCP, I'd not trust it in UDP
either.

Did you read the thread you pointed to? Your question was answerd by
posting #4 in the thread you cited:

1) Yes, packets will be fragmented at the network layer (IP), but this
is something you do not have to worry about since the network
layer will reassemble the fragments before passing them back up
to the transport layer (UDP). UDP garentees preserved message
boundaries, so you never have to worry about only receiving a
packet fragment :~).

Actually, I believe the link I sent (which I skimmed) had people
coming down on both sides of the matter. Some said that UDP would be
fine for small datagrams, while others said it would be fine,
irrespective of size.

A few other references:

http://tools.ietf.org/html/rfc791

1.1. Motivation

[...] The internet protocol provides for transmitting blocks of data
called datagrams from sources to destinations, [...] The internet
protocol also provides for fragmentation and reassembly of long
datagrams, if necessary, for transmission through "small packet"
networks.

I've personally seen this fail to occur in TCP - EG, it can cause a
stream of bytes to be written to tape with inconsistent block sizes if
transferred over rsh or ssh. Usually the block sizes are consistent,
but not always. Both SunOS 4.1.x and Ultrix had this issue; Ultrix
did it less often than SunOS, but Ultrix did do it. I don't know for
certain if later *ix have the same issue, because I've been diligently
working around it ever since, but I suspect they do.

I've seen old time socket programmers explain that it cannot be relied
upon in TCP; send() and recv() and (read() and write()) are system
calls that return a length so that you can loop on them until all
relevant data has been transferred. They don't return that length
just so you can ignore it.

From the Socket HOWTO

(http://docs.python.org/2/howto/sockets.html#socket-howto) :
Now we come to the major stumbling block of sockets - send and recv
operate on the network buffers. They do not necessarily handle all the
bytes you hand them (or expect from them), because their major focus
is handling the network buffers. In general, they return when the
associated network buffers have been filled (send) or emptied (recv).
They then tell you how many bytes they handled. It is your
responsibility to call them again until your message has been
completely dealt with.

HTH

 

[Grant Edwards]

Just to be pedantic: _TCP_ sockets reserve that right. UDP sockets
do not, and do in fact guarantee that each message is discrete. [It
appears that the OP is undoubtedly using TCP sockets.]

I haven't done a lot of UDP, but are you pretty sure UDP can't at
least fragment large packets? What's a router or switch to do if the
Path MTU isn't large enough for an original packet?

http://www.gamedev.net/topic/343577-fragmented-udp-packets/

You're conflating IP datagrams and Ethernet packets. The IP stack can
fragment an IP datagram into multiple Ethernet packets which are then
reassembled by the receiving IP stack into a single datagram before
being passed up to the next layer (in this case, UDP).

As long as you're saying this of UDP, I have no problem with it.

That is indeed what I'm saying. I apoligize if that was not clear in
my original posting.

I've seen TCP fragment and not be reassembled though, which suggests
to me that the reassembly's happening in UDP rather than IP.

That's something different. In TCP, there's no guarantee that
reads/writes correspond 1:1 to IP datagrams. TCP is a _stream_
protocol and there is no semantic meaning attached to the boundaries
between successive read/write calls the way there is with UDP.

If it's done by IP the same way for UDP and TCP,

The IP layer is supposed to reassemble receive datagrams for both --
but that's got nothing to do with atomicity of TCP writes/reads. The
TCP stack can (and often does) turn one write() call into multiple IP
datagrams. It can also turn multiple writes into a singel IP
datagram. On the other end, it can split up a single datagram into
multiple read()s and/or combined multiple datagrams into a single
read(). TCP is a stream service, not a datagram service like UDP.

I'd not trust it in UDP either.

The standards all require UDP datagrams to be preserved. All of the
UDP applications I've ever written or seen depend utterly on that, and
it's always worked that way for me. If you've seen it fail, then you
ought to file a bug report.

Actually, I believe the link I sent (which I skimmed) had people
coming down on both sides of the matter. Some said that UDP would be
fine for small datagrams, while others said it would be fine,
irrespective of size.

The maximum size of an IP datagram is 64KB, so it's not "fine
irrespecive of size". If your UDP implementation is working correctly
it will be fine below that limit.

A few other references:

http://tools.ietf.org/html/rfc791

1.1. Motivation

[...] The internet protocol provides for transmitting blocks of data
called datagrams from sources to destinations, [...] The internet
protocol also provides for fragmentation and reassembly of long
datagrams, if necessary, for transmission through "small packet"
networks.

I've personally seen this fail to occur in TCP

You can't say that, because there's no correspondance between IP
datgrams and TCP read/write block sizes the way there is in UDP.

With TCP there is nothing to fail (with respect to read/write block
sizes). TCP only guarantees that bytes will get there and get there in
the right order. It doesn't make any promises about block sizes.

I've seen old time socket programmers explain that it cannot be relied
upon in TCP; send() and recv() and (read() and write()) are system
calls that return a length so that you can loop on them until all
relevant data has been transferred. They don't return that length
just so you can ignore it.

That's true, but that's because of the design of the TCP _stream_
protocol, not because the IP datagram layer doesn't work right.

(http://docs.python.org/2/howto/sockets.html#socket-howto) : Now we
come to the major stumbling block of sockets - send and recv operate
on the network buffers. They do not necessarily handle all the bytes
you hand them (or expect from them), because their major focus is
handling the network buffers. In general, they return when the
associated network buffers have been filled (send) or emptied (recv).
They then tell you how many bytes they handled. It is your
responsibility to call them again until your message has been
completely dealt with.

If that's true for UDP, then the Python UDP implementation is broken,
and somebody should file a bug. UDP is a a _datagram_ service.
Either all the bytes in a write() should get sent or none of them.
Sending a paritial datagram is _not_ a valid option.

 

[Jean Dubois]

Op zondag 15 december 2013 02:03:14 UTC+1 schreef Dan Stromberg:

You can "svn checkout <url>". You might try Sliksvn if you're on
Windows, or if you're on Linux it's in synaptic or yum or whatever.
You can "wget <url>".
You can bring up the URL in a web browser and cut and paste.

I'm using Linux, I did the following:
svn checkout http://stromberg.dnsalias.org/svn/bufsock/
which resulted in a directory 'bufsock' being added to my home-directory,
Do I have to run further commands on the files in this directory?
How do I make Python aware of the existence of this new module?

thanks in advance
jean

 

[Chris Angelico]

I'm using Linux, I did the following:
svn checkout http://stromberg.dnsalias.org/svn/bufsock/
which resulted in a directory 'bufsock' being added to my home-directory,
Do I have to run further commands on the files in this directory?
How do I make Python aware of the existence of this new module?

Have a look in that directory. It seems to simply have a bufsock.py
which is the module to import.

That said, though, you may have to deal with dependencies. The source
code of bufsock.py references a python2x3 module (which Google tells
me is by the same author), so you may need to grab that, too.

ChrisA

 

[Roy Smith]

UDP is a a _datagram_ service. Either all the bytes in a write()
should get sent or none of them. Sending a paritial datagram is _not_
a valid option.

I would agree with the above if you said send() instead of write().
Python socket objects don't have write() methods, file objects do. You
can wrap a file around a socket with socket.makefile(), but I'm not sure
I would expect the UDP record boundary semantics to be honored once you
did that.

 

[Chris Angelico]

I would agree with the above if you said send() instead of write().
Python socket objects don't have write() methods, file objects do. You
can wrap a file around a socket with socket.makefile(), but I'm not sure
I would expect the UDP record boundary semantics to be honored once you
did that.

The underlying C API allows you, on Unix-like systems at least, to use
the standard write() function to send UDP packets (as long as you
first connect() - otherwise you need sendto() to specify a
destination). I don't usually use that method, but I would expect that
one call to write() becomes one UDP packet.

How that works with socket.makefile() in Python I have no idea.

ChrisA

 

[Roy Smith]

The underlying C API allows you, on Unix-like systems at least, to use
the standard write() function to send UDP packets (as long as you
first connect() - otherwise you need sendto() to specify a
destination). I don't usually use that method, but I would expect that
one call to write() becomes one UDP packet.

At the Unix system call level, yes. But, given that this is a Python
newsgroup, I made the assumption we were talking about the Python API
level. Silly me

 

[Dan Stromberg]

I'm using Linux, I did the following:
svn checkout http://stromberg.dnsalias.org/svn/bufsock/
which resulted in a directory 'bufsock' being added to my home-directory,
Do I have to run further commands on the files in this directory?
How do I make Python aware of the existence of this new module?

You can put the files (bufsock.py and python2x3.py) in your current
working directory - Python will import from your CWD. I believe
python2x3.py should be checked out via an external reference since you
used svn.

You can put the files in your site-packages directory.

You can put the files in a directory like ~/lib, and then
sys.path.insert(0, os.path.expanduser('~/lib')) .

I probably should make it pip'able, but I don't think it's going to
happen today.

 

[Grant Edwards]

I would agree with the above if you said send() instead of write().

Good point -- I meant send(). I keep forgetting that the libc socket
write() operation is missing in Python and only the send() call has
been made visible. In C write() and send() are effectively the same
thing (the parameters are arranged a little differently, but they
behave identically otherwise).

Python socket objects don't have write() methods, file objects do. You
can wrap a file around a socket with socket.makefile(), but I'm not sure
I would expect the UDP record boundary semantics to be honored once you
did that.

No, I wouldn't exect that.