Learning python networking

[Paul Pittlerson]

I'm trying to learn about socket, how to create and handle connections in python.

This is the best I could come up with on my own, please take a look and give me critique:
Server script:
http://pastebin.com/KtapYfM0

Client script:
http://pastebin.com/t4dYygmX

How to run it:
I open 3 terminals, in one I start the server script, and enter into it something like 'accept 2'

Then in the two other terminals I start client scripts which will "connect to the server" so to speak.

Now I can communicate between them by sending messages, and exit the whole operation by typing 'exit' into the server.

Is the code overly complicated? More precisely: is there a more elegant andsimple way to achieve the same thing? Some people have mentioned things like twisted and asyncore, but I don't know anything about them. If it turns out this kind of concept is very straight forward to set up in either of those I would be interested in sample code.

I'm specifically looking into this kind of communication because I want to make a small multiplayer game.

 

[Chris Angelico]

I'm trying to learn about socket, how to create and handle connections inpython.

Awesome! I *love* socket networking. (Really. It's awesome. I've
written a couple of MUD servers and quite a few MUD clients.)

This is the best I could come up with on my own, please take a look and give me critique:
Server script:
http://pastebin.com/KtapYfM0

Client script:
http://pastebin.com/t4dYygmX

On this list we prefer in-line code as part of the post, but these are
a little long. Posting like this creates a dependency on that web
site, so a lot of people either can't or won't see your code.

How to run it:
I open 3 terminals, in one I start the server script, and enter into it something like 'accept 2'

Then in the two other terminals I start client scripts which will "connect to the server" so to speak.

Now I can communicate between them by sending messages, and exit the whole operation by typing 'exit' into the server.

From what I'm seeing in that code, all communication is one-way,

right? The server sends to the clients, nothing comes back?

Is the code overly complicated? More precisely: is there a more elegant and simple way to achieve the same thing? Some people have mentioned things like twisted and asyncore, but I don't know anything about them. If it turns out this kind of concept is very straight forward to set up in either of those I would be interested in sample code.

Those sorts of frameworks would be helpful if you need to scale to
infinity, but threads work fine when it's small.

I'm specifically looking into this kind of communication because I want to make a small multiplayer game.

Absolutely! The thing to look at is MUDs and chat servers. Ultimately,
a multiplayer game is really just a chat room with a really fancy
front end.

So, some comments on your code.

The server shouldn't require interaction at all. It should accept any
number of clients (rather than getting the exact number that you
enter), and drop them off the list when they're not there. That's a
bit of extra effort but it's hugely beneficial.

One extremely critical point about your protocol. TCP is a stream -
you don't have message boundaries. You can't depend on one send()
becoming one recv() at the other end. It might happen to work when you
do one thing at a time on localhost, but it won't be reliable on the
internet or when there's more traffic. So you'll need to delimit
messages; I recommend you use one of two classic ways: either prefix
it with a length (so you know how many more bytes to receive), or
terminate it with a newline (which depends on there not being a
newline in the text).

Another rather important point, in two halves. You're writing this for
Python 2, and you're writing with no Unicode handling. I strongly
recommend that you switch to Python 3 and support full Unicode. Your
current code might work fine so long as everyone uses the same
codepage, but then you'll meet someone from Greece or Russia and
they'll be typing in gibberish because you're decoding it wrongly.
(You can get that sort of thing even without crossing country borders,
but then it's tempting to just say "Don't use funny characters"; but
the problem isn't funny characters, of which there aren't any (I
swear, Spike Milligan is *so* not funny... okay, that's not true), but
of encodings and character sets.) Using Python 3.4 (which isn't yet
stable, but you can download betas) also gives you an asyncio module,
but I'd leave that aside for the moment; first figure out threading,
it's likely to be easier.

So here's how I'd structure a program like this.

# Server
bind socket to port and listen on it
while True:
accept socket
spawn thread

thread:
register self with list of connected clients
while socket connected:
receive data
try to parse a message out of data - if not, buffer it
handle message, eg by sending to all connected clients
unregister self from connected client list


# Client
connect to server
start thread
while True:
accept input from user
act on input, which might 'break'
shut down socket cleanly

thread:
receive data from socket
try to parse a message out of data, as with the server
# you might even be able to use the same code for both
handle message, eg by displaying to screen



That's a threaded system. It emphasizes that the server is doing one
"thing" for each connected client (listening for incoming socket
data), plus one more "thing" (listening for new clients), and the
client is doing two "things" (listening for commands from the human,
and listening for messages from the server). Moving to an async I/O
system doesn't change that, it just runs it all on one thread. So do
whichever's easier to get your head around

Note, by the way, that it's helpful to distinguish "data" and "text",
even in pseudo-code. It's impossible to send text across a socket -
you have to send bytes of data. If you keep this distinction clearly
in your head, you'll have no problem knowing when to encode and when
to decode. For what you're doing here, for instance, I would packetize
the bytes and then decode into text, and on sending, I'd encode text
(UTF-8 would be hands-down best here) and then packetize. There are
other options but that's how I'd do it.

This is the Flight Level 200 overview of socket handling. I'm happy to
expand on any part that interests or confuses you (especially if it
does both at once).

The world is so much fun when you can wield multiple computers!

ChrisA

 

[Dan Stromberg]

Nice response Chris. Seriously.

One extremely critical point about your protocol. TCP is a stream -
you don't have message boundaries. You can't depend on one send()
becoming one recv() at the other end. It might happen to work when you
do one thing at a time on localhost, but it won't be reliable on the
internet or when there's more traffic. So you'll need to delimit
messages; I recommend you use one of two classic ways: either prefix
it with a length (so you know how many more bytes to receive), or
terminate it with a newline (which depends on there not being a
newline in the text).

Completely agree, and I'll point out
http://stromberg.dnsalias.org/~dstromberg/bufsock.html , which makes
it a little easier to deal with delimiters like newlines or null
termination.

Another rather important point, in two halves. You're writing this for
Python 2, and you're writing with no Unicode handling. I strongly
recommend that you switch to Python 3 and support full Unicode.

Agreed. It's time modernize. Don't leave out people on the other
side of the world for no reason other than a modicum of convenience
for the developer.

Using Python 3.4 (which isn't yet
stable, but you can download betas) also gives you an asyncio module,
but I'd leave that aside for the moment; first figure out threading,
it's likely to be easier.

Personally, I don't like asynchronous I/O, EG twisted. It tends to
give very complex, uniprocessor solutions to problems, even when those
problems have quite simple, alternative solutions. I'd rather just
accept and fork in most cases, with or without multiprocessing.

 

[Chris Angelico]

Personally, I don't like asynchronous I/O, EG twisted. It tends to
give very complex, uniprocessor solutions to problems, even when those
problems have quite simple, alternative solutions. I'd rather just
accept and fork in most cases, with or without multiprocessing.

I haven't used the Python asyncio module yet (haven't gotten around to
it, no other reason), but in my Pike work, I've gotten quite friendly
with a callback system. It plays nicely with the structure I have for
code reloading, by separating code and data, and then simply running
everything off a single thread, one callback at a time. Works really
well when everything's waiting anyway. It does force you to think
about things differently, though. Case in point: My MUD client allows
an arbitrary number of code files to inspect a line of text that the
user's entered, before it goes to the server. Any one of them can
suppress the line, in which case subsequent hooks won't see it and it
won't go to the server. What about changing the line as it goes
through? That's easy enough, I guess. But what if that change involves
a popup message and user response... or a network request to a remote
server? (I do, in fact, have exactly that. Long URLs get shortened via
tinyurl.com.) A threaded model wouldn't even help much, here, unless
I'm prepared to spawn a separate thread for every command the user
enters, which is stupid overhead. Instead, I have a function that
reinjects the line and carries on as if it hadn't been suppressed in
the first place.

Maybe it's not the best way to do things, but it can be extremely
simple in the code. Most operations don't even need the
continuation-call mechanism that I used there; mostly, everything's
just (conceptually) "when this happens, call this function". The fact
that they're serialized doesn't matter. Events from the GUI, incoming
socket data (on any socket - there might be multiple), time-delay
events (eg a ticking clock), etc - everything works the same way. It's
really easy once you get your head around it.

ChrisA

 

[Dan Stromberg]

Maybe it's not the best way to do things, but it can be extremely
simple in the code.

For small projects, the added complexity doesn't bite you. At least, not much.

For large projects, with thousands or millions of callbacks, it can be
very difficult to track down bugs in who-knows-which callback, given
that they're all being called asynchronously. It's vastly simpler to
fire up a debugger against a process, or insert print statements that
are related to each other in time in some way.

It's
really easy once you get your head around it.

IMO, it's not a matter of wrapping your head around it, but respecting
complexity.

Three quotes come to mind:
* Make things as simple as possible, but not simpler.
* Fools ignore complexity. Pragmatists suffer it. Some can avoid it.
Geniuses remove it.
* Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it.

I'm not trying to call you or anyone else a fool, but the Alan Perlis
quote (the second) does give a flavor of what I'm trying to say.

The third quote, from Brian Kernighan, seems to underestimate the
complexity of asynchronous programming in the large - it's probably
not just twice as hard.

 

[Chris Angelico]

The third quote, from Brian Kernighan, seems to underestimate the
complexity of asynchronous programming in the large - it's probably
not just twice as hard.

Yeah, which is why I recommended a threaded approach to the OP. It
won't scale to millions of simultaneous connections... but he's
unlikely to have more than half a dozen in testing, and even in
production, most games aren't going to run more than a couple hundred
players. (A huge server might have more players than that
simultaneously logged in, but they'll be running different games. If
thread count becomes a problem, fork() to create a game, job done.)
Threads are easier to get your head around: this and that happen at
the same time. Sometimes it means creating lots of threads to do
similar things (one for the GUI, one for this socket, one for that
socket, one for the cron handler, etc, etc), but is that really a
problem? Probably not.

ChrisA

 

[Paul Pittlerson]

I'm sorry if this is a bit late of a response, but here goes.

Big thanks to Chris Angelico for his comprehensive reply, and yes, I do have some questions!

On Thursday, January 9, 2014 1:29:03 AM UTC+2, Chris Angelico wrote:
Those sorts of frameworks would be helpful if you need to scale to
infinity, but threads work fine when it's small.

That's what I thought, but I was just asking if it would be like trivially easy to set up this stuff in some of those frameworks. I'm sticking to threads for now because the learning curve of twisted seems too steep to be worth it at the moment.

Absolutely! The thing to look at is MUDs and chat servers. Ultimately,
a multiplayer game is really just a chat room with a really fancy
front end.

If you know of any open source projects or just instructional code of this nature in general I'll be interested to take a look. For example, you mentioned you had some similar projects of your own..?

The server shouldn't require interaction at all. It should accept any
number of clients (rather than getting the exact number that you
enter), and drop them off the list when they're not there. That's a
bit of extra effort but it's hugely beneficial.

I get what you are saying, but I should mention that I'm just making a 2 player strategy game at this point, which makes sense of the limited number of connections.

One extremely critical point about your protocol. TCP is a stream -
you don't have message boundaries. You can't depend on one send()
becoming one recv() at the other end. It might happen to work when you
do one thing at a time on localhost, but it won't be reliable on the
internet or when there's more traffic. So you'll need to delimit
messages; I recommend you use one of two classic ways: either prefix
it with a length (so you know how many more bytes to receive), or
terminate it with a newline (which depends on there not being a
newline in the text).

I don't understand. Can you show some examples of how to do this?

Another rather important point, in two halves. You're writing this for
Python 2, and you're writing with no Unicode handling. I strongly
recommend that you switch to Python 3 and support full Unicode.

Good point, however the framework I'm using for graphics does not currently support python3. I could make the server scripts be in python3, but I don't think the potential confusion is worth it until the whole thing can be in the same version.

Note, by the way, that it's helpful to distinguish "data" and "text",
even in pseudo-code. It's impossible to send text across a socket -
you have to send bytes of data. If you keep this distinction clearly
in your head, you'll have no problem knowing when to encode and when
to decode. For what you're doing here, for instance, I would packetize
the bytes and then decode into text, and on sending, I'd encode text
(UTF-8 would be hands-down best here) and then packetize. There are
other options but that's how I'd do it.

I'm not sure what you are talking about here. Would you care to elaborate on this please (it interests and confuses) ?


I'm posting this on google groups, so I hope the formatting turns out ok thanks.

 

[Denis McMahon]

I don't understand. Can you show some examples of how to do this?

How much do you understand about tcp/ip networking? because when trying
to build something on top of tcp/ip, it's a good idea to understand the
basics of tcp/ip first.

A tcp/ip connection is just a pipe that you pour data (octets, more or
less analagous to bytes or characters) into at one end, and it comes out
at the other end.

For your stream of octets (bytes / characters) to have any meaning to a
higher level program, then the applications using the pipe at both ends
have to understand that a message has some structure.

The message structure might be to send an n character message length
count (where in a simple protocol n would have to be a fixed number)
followed by the specified number of characters.

Assuming your maximum message length is 9999 characters:

You could send the characters 9999 followed by 9999 characters of message
content.

The receiving end would receive 4 characters, convert them to the number
9999, and assume the next 9999 characters will be the message. Then it
expects another 4 character number.

You could send json encoded strings, in which case each message might
start with the "{" character and end with the "}" character, but you
would have to allow for the fact that "}" can also occur within a json
encoded string.

You might decide that each message is simply going to end with a specific
character or character sequence.

Whatever you choose, you need some way for the receiving application to
distinguish between individual messages in the stream of octets / bytes /
characters that is coming out of the pipe.

 

[Chris Angelico]

I'm sorry if this is a bit late of a response, but here goes.

Big thanks to Chris Angelico for his comprehensive reply, and yes, I do have some questions!

Best way to learn! And the thread's not even a week old, this isn't
late. Sometimes there've been responses posted to something from
2002... now THAT is thread necromancy!!

That's what I thought, but I was just asking if it would be like trivially easy to set up this stuff in some of those frameworks. I'm sticking to threads for now because the learning curve of twisted seems too steep to be worth it at the moment.

I really don't know, never used the frameworks. But threads are fairly
easy to get your head around. Let's stick with them.

If you know of any open source projects or just instructional code of this nature in general I'll be interested to take a look. For example, you mentioned you had some similar projects of your own..?

Here's something that I did up as a MUD-writing tutorial for Pike:

http://rosuav.com/piketut.zip

I may need to port that tutorial to Python at some point. In any case,
it walks you through the basics. (Up to section 4, everything's the
same, just different syntax for the different languages. Section 5 is
Pike-specific.)

I get what you are saying, but I should mention that I'm just making a 2 player strategy game at this point, which makes sense of the limited numberof connections.

One of the fundamentals of the internet is that connections *will*
break. A friend of mine introduced me to Magic: The Gathering via a
program that couldn't handle drop-outs, and it got extremely
frustrating - we couldn't get a game going. Build your server such
that your clients can disconnect and reconnect, and you protect
yourself against half the problem; allow them to connect and kick the
other connection off, and you solve the other half. (Sometimes, the
server won't know that the client has gone, so it helps to be able to
kick like that.) It might not be an issue when you're playing around
with localhost, and you could even get away with it on a LAN, but on
the internet, it's so much more friendly to your users to let them
connect multiple times like that.

I don't understand. Can you show some examples of how to do this?

Denis gave a decent explanation of the problem, with a few
suggestions. One of the easiest to work with (and trust me, you will
LOVE the ease of debugging this kind of system) is the line-based
connection. You just run a loop like this:

buffer = b''

def gets():
while '\n' not in buffer:
data = sock.recv(1024)
if not data:
# Client is disconnected, handle it gracefully
return None # or some other sentinel
line, buffer = buffer.split(b'\n',1)
return line.decode().replace('\r', '')

You could put this into a class definition that wraps up all the
details. The key here is that you read as much as you can, buffering
it, and as soon as you have a newline, you return that. This works
beautifully with the basic TELNET client, so it's easy to see what's
going on. Its only requirement is that there be no newlines *inside*
commands. The classic MUD structure guarantees that (if you want a
paragraph of text, you have some marker that says "end of paragraph" -
commonly a dot on a line of its own, which is borrowed from SMTP), and
if you use json.dumps() then it'll use two characters "\\" and "n" to
represent a newline, so that's safe too.

The next easiest structure to work with is length-preceded, which
Denis explained. Again, you read until you have a full packet, but
instead of "while '\n' not in buffer", it would be "while
len(buffer)<packetlen". Whichever way you mark packets, you have to be
prepared for both problems: incomplete packets, and packets merged.
And both at once, too - one recv() call might return the tail of one
and the beginning of another.

Note that I've written the above example with the intent that it'll
work on either Python 2 or Python 3 (though it's not actually tested
on either). The more code you can do that way, the easier it'll be...
see next point.

Good point, however the framework I'm using for graphics does not currently support python3. I could make the server scripts be in python3, but I don't think the potential confusion is worth it until the whole thing can bein the same version.

Hmm. Maybe, but on the flip side, it might be better to first learn
how to do things in Py3, and then work on making your code able to run
in Py2. That way, you force yourself to get everything right for Py3,
and then there's no big porting job to move. Apart from the fact that
you have to learn two variants of the language, there's nothing
stopping the server being Py3 while the client's in Py2.

I'm not sure what you are talking about here. Would you care to elaborateon this please (it interests and confuses) ?

Fundamentally, network protocols work with bytes. (Actually, they're
often called 'octets' - and a good number of protocols work with
*bits*, of which an octet is simply a group of eight.) They don't work
with characters. But humans want to send text. I'll use these posts as
an example.

1) Human types text into mail client, newsreader, or web browser.
2) Client wraps that text up somehow and sends it to a server.
3) Server sends it along to other servers.
4) Server sends stuff to another client.
5) Client unwraps the text and shows it to a human.

For there to be viable communication, the text typed in step 1 has to
be the same as the text shown in step 5. So we need to have some kind
of system that says "This character that I see on my screen is
represented by this byte or sequence of bytes". That's called an
encoding. It's simply a mapping of characters to byte sequences.
(There are other complexities, too, but I'll handwave those for the
moment. For now, let's pretend that one character that you see on the
screen - more properly termed a glyph - is represented by one stream
of bytes.) The best way to handle this is Unicode, because it's fairly
safe to assume that most people can handle it. So you design your
protocol to use Unicode characters and UTF-8 encoding. That means
that:

* The character 'A' (LATIN CAPITAL LETTER A) is represented by code
point U+0041 and byte sequence 0x41
* The character '©' (COPYRIGHT SIGN) is code point U+00A9 or 0xC2 0xA9
* '☺' (WHITE SMILING FACE) is U+263A or 0xE2 0x98 0xBA
* 'ð’…' (CUNEIFORM SIGN URU TIMES KI) is U+12345 or 0xF0 0x92 0x8D0x85

(Tip: http://www.fileformat.info/ is a great place to play around with
these sorts of things.)

An agreement like this means that one human can type a white smiling
face, his client will interpret it as U+263A, the email and news posts
will contain E2 98 BA, and the human at the other end will see a white
smiling face. There's more to it than that, at least with these posts,
because not everyone uses UTF-8 (so the encoding has to be declared),
but if you're creating a brand new protocol, you can simply mandate
it. I strongly recommend UTF-8, by the way; it's compact for text
that's mostly Latin characters, it's well known, and it covers the
entire Unicode range (unlike, say, CP-1252 as used on Windows, or the
ISO-8859-? series).

Since you're working with JSON, you could choose to work with ASCII,
as JSON has its own notation for incorporating non-ASCII characters in
an ASCII stream. But I think it's probably better to use UTF-8.

One of the huge advantages of Python 3 over Python 2 is that it forces
you to think about this up-front. There is a stark divide between
bytes and text. In Python 2, you can sorta pretend that ASCII text and
bytes are the same thing, which often leads to programs that work
perfectly until they get to a "weird character". Fact is, there are no
weird characters. I recommend this talk by Ned Batchelder:

http://nedbatchelder.com/text/unipain.html

Watch it, comprehend it, and code with his Facts of Life and Pro Tips
in mind, and you'll have no pain.

I'm posting this on google groups, so I hope the formatting turns out ok thanks.

Your lines are coming out extremely long, but the biggest annoyance of
GG (double-spaced replies) isn't happening. Thank you.

ChrisA

 

[Frank Millman]

You just run a loop like this:

buffer = b''

def gets():
while '\n' not in buffer:
data = sock.recv(1024)
if not data:
# Client is disconnected, handle it gracefully
return None # or some other sentinel
line, buffer = buffer.split(b'\n',1)
return line.decode().replace('\r', '')

I think you may have omitted a line there -

def gets():
while '\n' not in buffer:
data = sock.recv(1024)
if not data:
# Client is disconnected, handle it gracefully
return None # or some other sentinel
#-->
buffer = buffer + data
#-->
line, buffer = buffer.split(b'\n',1)
return line.decode().replace('\r', '')

Also, as I am looking at it, I notice that the second line should say -

while b'\n' not in buffer:

I feel a bit guilty nitpicking, as you have provided a wonderfully
comprehensive answer, but I wanted to make sure the OP did not get confused.

Frank Millman

 

[Chris Angelico]

I think you may have omitted a line there -

def gets():
while '\n' not in buffer:
data = sock.recv(1024)
if not data:
# Client is disconnected, handle it gracefully
return None # or some other sentinel
#-->
buffer = buffer + data
#-->
line, buffer = buffer.split(b'\n',1)
return line.decode().replace('\r', '')

Yes, indeed I did, thanks. Apart from using augmented assignment,
that's exactly what I would have put there, if I'd actually taken a
moment to test the code.

Also, as I am looking at it, I notice that the second line should say -

while b'\n' not in buffer:

Right again. Fortunately, Py3 would catch that one with a TypeError.
See? This is why you should use Py3.

I feel a bit guilty nitpicking, as you have provided a wonderfully
comprehensive answer, but I wanted to make sure the OP did not get confused.

No no, nitpicking is exactly what ensures that the end result is
correct. If I got offended at you correcting my code, it would imply
that I think myself perfect (or at least, that I consider you to be
utterly incapable of noticing my errors), which is provably false
One of the mind-set changes that I had to introduce at work was that
people don't own code, the repository does - if you see an improvement
to something I wrote, or I see an improvement to something you wrote,
they're improvements to be committed, not turf wars to be battled
over.

Especially on something like this, please *do* catch other people's mistakes

ChrisA

 

[Chris Angelico]

One of the fundamentals of the internet is that connections *will*
break. A friend of mine introduced me to Magic: The Gathering via a
program that couldn't handle drop-outs, and it got extremely
frustrating - we couldn't get a game going. Build your server such
that your clients can disconnect and reconnect, and you protect
yourself against half the problem; allow them to connect and kick the
other connection off, and you solve the other half.

Case in point, and a very annoying one: Phone queues do NOT handle
drop-outs. There's no way to reconnect to the queue and resume your
place, you have to start over from the back of the queue. I'm
currently on hold to my ISP because of an outage, and the cordless
phone ran out of battery 27 minutes into an estimated 30-minute wait
time. (Though I suspect it'd be a lot longer than 30 minutes. Those
wait times are notoriously inaccurate.) So now I'm waiting, AGAIN, and
those previous 27 minutes of sitting around with their on-hold music
playing through speakerphone were of no value whatsoever. I can't
transfer to a different handset or connection, I have to just hope
that this one will get through.

With TCP-based servers, it's easy to do better than that - all you
have to do is separate the connection state from the actual socket,
and hang onto a "connection" for some period of time after its socket
disconnects (say, 10-15 minutes). Your users will thank you!

ChrisA

 

[Chris Angelico]

But note VERY carefully that this can open HUGE security holes if not done with extreme care.

Leaving a dangling connection (not session, TCP closes sessions) open is an invitation so bad things happening.

Not sure what you mean here. I'm assuming an authentication system
that stipulates one single active connection per authenticated user
(if you reauthenticate with the same credentials, it'll disconnect the
other one on the presumption that the connection's been lost). In
terms of resource wastage, there's no difference between disconnecting
now and letting it time out, and waiting the ten minutes (or whatever)
and then terminating cleanly. Or do you mean another user gaining
access? It's still governed by the same authentication.

ChrisA

 

[Chris Angelico]

I'm assuming an authentication system
that stipulates one single active connection per authenticated user

Incidentally, in an environment where everything's trusted (LAN or
localhost), the "authentication system" can be as simple as "type a
user name". I've done systems like that; first line entered becomes
the handle or key, and it does the same kick-off system on duplicate.

ChrisA

 

[Chris Angelico]

I was assuming another user picking up the connection using sniffed credentials (and yes, despite all the work on ssh, not all man-in-the-middle attacks have been killed).

If that can happen, then I would much prefer that it kick my
connection off - at least that way, I have some chance of knowing it's
happened. But I suspect that this sort of thing is way WAY out of the
league of the OP's stated problem

ChrisA