Pickling over a socket

[Roger Alexander]

Hi,

I'm trying to understand how to pickle Python objects over a TCP
socket.

In the example below (based on code from Foundations of Python Network
Programming), a client creates a dictionary (lines 34-38) and uses
pickle.dump at line 42 to write the pickled object using file handle
make from a socket. The server de-pickles with pickle.load (line 24),
again using a file handle made from a socket.

When I run the program, the following output is produced:

Listening at ('127.0.0.1', 1060)
Accepted connection from ('127.0.0.1', 49938)
Traceback (most recent call last):
File "pickles.py", line 24, in <module>
d = pickle.load( s_fh )
File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
return Unpickler(file).load()
File "/usr/local/lib/python2.7/pickle.py", line 857, in load
key = read(1)
File "/usr/local/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
socket.error: [Errno 107] Transport endpoint is not connected

I'm at a loss, can anyone provide any guidance?

Thanks,

Roger Alexander

1 import pickle
2 import socket, sys
3
4 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
5
6 HOST = sys.argv.pop() if len(sys.argv) == 3 else '127.0.0.1'
7 PORT = 1060
8
9 if sys.argv[1:] == ['server']:
10
11 s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
12 s.bind((HOST, PORT))
13 s.listen(1)
14
15 print 'Listening at', s.getsockname()
16
17 sc, sockname = s.accept()
18
19 print 'Accepted connection from', sockname
20
21 sc.shutdown(socket.SHUT_WR)
22 sf = s.makefile( "rb" )
23
24 d = pickle.load(sf)
25
26 sc.close()
27 s.close()
28
29 elif sys.argv[1:] == ['client']:
30
31 s.connect((HOST, PORT))
32 s.shutdown(socket.SHUT_RD)
33
34 d = dict()
35
36 d[ 'Name' ] = 'Jake Thompson.'
37 d[ 'Age' ] = 25
38 d[ 'Location' ] = 'Washington, D.C.'
39
40 sf = s.makefile( "wb" )
41
42 pickle.dump( d, sf, pickle.HIGHEST_PROTOCOL )
43
44 s.close()
45
46 else:
47 print >>sys.stderr, 'usage: streamer.py server|client [host]'

 

[Chris Rebert]

Hi,

I'm trying to understand how to pickle Python objects over a TCP
socket.

In the example below (based on code from Foundations of Python Network
Programming), a client creates a dictionary (lines 34-38) and uses
pickle.dump at line 42 to write the pickled object using file handle
make from a socket. The server de-pickles with pickle.load  (line 24),
again using a file handle made from a socket.

When I run the program, the following output is produced:

   Listening at ('127.0.0.1', 1060)
   Accepted connection from ('127.0.0.1', 49938)
   Traceback (most recent call last):
   File "pickles.py", line 24, in <module>
       d = pickle.load( s_fh )
   File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
       return Unpickler(file).load()
   File "/usr/local/lib/python2.7/pickle.py", line 857, in load
       key = read(1)
   File "/usr/local/lib/python2.7/socket.py", line 380, in read
       data = self._sock.recv(left)
   socket.error: [Errno 107] Transport endpoint is not connected

I'm at a loss, can anyone provide any guidance?

Thanks,

Roger Alexander

 1  import pickle
 2  import socket, sys
 3
 4  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 5
 6  HOST = sys.argv.pop() if len(sys.argv) == 3 else '127..0.0.1'
 7  PORT = 1060
 8
 9  if sys.argv[1:] == ['server']:
10
11      s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
12      s.bind((HOST, PORT))
13      s.listen(1)
14
15      print 'Listening at', s.getsockname()
16
17      sc, sockname = s.accept()
18
19      print 'Accepted connection from', sockname
20
21      sc.shutdown(socket.SHUT_WR)

[Haven't done any network programming, so please excuse the naivete of
this suggestion.]

Have you tried removing line #21 and/or #32?

http://docs.python.org/library/socket.html#socket.socket.shutdown :
"socket.shutdown(how) - Shut down one or both halves of the
connection. [...] Depending on the platform, shutting down one half of
the connection can also close the opposite half"

Cheers,
Chris
--
http://blog.rebertia.com

22      sf = s.makefile( "rb" )
23
24      d = pickle.load(sf)
25
26      sc.close()
27      s.close()
28
29  elif sys.argv[1:] == ['client']:
30
31      s.connect((HOST, PORT))
32      s.shutdown(socket.SHUT_RD)

 

[Chris Angelico]

Hi,

I'm trying to understand how to pickle Python objects over a TCP
socket.

In the example below (based on code from Foundations of Python Network
Programming), a client creates a dictionary (lines 34-38) and uses
pickle.dump at line 42 to write the pickled object using file handle
make from a socket. The server de-pickles with pickle.load  (line 24),
again using a file handle made from a socket.

Whenever there's a problem, create simplicity. I would recommend not
using the file-from-socket method, and simply using pickle.dumps() and
pickle.loads() to pickle to/from strings; those strings can then be
sent/received over the socket using standard recv/send functions.

Also, Chris Rebert's idea is a good one, and worth trying.

Chris Angelico

 

[Dan Stromberg]

Hi,

I'm trying to understand how to pickle Python objects over a TCP
socket.

In the example below (based on code from Foundations of Python Network
Programming), a client creates a dictionary (lines 34-38) and uses
pickle.dump at line 42 to write the pickled object using file handle
make from a socket. The server de-pickles with pickle.load  (line 24),
again using a file handle made from a socket.

When I run the program, the following output is produced:

   Listening at ('127.0.0.1', 1060)
   Accepted connection from ('127.0.0.1', 49938)
   Traceback (most recent call last):
   File "pickles.py", line 24, in <module>
       d = pickle.load( s_fh )
   File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
       return Unpickler(file).load()
   File "/usr/local/lib/python2.7/pickle.py", line 857, in load
       key = read(1)
   File "/usr/local/lib/python2.7/socket.py", line 380, in read
       data = self._sock.recv(left)
   socket.error: [Errno 107] Transport endpoint is not connected

I'm at a loss, can anyone provide any guidance?

Thanks,

Roger Alexander

I played around with it until something worked, and ended up with the
below. The most significant change was probably using sc.makefile
instead of s.makefile in the server, but I seemed to need some data
framing too despite the pickling. It's possible you won't need that
if you just flush your file in the client; I don't much pickling
experience - in particular, I don't know if you can concatenate
pickled objects and load them serially from a file-like object without
any (additional) framing.

I like to use bufsock for this sort of stuff, but I'm probably unique
in that. ^_^ http://stromberg.dnsalias.org/~strombrg/bufsock.html

#!/usr/bin/python

import time
import pickle
import socket, sys
import pprint

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

HOST = sys.argv.pop() if len(sys.argv) == 3 else '127.0.0.1'
PORT = 1060

if sys.argv[1:] == ['server']:

s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)

print 'Listening at', s.getsockname()

sc, sockname = s.accept()

print 'Accepted connection from', sockname

sf = sc.makefile( "rb" )

length_list = []
while True:
char = sf.read(1)
if char == '\n':
break
else:
length_list.append(int(char))
length = 0
for digit in length_list:
length = length * 10 + digit
data = sf.read(length)

d = pickle.loads(data)

pprint.pprint(d)

sc.shutdown(socket.SHUT_RDWR)
sc.close()
s.close()

elif sys.argv[1:] == ['client']:

s.connect((HOST, PORT))
# s.shutdown(socket.SHUT_RD)

d = dict()

d[ 'Name' ] = 'Jake Thompson.'
d[ 'Age' ] = 25
d[ 'Location' ] = 'Washington, D.C.'

sf = s.makefile( "wb" )

string = pickle.dumps( d, pickle.HIGHEST_PROTOCOL )
sf.write('%d\n' % len(string))
sf.write(string)
sf.flush()

#time.sleep(10)
sf.close()
s.shutdown(socket.SHUT_RDWR)
# s.close()

else:
print >>sys.stderr, 'usage: streamer.py server|client [host]'

 

[Chris Angelico]

I played around with it until something worked, and ended up with the
below.  The most significant change was probably using sc.makefile
instead of s.makefile in the server...

Oh! I didn't notice that in the OP. Yep, that would do it!

ChrisA

 

[Roger Alexander]

Thanks everybody, got it working.

I appreciate the help!

Roger.

 

[Jean-Paul Calderone]

Thanks everybody, got it working.

 I appreciate the help!

Roger.

It's too bad none of the other respondents pointed out to you that you
_shouldn't do this_! Pickle is not suitable for use over the network
like this. Your server accepts arbitrary code from clients and
executes it. It is completely insecure. Do not use pickle and
sockets together. Notice the large red box at the top of <http://
docs.python.org/library/pickle.html>.

Jean-Paul

 

[Bastian Ballmann]

Am Tue, 19 Apr 2011 19:28:50 -0700 (PDT)

It is completely insecure. Do not use pickle and
sockets together.

Yes pickle is like eval, but that doesnt mean that one should never
ever use it over a socket connection.
What about ssl sockets where client and server authenticate each other?
Or you encrypt the pickle dump with symmetric encryption and only load
it if you can decrypt it? There are ways to ensure that the data you
get can be handled as trusted.
Greets

Basti

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk2ugM8ACgkQEQHD8bvs9q1IIACfeHPy6/Zd7jQ0haEZO7JreUYt
pCYAnRBZEXSwxYBpHlqRtWjJZRdGk0qv
=TaGe
-----END PGP SIGNATURE-----

 

[Chris Angelico]

Yes pickle is like eval, but that doesnt mean that one should never
ever use it over a socket connection.
What about ssl sockets where client and server authenticate each other?
Or you encrypt the pickle dump with symmetric encryption and only load
it if you can decrypt it? There are ways to ensure that the data you
get can be handled as trusted.

No, I disagree. And I'll cite Caesary as evidence of why.

Caesary is a multiplayer game that uses Flash as its client. (I'm told
the back end is Java, which would explain why it starts lagging
horribly when everyone's online at once.) It has some measure of
authentication of the client, but it's not difficult to spoof;
obviously you could go more elaborate and harder to spoof, but that
still doesn't solve the problem. Even public/private key systems won't
work here; someone could get hold of your client and its private key,
and poof.

Caesary uses an Adobe Message Format system, whereby complex objects
get serialized and transmitted in both directions. It's fundamentally
the same as pickling. When I started poking around with things, it
took me very little time to start transmitting my own requests to the
server; my requests were benign (asking it for information), but other
people figured out the same thing and were rather less ethical.

That's why I tend to use and create much simpler protocols for network
transmission. Also, I like to use a MUD client to test my servers,
ergo textual protocols similar to SMTP. Sure, it may be a tad more
verbose than some, but it's usually easy to parse and verify.

Chris Angelico

 

[Bastian Ballmann]

Am Wed, 20 Apr 2011 16:59:19 +1000

Even public/private key systems won't
work here; someone could get hold of your client and its private key,
and poof.

Oh yeah but than all kinds of trusted computing wont work. Sure
one can see it on the net these days looking at the rsa or commodo or
ps3 hack and the like.

No system is totally secure. You can _always_ poke around if a program
uses user input. For example one can totally own a complete computer by
nothing more than a single sql injection attack even if the programmer
implemented some filters. Now would you say one shouldnt use sql
databases cause of that?

My point is using ssl authentication / encryption together with another
symmetric encryption builds up two layers, which I would say is secure
enough to handle the data as trusted.

Greets

Basti

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk2ujHsACgkQEQHD8bvs9q3a+gCgt7OPN8CJqhem9hMa77a7+Ud+
U4UAn2uHBqOWYaC94xY8RwM9OPhZCXqk
=isJb
-----END PGP SIGNATURE-----

 

[Thomas Rachel]

Am 20.04.2011 09:34, schrieb Bastian Ballmann:

No system is totally secure. You can _always_ poke around if a program
uses user input.

It depends on what the program does with the input. If it treats it
appropriately, nothing can happen.

For example one can totally own a complete computer by
nothing more than a single sql injection attack even if the programmer
implemented some filters.

What do yu want with filters here? Not filtering is appropriate against
SQL injection, but escaping.

If Little Bobby Tables is really called "Robert'); DROP TABLE STUDENTS;
--", it is wrong to reject this string - instead, all dangerous
characters inside it must be quoted (in this case: ') and then it does
not harm at all.

Now would you say one shouldnt use sql
databases cause of that?

No, just beware of what can happen and use the dbs and its functions
appropriately.


Thomas

 

[Bastian Ballmann]

Am Wed, 20 Apr 2011 10:25:14 +0200
schrieb Thomas Rachel

It depends on what the program does with the input. If it treats it
appropriately, nothing can happen.

Yes, but the question seems to be what is appropriately.

What do yu want with filters here? Not filtering is appropriate
against SQL injection, but escaping.

Escaping in strings, filtering with numbers etc.

If Little Bobby Tables is really called "Robert'); DROP TABLE
STUDENTS; --", it is wrong to reject this string - instead, all
dangerous characters inside it must be quoted (in this case: ') and
then it does not harm at all.

Well you forgot to escape ; and \ but this seems to slide into OT
Greets

Basti

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk2uoHUACgkQEQHD8bvs9q1biACfcgvU3T2qAnkoZuCYEMPnij8y
R6UAnRiiyuT69xaNUbuiPcwWjRasdcFc
=OoEx
-----END PGP SIGNATURE-----

 

[Chris Angelico]

Well you forgot to escape ; and \ but this seems to slide into OT

The semicolon doesn't need to be escaped in a quoted string, and the
backslash does only if it's the escape character. The
string-safetifier function that I used with DB2 was called "dblapos"
because it simply doubled every apostrophe - nothing else needed. On
the other hand, mysql_real_escape_string will escape quite a few
characters, for convenience in reading dumps.

Am Wed, 20 Apr 2011 18:43:01 +1000


If you generalize it in this way you should never trust any user input
regardless if it comes from the net or from local or the environment
etc.

Yes, but the other half of the issue is that you have to treat
anything that comes over the network as "user input", even if you
think it's from your own program that you control.

Yes and the database is poorly protected, but this happens way too
often.

That's just *sad*.

I totally disagree. Buffer overflow is just a matter of size checking,
but sql injection is a matter of syntax. It's more than just throwing
the input into a magic auto-escape function.

Buffer overruns can happen in all sorts of places; SQL injection can
only happen where you talk to the database. And it IS just a matter of
using a magic auto-escape function, if your library is set up right -
unless, of course, you allow your users to submit SQL themselves (eg a
WHERE clause). That's almost impossible to sanitize, which is why I
would never EVER allow such a thing unless it's actually a trusted
environment (eg PHPMyAdmin - anyone who has access to PMA has access
to the database anyway).

We both agree that one should never trust user input blindly and we also
seem to conform to one can use user input in a appropriate way that's
not the case, but if i read your mail i think you want to tell me one
should never ever use the internet or only write programs without user
input at all.

Not at all; just never *trust* user input. Where thou typest foo,
someone someday will type...

ChrisA

 

[Bastian Ballmann]

Am Wed, 20 Apr 2011 19:26:44 +1000

Yes, but the other half of the issue is that you have to treat
anything that comes over the network as "user input", even if you
think it's from your own program that you control.

Sure.

Buffer overruns can happen in all sorts of places; SQL injection can
only happen where you talk to the database. And it IS just a matter of
using a magic auto-escape function, if your library is set up right -

No. Not all data is strings.

Not at all; just never *trust* user input. Where thou typest foo,
someone someday will type...

I never *trust* the user *blindly* as you do with your
magic-escape-function so where do we disagree?
Greets

Basti

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk2uqkIACgkQEQHD8bvs9q2ZqQCfde7liCQCLPxQAWQFwKiEpU57
RDEAnjas0RhS9rK4BV40Ykm/0OndZ33z
=42XK
-----END PGP SIGNATURE-----