Pickling over a socket

Discussion in 'Python' started by Roger Alexander, Apr 19, 2011.

  1. Hi,

    I'm trying to understand how to pickle Python objects over a TCP
    socket.

    In the example below (based on code from Foundations of Python Network
    Programming), a client creates a dictionary (lines 34-38) and uses
    pickle.dump at line 42 to write the pickled object using file handle
    make from a socket. The server de-pickles with pickle.load (line 24),
    again using a file handle made from a socket.

    When I run the program, the following output is produced:

    Listening at ('127.0.0.1', 1060)
    Accepted connection from ('127.0.0.1', 49938)
    Traceback (most recent call last):
    File "pickles.py", line 24, in <module>
    d = pickle.load( s_fh )
    File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
    File "/usr/local/lib/python2.7/pickle.py", line 857, in load
    key = read(1)
    File "/usr/local/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
    socket.error: [Errno 107] Transport endpoint is not connected

    I'm at a loss, can anyone provide any guidance?

    Thanks,

    Roger Alexander

    1 import pickle
    2 import socket, sys
    3
    4 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    5
    6 HOST = sys.argv.pop() if len(sys.argv) == 3 else '127.0.0.1'
    7 PORT = 1060
    8
    9 if sys.argv[1:] == ['server']:
    10
    11 s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    12 s.bind((HOST, PORT))
    13 s.listen(1)
    14
    15 print 'Listening at', s.getsockname()
    16
    17 sc, sockname = s.accept()
    18
    19 print 'Accepted connection from', sockname
    20
    21 sc.shutdown(socket.SHUT_WR)
    22 sf = s.makefile( "rb" )
    23
    24 d = pickle.load(sf)
    25
    26 sc.close()
    27 s.close()
    28
    29 elif sys.argv[1:] == ['client']:
    30
    31 s.connect((HOST, PORT))
    32 s.shutdown(socket.SHUT_RD)
    33
    34 d = dict()
    35
    36 d[ 'Name' ] = 'Jake Thompson.'
    37 d[ 'Age' ] = 25
    38 d[ 'Location' ] = 'Washington, D.C.'
    39
    40 sf = s.makefile( "wb" )
    41
    42 pickle.dump( d, sf, pickle.HIGHEST_PROTOCOL )
    43
    44 s.close()
    45
    46 else:
    47 print >>sys.stderr, 'usage: streamer.py server|client [host]'
     
    Roger Alexander, Apr 19, 2011
    #1
    1. Advertising

  2. Roger Alexander

    Chris Rebert Guest

    On Tue, Apr 19, 2011 at 11:53 AM, Roger Alexander <> wrote:
    > Hi,
    >
    > I'm trying to understand how to pickle Python objects over a TCP
    > socket.
    >
    > In the example below (based on code from Foundations of Python Network
    > Programming), a client creates a dictionary (lines 34-38) and uses
    > pickle.dump at line 42 to write the pickled object using file handle
    > make from a socket. The server de-pickles with pickle.load  (line 24),
    > again using a file handle made from a socket.
    >
    > When I run the program, the following output is produced:
    >
    >    Listening at ('127.0.0.1', 1060)
    >    Accepted connection from ('127.0.0.1', 49938)
    >    Traceback (most recent call last):
    >    File "pickles.py", line 24, in <module>
    >        d = pickle.load( s_fh )
    >    File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
    >        return Unpickler(file).load()
    >    File "/usr/local/lib/python2.7/pickle.py", line 857, in load
    >        key = read(1)
    >    File "/usr/local/lib/python2.7/socket.py", line 380, in read
    >        data = self._sock.recv(left)
    >    socket.error: [Errno 107] Transport endpoint is not connected
    >
    > I'm at a loss, can anyone provide any guidance?
    >
    > Thanks,
    >
    > Roger Alexander
    >
    >  1  import pickle
    >  2  import socket, sys
    >  3
    >  4  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    >  5
    >  6  HOST = sys.argv.pop() if len(sys.argv) == 3 else '127..0.0.1'
    >  7  PORT = 1060
    >  8
    >  9  if sys.argv[1:] == ['server']:
    > 10
    > 11      s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    > 12      s.bind((HOST, PORT))
    > 13      s.listen(1)
    > 14
    > 15      print 'Listening at', s.getsockname()
    > 16
    > 17      sc, sockname = s.accept()
    > 18
    > 19      print 'Accepted connection from', sockname
    > 20
    > 21      sc.shutdown(socket.SHUT_WR)


    [Haven't done any network programming, so please excuse the naivete of
    this suggestion.]

    Have you tried removing line #21 and/or #32?

    http://docs.python.org/library/socket.html#socket.socket.shutdown :
    "socket.shutdown(how) - Shut down one or both halves of the
    connection. [...] Depending on the platform, shutting down one half of
    the connection can also close the opposite half"

    Cheers,
    Chris
    --
    http://blog.rebertia.com

    > 22      sf = s.makefile( "rb" )
    > 23
    > 24      d = pickle.load(sf)
    > 25
    > 26      sc.close()
    > 27      s.close()
    > 28
    > 29  elif sys.argv[1:] == ['client']:
    > 30
    > 31      s.connect((HOST, PORT))
    > 32      s.shutdown(socket.SHUT_RD)

    <snip>
    > 42      pickle.dump( d, sf, pickle.HIGHEST_PROTOCOL )
    > 43
    > 44      s.close()
     
    Chris Rebert, Apr 19, 2011
    #2
    1. Advertising

  3. On Wed, Apr 20, 2011 at 4:53 AM, Roger Alexander <> wrote:
    > Hi,
    >
    > I'm trying to understand how to pickle Python objects over a TCP
    > socket.
    >
    > In the example below (based on code from Foundations of Python Network
    > Programming), a client creates a dictionary (lines 34-38) and uses
    > pickle.dump at line 42 to write the pickled object using file handle
    > make from a socket. The server de-pickles with pickle.load  (line 24),
    > again using a file handle made from a socket.


    Whenever there's a problem, create simplicity. I would recommend not
    using the file-from-socket method, and simply using pickle.dumps() and
    pickle.loads() to pickle to/from strings; those strings can then be
    sent/received over the socket using standard recv/send functions.

    Also, Chris Rebert's idea is a good one, and worth trying.

    Chris Angelico
     
    Chris Angelico, Apr 19, 2011
    #3
  4. On Tue, Apr 19, 2011 at 11:53 AM, Roger Alexander <> wrote:
    > Hi,
    >
    > I'm trying to understand how to pickle Python objects over a TCP
    > socket.
    >
    > In the example below (based on code from Foundations of Python Network
    > Programming), a client creates a dictionary (lines 34-38) and uses
    > pickle.dump at line 42 to write the pickled object using file handle
    > make from a socket. The server de-pickles with pickle.load  (line 24),
    > again using a file handle made from a socket.
    >
    > When I run the program, the following output is produced:
    >
    >    Listening at ('127.0.0.1', 1060)
    >    Accepted connection from ('127.0.0.1', 49938)
    >    Traceback (most recent call last):
    >    File "pickles.py", line 24, in <module>
    >        d = pickle.load( s_fh )
    >    File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
    >        return Unpickler(file).load()
    >    File "/usr/local/lib/python2.7/pickle.py", line 857, in load
    >        key = read(1)
    >    File "/usr/local/lib/python2.7/socket.py", line 380, in read
    >        data = self._sock.recv(left)
    >    socket.error: [Errno 107] Transport endpoint is not connected
    >
    > I'm at a loss, can anyone provide any guidance?
    >
    > Thanks,
    >
    > Roger Alexander


    I played around with it until something worked, and ended up with the
    below. The most significant change was probably using sc.makefile
    instead of s.makefile in the server, but I seemed to need some data
    framing too despite the pickling. It's possible you won't need that
    if you just flush your file in the client; I don't much pickling
    experience - in particular, I don't know if you can concatenate
    pickled objects and load them serially from a file-like object without
    any (additional) framing.

    I like to use bufsock for this sort of stuff, but I'm probably unique
    in that. ^_^ http://stromberg.dnsalias.org/~strombrg/bufsock.html

    #!/usr/bin/python

    import time
    import pickle
    import socket, sys
    import pprint

    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    HOST = sys.argv.pop() if len(sys.argv) == 3 else '127.0.0.1'
    PORT = 1060

    if sys.argv[1:] == ['server']:

    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind((HOST, PORT))
    s.listen(1)

    print 'Listening at', s.getsockname()

    sc, sockname = s.accept()

    print 'Accepted connection from', sockname

    sf = sc.makefile( "rb" )

    length_list = []
    while True:
    char = sf.read(1)
    if char == '\n':
    break
    else:
    length_list.append(int(char))
    length = 0
    for digit in length_list:
    length = length * 10 + digit
    data = sf.read(length)

    d = pickle.loads(data)

    pprint.pprint(d)

    sc.shutdown(socket.SHUT_RDWR)
    sc.close()
    s.close()

    elif sys.argv[1:] == ['client']:

    s.connect((HOST, PORT))
    # s.shutdown(socket.SHUT_RD)

    d = dict()

    d[ 'Name' ] = 'Jake Thompson.'
    d[ 'Age' ] = 25
    d[ 'Location' ] = 'Washington, D.C.'

    sf = s.makefile( "wb" )

    string = pickle.dumps( d, pickle.HIGHEST_PROTOCOL )
    sf.write('%d\n' % len(string))
    sf.write(string)
    sf.flush()

    #time.sleep(10)
    sf.close()
    s.shutdown(socket.SHUT_RDWR)
    # s.close()

    else:
    print >>sys.stderr, 'usage: streamer.py server|client [host]'
     
    Dan Stromberg, Apr 19, 2011
    #4
  5. On Wed, Apr 20, 2011 at 5:30 AM, Dan Stromberg <> wrote:
    > I played around with it until something worked, and ended up with the
    > below.  The most significant change was probably using sc.makefile
    > instead of s.makefile in the server...


    Oh! I didn't notice that in the OP. Yep, that would do it!

    ChrisA
     
    Chris Angelico, Apr 19, 2011
    #5
  6. Thanks everybody, got it working.

    I appreciate the help!

    Roger.
     
    Roger Alexander, Apr 19, 2011
    #6
  7. On Apr 19, 6:27 pm, Roger Alexander <> wrote:
    > Thanks everybody, got it working.
    >
    >  I appreciate the help!
    >
    > Roger.


    It's too bad none of the other respondents pointed out to you that you
    _shouldn't do this_! Pickle is not suitable for use over the network
    like this. Your server accepts arbitrary code from clients and
    executes it. It is completely insecure. Do not use pickle and
    sockets together. Notice the large red box at the top of <http://
    docs.python.org/library/pickle.html>.

    Jean-Paul
     
    Jean-Paul Calderone, Apr 20, 2011
    #7
  8. Am Tue, 19 Apr 2011 19:28:50 -0700 (PDT)
    schrieb Jean-Paul Calderone <>:

    > It is completely insecure. Do not use pickle and
    > sockets together.


    Yes pickle is like eval, but that doesnt mean that one should never
    ever use it over a socket connection.
    What about ssl sockets where client and server authenticate each other?
    Or you encrypt the pickle dump with symmetric encryption and only load
    it if you can decrypt it? There are ways to ensure that the data you
    get can be handled as trusted.
    Greets

    Basti

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.11 (GNU/Linux)

    iEYEARECAAYFAk2ugM8ACgkQEQHD8bvs9q1IIACfeHPy6/Zd7jQ0haEZO7JreUYt
    pCYAnRBZEXSwxYBpHlqRtWjJZRdGk0qv
    =TaGe
    -----END PGP SIGNATURE-----
     
    Bastian Ballmann, Apr 20, 2011
    #8
  9. On Wed, Apr 20, 2011 at 4:44 PM, Bastian Ballmann <> wrote:
    > Yes pickle is like eval, but that doesnt mean that one should never
    > ever use it over a socket connection.
    > What about ssl sockets where client and server authenticate each other?
    > Or you encrypt the pickle dump with symmetric encryption and only load
    > it if you can decrypt it? There are ways to ensure that the data you
    > get can be handled as trusted.


    No, I disagree. And I'll cite Caesary as evidence of why.

    Caesary is a multiplayer game that uses Flash as its client. (I'm told
    the back end is Java, which would explain why it starts lagging
    horribly when everyone's online at once.) It has some measure of
    authentication of the client, but it's not difficult to spoof;
    obviously you could go more elaborate and harder to spoof, but that
    still doesn't solve the problem. Even public/private key systems won't
    work here; someone could get hold of your client and its private key,
    and poof.

    Caesary uses an Adobe Message Format system, whereby complex objects
    get serialized and transmitted in both directions. It's fundamentally
    the same as pickling. When I started poking around with things, it
    took me very little time to start transmitting my own requests to the
    server; my requests were benign (asking it for information), but other
    people figured out the same thing and were rather less ethical.

    That's why I tend to use and create much simpler protocols for network
    transmission. Also, I like to use a MUD client to test my servers,
    ergo textual protocols similar to SMTP. Sure, it may be a tad more
    verbose than some, but it's usually easy to parse and verify.

    Chris Angelico
     
    Chris Angelico, Apr 20, 2011
    #9
  10. Am Wed, 20 Apr 2011 16:59:19 +1000
    schrieb Chris Angelico <>:

    > Even public/private key systems won't
    > work here; someone could get hold of your client and its private key,
    > and poof.


    Oh yeah but than all kinds of trusted computing wont work. Sure
    one can see it on the net these days looking at the rsa or commodo or
    ps3 hack and the like.

    No system is totally secure. You can _always_ poke around if a program
    uses user input. For example one can totally own a complete computer by
    nothing more than a single sql injection attack even if the programmer
    implemented some filters. Now would you say one shouldnt use sql
    databases cause of that? ;)

    My point is using ssl authentication / encryption together with another
    symmetric encryption builds up two layers, which I would say is secure
    enough to handle the data as trusted.

    Greets

    Basti

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.11 (GNU/Linux)

    iEYEARECAAYFAk2ujHsACgkQEQHD8bvs9q3a+gCgt7OPN8CJqhem9hMa77a7+Ud+
    U4UAn2uHBqOWYaC94xY8RwM9OPhZCXqk
    =isJb
    -----END PGP SIGNATURE-----
     
    Bastian Ballmann, Apr 20, 2011
    #10
  11. Am 20.04.2011 09:34, schrieb Bastian Ballmann:

    > No system is totally secure. You can _always_ poke around if a program
    > uses user input.


    It depends on what the program does with the input. If it treats it
    appropriately, nothing can happen.


    > For example one can totally own a complete computer by
    > nothing more than a single sql injection attack even if the programmer
    > implemented some filters.


    What do yu want with filters here? Not filtering is appropriate against
    SQL injection, but escaping.

    If Little Bobby Tables is really called "Robert'); DROP TABLE STUDENTS;
    --", it is wrong to reject this string - instead, all dangerous
    characters inside it must be quoted (in this case: ') and then it does
    not harm at all.


    > Now would you say one shouldnt use sql
    > databases cause of that? ;)


    No, just beware of what can happen and use the dbs and its functions
    appropriately.


    Thomas
     
    Thomas Rachel, Apr 20, 2011
    #11
  12. [OT] Re: Pickling over a socket

    Am Wed, 20 Apr 2011 10:25:14 +0200
    schrieb Thomas Rachel
    <>:

    > It depends on what the program does with the input. If it treats it
    > appropriately, nothing can happen.


    Yes, but the question seems to be what is appropriately.


    > What do yu want with filters here? Not filtering is appropriate
    > against SQL injection, but escaping.


    Escaping in strings, filtering with numbers etc.


    > If Little Bobby Tables is really called "Robert'); DROP TABLE
    > STUDENTS; --", it is wrong to reject this string - instead, all
    > dangerous characters inside it must be quoted (in this case: ') and
    > then it does not harm at all.


    Well you forgot to escape ; and \ but this seems to slide into OT ;)
    Greets

    Basti

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.11 (GNU/Linux)

    iEYEARECAAYFAk2uoHUACgkQEQHD8bvs9q1biACfcgvU3T2qAnkoZuCYEMPnij8y
    R6UAnRiiyuT69xaNUbuiPcwWjRasdcFc
    =OoEx
    -----END PGP SIGNATURE-----
     
    Bastian Ballmann, Apr 20, 2011
    #12
  13. On Wed, Apr 20, 2011 at 7:17 PM, Bastian Ballmann <> wrote:
    > Well you forgot to escape ; and \ but this seems to slide into OT ;)


    The semicolon doesn't need to be escaped in a quoted string, and the
    backslash does only if it's the escape character. The
    string-safetifier function that I used with DB2 was called "dblapos"
    because it simply doubled every apostrophe - nothing else needed. On
    the other hand, mysql_real_escape_string will escape quite a few
    characters, for convenience in reading dumps.

    > Am Wed, 20 Apr 2011 18:43:01 +1000
    > schrieb Chris Angelico <>:
    >
    >> So, like Jean-Paul said, you simply do not trust anything that comes
    >> from the network. Ever.

    >
    > If you generalize it in this way you should never trust any user input
    > regardless if it comes from the net or from local or the environment
    > etc.


    Yes, but the other half of the issue is that you have to treat
    anything that comes over the network as "user input", even if you
    think it's from your own program that you control.

    >> Urrrm. You can "own" a "complete computer" with SQL injection? Then
    >> someone has some seriously weak protection.

    >
    > Yes and the database is poorly protected, but this happens way too
    > often.


    That's just *sad*.

    >> SQL injection is easier to
    >> protect against than buffer overruns, and with a lot of function
    >> libraries

    >
    > I totally disagree. Buffer overflow is just a matter of size checking,
    > but sql injection is a matter of syntax. It's more than just throwing
    > the input into a magic auto-escape function.


    Buffer overruns can happen in all sorts of places; SQL injection can
    only happen where you talk to the database. And it IS just a matter of
    using a magic auto-escape function, if your library is set up right -
    unless, of course, you allow your users to submit SQL themselves (eg a
    WHERE clause). That's almost impossible to sanitize, which is why I
    would never EVER allow such a thing unless it's actually a trusted
    environment (eg PHPMyAdmin - anyone who has access to PMA has access
    to the database anyway).

    > We both agree that one should never trust user input blindly and we also
    > seem to conform to one can use user input in a appropriate way that's
    > not the case, but if i read your mail i think you want to tell me one
    > should never ever use the internet or only write programs without user
    > input at all.


    Not at all; just never *trust* user input. Where thou typest foo,
    someone someday will type...

    ChrisA
     
    Chris Angelico, Apr 20, 2011
    #13
  14. Am Wed, 20 Apr 2011 19:26:44 +1000
    schrieb Chris Angelico <>:

    > Yes, but the other half of the issue is that you have to treat
    > anything that comes over the network as "user input", even if you
    > think it's from your own program that you control.


    Sure.


    > Buffer overruns can happen in all sorts of places; SQL injection can
    > only happen where you talk to the database. And it IS just a matter of
    > using a magic auto-escape function, if your library is set up right -


    No. Not all data is strings.


    > Not at all; just never *trust* user input. Where thou typest foo,
    > someone someday will type...


    I never *trust* the user *blindly* as you do with your
    magic-escape-function so where do we disagree?
    Greets

    Basti

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.11 (GNU/Linux)

    iEYEARECAAYFAk2uqkIACgkQEQHD8bvs9q2ZqQCfde7liCQCLPxQAWQFwKiEpU57
    RDEAnjas0RhS9rK4BV40Ykm/0OndZ33z
    =42XK
    -----END PGP SIGNATURE-----
     
    Bastian Ballmann, Apr 20, 2011
    #14
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Laszlo Nagy
    Replies:
    1
    Views:
    4,926
    Mark Wooding
    Jan 27, 2009
  2. Jean-Paul Calderone
    Replies:
    0
    Views:
    992
    Jean-Paul Calderone
    Jan 27, 2009
  3. Laszlo Nagy
    Replies:
    0
    Views:
    566
    Laszlo Nagy
    Feb 1, 2009
  4. Steve Holden
    Replies:
    0
    Views:
    684
    Steve Holden
    Feb 1, 2009
  5. Steve Holden
    Replies:
    1
    Views:
    730
Loading...

Share This Page