Securing 'pickle'

Discussion in 'Python' started by Ben Finney, Jul 11, 2003.

  1. Ben Finney

    Ben Finney Guest

    On Fri, 11 Jul 2003 13:20:48 +1200, David McNab wrote:
    > I'm writing a web app framework which stores pickles in client
    > cookies.


    Sounds like a waste of bandwidth, in addition to the security concerns
    you raise.

    Why not store the pickles on the server, and set a session cookie to
    refer to them? That way, you only send a short session ID instead of
    the whole pickle, and messing with the cookie doesn't alter the pickles.

    (Mmm, all this talk of food...)

    --
    \ "I don't know anything about music. In my line you don't have |
    `\ to." -- Elvis Aaron Presley (1935-1977) |
    _o__) |
    http://bignose.squidly.org/ 9CFE12B0 791A4267 887F520C B7AC2E51 BD41714B
    Ben Finney, Jul 11, 2003
    #1
    1. Advertising

  2. Ben Finney

    David McNab Guest

    Hi,

    I'm writing a web app framework which stores pickles in client cookies.

    The obvious security risk is that some 5cr1p7 X1ddi35 will inevitably try
    tampering with the cookie and malforming it in an attempt to get the
    server-side python code to run arbitrary code, or something similarly
    undesirable.

    To protect against this, I've subclassed pickle.Unpickler, and added
    overrides of the methods load_global, load_inst, load_obj and find_class.

    My override methods simply raise exceptions unconditionally, which causes
    any unpickle to fail if the pickle tries to unpack anything even
    resembling code or an object.

    I did this in preference to using the reputable 'bencode' module from
    BitTorrent, because bencode doesn't support floats.

    My question - have I done enough, or are there still ways where my hobbled
    unpickler could be subverted by a malformed cookie?

    Cheers
    David
    David McNab, Jul 11, 2003
    #2
    1. Advertising

  3. Paul Rubin wrote:

    > Because now you need a mechanism to store the session info on the
    > server, and you might want it to work across multiple load-balanced
    > servers that fail over to one another, etc.


    That's far superior to presenting the opportunity to exploits in the
    first place, in my opinion. Depending on the contents of the contents
    of that cookie, what you suggest may not be a problem at all (depending
    on how critical the data contained therein is).

    --
    Erik Max Francis && && http://www.alcyone.com/max/
    __ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
    / \ I would rather understand one cause than be king of Persia.
    \__/ Democritus
    Erik Max Francis, Jul 11, 2003
    #3
  4. Ben Finney

    Ian Bicking Guest

    On Thu, 2003-07-10 at 20:20, David McNab wrote:
    > I'm writing a web app framework which stores pickles in client cookies.
    >
    > The obvious security risk is that some 5cr1p7 X1ddi35 will inevitably try
    > tampering with the cookie and malforming it in an attempt to get the
    > server-side python code to run arbitrary code, or something similarly
    > undesirable.
    >
    > To protect against this, I've subclassed pickle.Unpickler, and added
    > overrides of the methods load_global, load_inst, load_obj and find_class.


    A much easier way to secure your pickle is to sign it, like:

    cookie = dumps(object)
    secret = 'really secret!'
    hasher = md5.new()
    hasher.update(secret)
    hasher.update(cookie)
    cookie_signature = md5.digest()

    You may then wish to base64 encode both (.encode('base64')), pop them
    into one value, and you're off. Though I suppose at that point you may
    be hitting the maximum value of a cookie. Hidden fields will work
    nicely, though.

    Decoding and verifying is an exercise left to the reader.

    Ian
    Ian Bicking, Jul 11, 2003
    #4
  5. Ben Finney

    Paul Rubin Guest

    Erik Max Francis <> writes:
    > > Because now you need a mechanism to store the session info on the
    > > server, and you might want it to work across multiple load-balanced
    > > servers that fail over to one another, etc.

    >
    > That's far superior to presenting the opportunity to exploits in the
    > first place, in my opinion. Depending on the contents of the contents
    > of that cookie, what you suggest may not be a problem at all (depending
    > on how critical the data contained therein is).


    I'm not sure what you're saying here. My suggestion is to
    authenticate the cookies with a cryptographic checksum and verify the
    authentication before deserializing the cookies. That's probably the
    simplest approach. Keeping session info on a multi-process server (or
    worse, a multi-server network) needs some kind of concurrent storage
    mechanism. I don't see a robust, secure, low-overhead way to do that
    with out-of-the-box Python. Any suggestions?
    Paul Rubin, Jul 11, 2003
    #5
  6. Ben Finney

    Ian Bicking Guest

    On Thu, 2003-07-10 at 20:52, Erik Max Francis wrote:
    > > Because now you need a mechanism to store the session info on the
    > > server, and you might want it to work across multiple load-balanced
    > > servers that fail over to one another, etc.

    >
    > That's far superior to presenting the opportunity to exploits in the
    > first place, in my opinion. Depending on the contents of the contents
    > of that cookie, what you suggest may not be a problem at all (depending
    > on how critical the data contained therein is).


    Security isn't a big deal -- or rather, securing cookies isn't a big
    deal. I think reliability will be a bigger problem. Cookies can cause
    problems even when you are just storing a simple session ID. If you
    start storing more information you're likely to run up against other
    problems -- cookies can be hard to dispose of, who knows where they'll
    get chopped off to preserve storage (it happens quickly), and IE has a
    bug where you can't redirect and set a cookie at the same time, which
    can really drive you crazy if you don't know about it.

    Hidden fields are a much better way of keeping information on the
    client. They tend to make for more navigable pages too. But if you
    really want session, not transaction data, then you just need to figure
    out server-side sessions. The biggest advantage of a web application is
    that it runs in a controlled environment (the server) and you should
    take advantage of that.

    Ian
    Ian Bicking, Jul 11, 2003
    #6
  7. Ben Finney

    Paul Rubin Guest

    Ian Bicking <> writes:
    > A much easier way to secure your pickle is to sign it, like:
    >
    > cookie = dumps(object)
    > secret = 'really secret!'
    > hasher = md5.new()
    > hasher.update(secret)
    > hasher.update(cookie)
    > cookie_signature = md5.digest()


    That method is vulnerable to an "appending" attack against md5. I'll
    spare the gory details, but you should call md5 through the HMAC
    module to make the signature instead of using md5 directly. HMAC is
    designed to stop that attack.

    > You may then wish to base64 encode both (.encode('base64')), pop them
    > into one value, and you're off. Though I suppose at that point you may
    > be hitting the maximum value of a cookie. Hidden fields will work
    > nicely, though.


    You could split the session info into several cookies, but in that
    situation you should authenticate the whole cookie set with a single
    signature. Otherwise someone could paste together several cookies
    from separate sessions, and possibly confuse your server.
    Paul Rubin, Jul 11, 2003
    #7
  8. Ben Finney

    Dave Cole Guest


    > On Thu, 2003-07-10 at 20:20, David McNab wrote:
    > > I'm writing a web app framework which stores pickles in client cookies.
    > >
    > > The obvious security risk is that some 5cr1p7 X1ddi35 will inevitably try
    > > tampering with the cookie and malforming it in an attempt to get the
    > > server-side python code to run arbitrary code, or something similarly
    > > undesirable.
    > >
    > > To protect against this, I've subclassed pickle.Unpickler, and added
    > > overrides of the methods load_global, load_inst, load_obj and find_class.

    >
    > A much easier way to secure your pickle is to sign it, like:
    >
    > cookie = dumps(object)
    > secret = 'really secret!'
    > hasher = md5.new()
    > hasher.update(secret)
    > hasher.update(cookie)
    > cookie_signature = md5.digest()
    >
    > You may then wish to base64 encode both (.encode('base64')), pop
    > them into one value, and you're off. Though I suppose at that point
    > you may be hitting the maximum value of a cookie. Hidden fields
    > will work nicely, though.
    >
    > Decoding and verifying is an exercise left to the reader.


    That is exactly what Albatross does with pickles sent to the browser.
    In case it is interesting to anyone, here is the class that does the
    work of signing and checking the sign.

    - Dave

    class PickleSignMixin:

    def __init__(self, secret):
    self.__secret = secret

    def pickle_sign(self, text):
    m = md5.new()
    m.update(self.__secret)
    m.update(text)
    text = m.digest() + text
    return text

    def pickle_unsign(self, text):
    digest = text[:16]
    text = text[16:]
    m = md5.new()
    m.update(self.__secret)
    m.update(text)
    if m.digest() == digest:
    return text
    return ''


    --
    http://www.object-craft.com.au
    Dave Cole, Jul 11, 2003
    #8
  9. Ben Finney

    Paul Rubin Guest

    Dave Cole <> writes:
    > I have been googling for information on the "appending" attack against
    > md5 and cannot find anything that clearly describes it. Do you have
    > any links handy?


    I think RFC 2104 (the HMAC spec) might describe it. Basically, think
    about how md5 works. You load the md5 context with the secret key
    (say 20 bytes) then your data (say 20 bytes), then some padding to
    fill the 64 byte context, and run the compression function:

    md5_compress(key + data + 24 bytes of padding)

    Call the 24 padding bytes P. They are just 16 0's plus an 8 byte
    length, iirc.

    The hash output is just the md5 chaining variables after running the
    compression function.

    Now look at the 100 byte string

    E = your data + P (same as above) + 36 bytes of evil stuff

    Even without knowing your secret key, if the attacker knows your data
    (which may not be secret), and md5(key+data) (which you've included in
    the cookie), he can compute the signature of E. It's just the result
    of running the compression function on his evil stuff plus appropriate
    additional padding, with the chaining variables set to the original
    md5 hash that you already sent him.

    This is not really a failure of md5, which is supposed to be a message
    digest algorithm, not a MAC. Rather, the authentication fails because
    md5 is being used in a way it was not intended to be used.

    The solution is to use HMAC. See RFC 2104 for details.
    Paul Rubin, Jul 11, 2003
    #9
  10. Ben Finney

    John J. Lee Guest

    Ian Bicking <> writes:
    [...]
    > Security isn't a big deal -- or rather, securing cookies isn't a big
    > deal.


    I don't understand. The problem is that pickles can be constructed
    that can damage systems when unpickled, is that right? If that's
    true, then surely unpickling cookie data is unsafe, because stuff
    coming in from the network has to be regarded as malevolent. Are you
    saying that web server environments are sufficiently-well bolted down
    that no pickle attack will work? But belt-and-braces is the best
    policy, isn't it?


    > and IE has a
    > bug where you can't redirect and set a cookie at the same time, which
    > can really drive you crazy if you don't know about it.

    [...]

    Hah. There's a slight irony there, given that they fought against
    restrictions on setting cookies from 'unverified' third parties when
    the (more-or-less stillborn) cookie RFCs were being written. So they
    argue against that, then end up partially implementing it by
    accident...


    John
    John J. Lee, Jul 11, 2003
    #10
  11. Ben Finney

    Paul Rubin Guest

    (John J. Lee) writes:
    > > Security isn't a big deal -- or rather, securing cookies isn't a big
    > > deal.

    >
    > I don't understand. The problem is that pickles can be constructed
    > that can damage systems when unpickled, is that right? If that's
    > true, then surely unpickling cookie data is unsafe, because stuff
    > coming in from the network has to be regarded as malevolent. Are you
    > saying that web server environments are sufficiently-well bolted down
    > that no pickle attack will work? But belt-and-braces is the best
    > policy, isn't it?


    The point is that you can use cryptographic signatures to make sure
    any cookie you receive is one that the server actually sent, before
    deciding to unpickle it. That means if the attacker constructs a
    malicious cookie, you never unpickle it.
    Paul Rubin, Jul 11, 2003
    #11
  12. >
    >
    >>Security isn't a big deal -- or rather, securing cookies isn't a big
    >>deal.
    >>
    >>

    >
    >I don't understand. The problem is that pickles can be constructed
    >that can damage systems when unpickled, is that right? If that's
    >true, then surely unpickling cookie data is unsafe, because stuff
    >coming in from the network has to be regarded as malevolent. Are you
    >saying that web server environments are sufficiently-well bolted down
    >that no pickle attack will work? But belt-and-braces is the best
    >policy, isn't it?
    >


    I'm sorry, I just caught this thread and I don't know your problem very
    well.
    I'm extensively using this (see below). It is unable to pickle class
    instances but
    you won't accidentally run constructors. I think it is safe for
    sending/receiving data.
    I did not try to publish this yet but it would be great to know if this
    is safe or not
    so please make comments.

    Laci 1.0

    import cStringIO
    import cPickle

    def dumps(obj):
    f = cStringIO.StringIO()
    p = cPickle.Pickler(f,1)
    p.dump(obj)
    return f.getvalue()

    def loads(s):
    f = cStringIO.StringIO(s)
    p = cPickle.Unpickler(f)
    p.find_global = None
    return p.load()
    =?ISO-8859-1?Q?Nagy_L=E1szl=F3_Zsolt?=, Jul 11, 2003
    #12
  13. Ben Finney

    Alan Kennedy Guest

    Paul Rubin wrote:

    > My suggestion is to
    > authenticate the cookies with a cryptographic checksum and verify the
    > authentication before deserializing the cookies. That's probably the
    > simplest approach. Keeping session info on a multi-process server (or
    > worse, a multi-server network) needs some kind of concurrent storage
    > mechanism.


    Paul,

    Do you mean transmit the checksum to the client with the cookie? And
    check that they match when the cookie and checksum come back?

    Or is the checksum stored on the server, in some form of lookup
    dictionary keyed by some user session identifier?

    regards,

    --
    alan kennedy
    -----------------------------------------------------
    check http headers here: http://xhaus.com/headers
    email alan: http://xhaus.com/mailto/alan
    Alan Kennedy, Jul 11, 2003
    #13
  14. Ben Finney

    Paul Rubin Guest

    Alan Kennedy <> writes:
    > Do you mean transmit the checksum to the client with the cookie? And
    > check that they match when the cookie and checksum come back?


    Yes. See other posts in the thread for sample code.

    > Or is the checksum stored on the server, in some form of lookup
    > dictionary keyed by some user session identifier?


    If you have a convenient way to do that, it's best to just send a
    session number in the cookie, and keep all the session data on the
    server. Then you don't ever have to unpickle anything.
    Paul Rubin, Jul 11, 2003
    #14
  15. >
    >
    >
    >
    >>My suggestion is to
    >>authenticate the cookies with a cryptographic checksum and verify the
    >>authentication before deserializing the cookies. That's probably the
    >>simplest approach. Keeping session info on a multi-process server (or
    >>worse, a multi-server network) needs some kind of concurrent storage
    >>mechanism.
    >>
    >>

    >
    >Paul,
    >
    >Do you mean transmit the checksum to the client with the cookie? And
    >check that they match when the cookie and checksum come back?
    >
    >Or is the checksum stored on the server, in some form of lookup
    >dictionary keyed by some user session identifier?
    >

    I think he wanted to write a digital signature instead. Right?

    Laci 1.0
    =?ISO-8859-1?Q?Nagy_L=E1szl=F3_Zsolt?=, Jul 11, 2003
    #15
  16. Ben Finney

    Paul Rubin Guest

    Nagy László Zsolt <> writes:
    > >Or is the checksum stored on the server, in some form of lookup
    > >dictionary keyed by some user session identifier?
    > >

    > I think he wanted to write a digital signature instead. Right?


    I used "cryptographic checksum" in a broad sense. More specifically
    the suggestion is to use a secret-key authentication code like HMAC-MD5.
    Paul Rubin, Jul 11, 2003
    #16
  17. Ben Finney

    Ian Bicking Guest

    On Fri, 2003-07-11 at 10:00, John J. Lee wrote:
    > Ian Bicking <> writes:
    > [...]
    > > Security isn't a big deal -- or rather, securing cookies isn't a big
    > > deal.

    >
    > I don't understand. The problem is that pickles can be constructed
    > that can damage systems when unpickled, is that right? If that's
    > true, then surely unpickling cookie data is unsafe, because stuff
    > coming in from the network has to be regarded as malevolent. Are you
    > saying that web server environments are sufficiently-well bolted down
    > that no pickle attack will work? But belt-and-braces is the best
    > policy, isn't it?


    I should have said "securing cookies isn't hard", so that's not the
    reason not to use them (though you shouldn't just use plain-vanilla
    cookies).

    Ian
    Ian Bicking, Jul 11, 2003
    #17
  18. Ben Finney

    Paul Rubin Guest

    Ian Bicking <> writes:
    > I should have said "securing cookies isn't hard", so that's not the
    > reason not to use them (though you shouldn't just use plain-vanilla
    > cookies).


    The signature scheme we've discussed does cause some configuration
    hassle. There has to be a host-specific secret key and it has to be
    kept secret. If it leaks to an attacker, the attacker can then create
    malicious cookies. So the scheme has to be used with care.
    Paul Rubin, Jul 11, 2003
    #18
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. abdulrauf

    securing pages and forms from users

    abdulrauf, Jul 31, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    327
    Eric Wise
    Jul 31, 2003
  2. Ian B
    Replies:
    2
    Views:
    2,233
    Ian B
    Jan 23, 2004
  3. Rob Meade

    Securing/Encrypting QueryStrings

    Rob Meade, Jan 22, 2004, in forum: ASP .Net
    Replies:
    0
    Views:
    364
    Rob Meade
    Jan 22, 2004
  4. a pickle's pickle

    , Aug 2, 2005, in forum: Python
    Replies:
    4
    Views:
    385
  5. Michele Simionato
    Replies:
    2
    Views:
    1,881
    Michele Simionato
    May 23, 2008
Loading...

Share This Page