Alternatives for pickle?

Discussion in 'Python' started by Antoon Pardon, Oct 11, 2004.

  1. I'm writing a little game, a gridler application, where you
    can turn pixmaps into puzzle's and try to solve them. I already
    have the data structure for such a puzzle worked out, one of
    the problems is writing it to a file and reading it back in.

    I first went to the pickle module but there I read this.

    | Warning: The pickle module is not intended to be secure against
    | erroneous or maliciously constructed data. Never unpickle data
    | received from an untrusted or unauthenticated source.

    But since this is for a game and people should be able to
    exchange puzzles, it seems a heavy requirement to ask of
    the users to check a puzzle file for security hazards.


    I also thought about writing out a string that, when read
    back in and fed to eval would recreate the structure. But
    that seems to be just as insecure if not more so.

    So how do you serialize data in python, when you want
    a somewhat secure mechanisme. Preferably where a user
    can make a puzzle file by hand in a text editor.

    --
    Antoon Pardon
     
    Antoon Pardon, Oct 11, 2004
    #1
    1. Advertising

  2. Antoon Pardon wrote:
    > So how do you serialize data in python, when you want
    > a somewhat secure mechanisme. Preferably where a user
    > can make a puzzle file by hand in a text editor.


    There's a YAML module for Python, although if I recall
    correctly this module also suffers from security issues.

    You could try Gnosis utils xml pickling or the xml marshaler
    from the pyxml package. They're slow, but safe.

    --Irmen
     
    Irmen de Jong, Oct 11, 2004
    #2
    1. Advertising

  3. Antoon Pardon

    Robert Kern Guest

    Antoon Pardon wrote:
    > I'm writing a little game, a gridler application, where you
    > can turn pixmaps into puzzle's and try to solve them. I already
    > have the data structure for such a puzzle worked out, one of
    > the problems is writing it to a file and reading it back in.
    >
    > I first went to the pickle module but there I read this.
    >
    > | Warning: The pickle module is not intended to be secure against
    > | erroneous or maliciously constructed data. Never unpickle data
    > | received from an untrusted or unauthenticated source.
    >
    > But since this is for a game and people should be able to
    > exchange puzzles, it seems a heavy requirement to ask of
    > the users to check a puzzle file for security hazards.
    >
    >
    > I also thought about writing out a string that, when read
    > back in and fed to eval would recreate the structure. But
    > that seems to be just as insecure if not more so.


    Indeed. Don't do that.

    > So how do you serialize data in python, when you want
    > a somewhat secure mechanisme. Preferably where a user
    > can make a puzzle file by hand in a text editor.


    I think this is a case where you need to come up with your own file
    format and parse it yourself. Pickle and other such mechanisms have
    security problems because they are so general. They will create objects
    that you don't want.

    You can always jump on the XML bandwagon if that is convenient for you.
    Python has XML modules in the standard library. Depending on the
    complexity of the structure, it might even be convenient to edit by hand
    in a text editor.

    --
    Robert Kern


    "In the fields of hell where the grass grows high
    Are the graves of dreams allowed to die."
    -- Richard Harter
     
    Robert Kern, Oct 11, 2004
    #3
  4. Antoon Pardon

    Jeff Shannon Guest

    Antoon Pardon wrote:

    >I also thought about writing out a string that, when read
    >back in and fed to eval would recreate the structure. But
    >that seems to be just as insecure if not more so.
    >
    >


    As I understand it, this is *exactly* what pickle already does.

    >So how do you serialize data in python, when you want
    >a somewhat secure mechanisme. Preferably where a user
    >can make a puzzle file by hand in a text editor.
    >
    >


    I'd agree with the earlier comment -- define your own file format, and
    write code to parse that format and instantiate the necessary objects.
    If it's hard to define something that's both effective for your
    purposes, and hard to hand-code in a text editor, then consider writing
    a puzzle-editor app that will allow GUI creation of puzzles which can be
    saved in your custom file format.

    Jeff Shannon
    Technician/Programmer
    Credit International
     
    Jeff Shannon, Oct 12, 2004
    #4
  5. Antoon Pardon <> wrote in message news:<>...
    > I'm writing a little game, a gridler application, where you
    > can turn pixmaps into puzzle's and try to solve them. I already
    > have the data structure for such a puzzle worked out, one of
    > the problems is writing it to a file and reading it back in.
    >
    > I first went to the pickle module but there I read this.
    >
    > | Warning: The pickle module is not intended to be secure against
    > | erroneous or maliciously constructed data. Never unpickle data
    > | received from an untrusted or unauthenticated source.
    >


    Hmmm..... I wonder how easy it is to craft a malicious pickle that
    will automatically run code objects just because they are unpickled.
    My guess is that it's quite difficult - I've never heard of it *ever*
    being done. Someone would have to be *very* malicious to work out how
    to do it on the off chance of planting a back door into someone's
    machine through a program like yours. No offence intended, but if they
    were going to go to all that effort I expect they might aim for
    something with a wider audience.

    I would expect it to be 'safe enough', but that might not be safe
    enough for you !

    Creating your own data format is probably the way forward - and
    probably not that difficult either.

    Regards,

    Fuzzy

    http://www.voidspace.org.uk/atlantibots/pythonutils.html

    > But since this is for a game and people should be able to
    > exchange puzzles, it seems a heavy requirement to ask of
    > the users to check a puzzle file for security hazards.
    >
    >
    > I also thought about writing out a string that, when read
    > back in and fed to eval would recreate the structure. But
    > that seems to be just as insecure if not more so.
    >
    > So how do you serialize data in python, when you want
    > a somewhat secure mechanisme. Preferably where a user
    > can make a puzzle file by hand in a text editor.
     
    Michael Foord, Oct 12, 2004
    #5
  6. Antoon Pardon wrote:
    > | Warning: The pickle module is not intended to be secure against
    > | erroneous or maliciously constructed data. Never unpickle data
    > | received from an untrusted or unauthenticated source.
    >
    > But since this is for a game and people should be able to
    > exchange puzzles, it seems a heavy requirement to ask of
    > the users to check a puzzle file for security hazards.
    >


    http://twistedmatrix.com/products/spread#jelly

    I haven't used it myself, though. In fact you might be able to use
    twisted in other ways as well.

    Shalabh

    --
    http://www.qlime.org
     
    Shalabh Chaturvedi, Oct 12, 2004
    #6
  7. Antoon Pardon

    Paul Rubin Guest

    Antoon Pardon <> writes:
    > So how do you serialize data in python, when you want
    > a somewhat secure mechanisme. Preferably where a user
    > can make a puzzle file by hand in a text editor.


    There are a lot of different serialization formats in the Python
    library but the general ones are not secure and the secure ones are
    not general. You may have to concoct an ad-hoc format just for your
    puzzles.
     
    Paul Rubin, Oct 12, 2004
    #7
  8. On 11 Oct 2004 08:26:12 GMT, Antoon Pardon <> wrote:

    >I'm writing a little game, a gridler application, where you
    >can turn pixmaps into puzzle's and try to solve them. I already
    >have the data structure for such a puzzle worked out, one of
    >the problems is writing it to a file and reading it back in.
    >
    >I first went to the pickle module but there I read this.
    >
    >| Warning: The pickle module is not intended to be secure against
    >| erroneous or maliciously constructed data. Never unpickle data
    >| received from an untrusted or unauthenticated source.
    >
    >But since this is for a game and people should be able to
    >exchange puzzles, it seems a heavy requirement to ask of
    >the users to check a puzzle file for security hazards.
    >
    >
    >I also thought about writing out a string that, when read
    >back in and fed to eval would recreate the structure. But
    >that seems to be just as insecure if not more so.
    >
    >So how do you serialize data in python, when you want
    >a somewhat secure mechanisme. Preferably where a user
    >can make a puzzle file by hand in a text editor.
    >

    I would consider saving and retrieving your puzzle info in a simple csv format.
    You can invent your own very simple interpreter based on lines of the
    form
    cmd, whatever...

    the csv module has methods and options to control delimiters and quoting etc,
    but e.g., by default:

    >>> import csv
    >>> lines = """

    ... this, can, be, a, command, format
    ... cmd, easy to edit with editor, note what happened to spaces
    ... cmd, arg, "quoted arg", etc
    ... do, something, else
    ... push, something on a stack
    ... set, something,to,a,value
    ... call, afunction, arg1, arg2, etc
    ... etc
    ... """
    >>> rdr = csv.reader(lines.splitlines())
    >>> for row in rdr: print row

    ...
    []
    ['this', ' can', ' be', ' a', ' command', ' format']
    ['cmd', ' easy to edit with editor', ' note what happened to spaces']
    ['cmd', ' arg', ' "quoted arg"', ' etc']
    ['do', ' something', ' else']
    ['push', ' something on a stack']
    ['set', ' something', 'to', 'a', 'value']
    ['call', ' afunction', ' arg1', ' arg2', ' etc']
    ['etc']

    Anything that will iterate by lines should be ok to pass to csv.reader
    so you can pass an open file, e.g., file('mypuzzlesetup.txt')

    Obviously, you can interpret row[0] as operation and do what you like, e.g.,


    >>> rdr = csv.reader("""\

    ... abs, 123
    ... abs, -123
    ... sum, 1,2,3
    ... sum, 100,200,5
    ... xxx, what?
    ... """.splitlines())

    >>> for row in rdr:

    ... cmd = getattr(__builtins__, row[0], None)
    ... if cmd is None: print 're-edit your info:',row
    ... else:
    ... args = map(int, row[1:])
    ... if len(args)==1: args=args[0]
    ... print row,'=>', cmd(args)
    ...
    ['abs', ' 123'] => 123
    ['abs', ' -123'] => 123
    ['sum', ' 1', '2', '3'] => 6
    ['sum', ' 100', '200', '5'] => 305
    re-edit your info: ['xxx', ' what?']

    The csv module also has stuff to control delimiters and a writer method, etc.
    See help(csv) interactively, after importing csv.

    Note that no one can introduce an xxx do do anything weird, and if you
    validate all the command formats, and don't do int conversions etc without
    try/except, etc, you should be able to reject anything not safe, and write
    informative error messages. You don't necessarily have to abort on any and
    all errors, but you can have options for that if you want to get fancy.

    HTH

    Regards,
    Bengt Richter
     
    Bengt Richter, Oct 13, 2004
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Robert V. Hanson
    Replies:
    2
    Views:
    697
    Yan-Hong Huang[MSFT]
    Jul 3, 2003
  2. Don Pedro
    Replies:
    0
    Views:
    502
    Don Pedro
    Jul 3, 2003
  3. Mark Kamoski

    Re: alternatives to SmartNavigation

    Mark Kamoski, Aug 28, 2003, in forum: ASP .Net
    Replies:
    0
    Views:
    457
    Mark Kamoski
    Aug 28, 2003
  4. a pickle's pickle

    , Aug 2, 2005, in forum: Python
    Replies:
    4
    Views:
    407
  5. Michele Simionato
    Replies:
    2
    Views:
    1,936
    Michele Simionato
    May 23, 2008
Loading...

Share This Page