Alternatives for pickle?

[Antoon Pardon]

I'm writing a little game, a gridler application, where you
can turn pixmaps into puzzle's and try to solve them. I already
have the data structure for such a puzzle worked out, one of
the problems is writing it to a file and reading it back in.

I first went to the pickle module but there I read this.

| Warning: The pickle module is not intended to be secure against
| erroneous or maliciously constructed data. Never unpickle data
| received from an untrusted or unauthenticated source.

But since this is for a game and people should be able to
exchange puzzles, it seems a heavy requirement to ask of
the users to check a puzzle file for security hazards.


I also thought about writing out a string that, when read
back in and fed to eval would recreate the structure. But
that seems to be just as insecure if not more so.

So how do you serialize data in python, when you want
a somewhat secure mechanisme. Preferably where a user
can make a puzzle file by hand in a text editor.

 

[Irmen de Jong]

So how do you serialize data in python, when you want
a somewhat secure mechanisme. Preferably where a user
can make a puzzle file by hand in a text editor.

There's a YAML module for Python, although if I recall
correctly this module also suffers from security issues.

You could try Gnosis utils xml pickling or the xml marshaler
from the pyxml package. They're slow, but safe.

--Irmen

 

[Robert Kern]

I'm writing a little game, a gridler application, where you
can turn pixmaps into puzzle's and try to solve them. I already
have the data structure for such a puzzle worked out, one of
the problems is writing it to a file and reading it back in.

I first went to the pickle module but there I read this.

| Warning: The pickle module is not intended to be secure against
| erroneous or maliciously constructed data. Never unpickle data
| received from an untrusted or unauthenticated source.

But since this is for a game and people should be able to
exchange puzzles, it seems a heavy requirement to ask of
the users to check a puzzle file for security hazards.


I also thought about writing out a string that, when read
back in and fed to eval would recreate the structure. But
that seems to be just as insecure if not more so.

Indeed. Don't do that.

So how do you serialize data in python, when you want
a somewhat secure mechanisme. Preferably where a user
can make a puzzle file by hand in a text editor.

I think this is a case where you need to come up with your own file
format and parse it yourself. Pickle and other such mechanisms have
security problems because they are so general. They will create objects
that you don't want.

You can always jump on the XML bandwagon if that is convenient for you.
Python has XML modules in the standard library. Depending on the
complexity of the structure, it might even be convenient to edit by hand
in a text editor.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[Jeff Shannon]

I also thought about writing out a string that, when read
back in and fed to eval would recreate the structure. But
that seems to be just as insecure if not more so.

As I understand it, this is *exactly* what pickle already does.

So how do you serialize data in python, when you want
a somewhat secure mechanisme. Preferably where a user
can make a puzzle file by hand in a text editor.

I'd agree with the earlier comment -- define your own file format, and
write code to parse that format and instantiate the necessary objects.
If it's hard to define something that's both effective for your
purposes, and hard to hand-code in a text editor, then consider writing
a puzzle-editor app that will allow GUI creation of puzzles which can be
saved in your custom file format.

Jeff Shannon
Technician/Programmer
Credit International

 

[Michael Foord]

I'm writing a little game, a gridler application, where you
can turn pixmaps into puzzle's and try to solve them. I already
have the data structure for such a puzzle worked out, one of
the problems is writing it to a file and reading it back in.

I first went to the pickle module but there I read this.

| Warning: The pickle module is not intended to be secure against
| erroneous or maliciously constructed data. Never unpickle data
| received from an untrusted or unauthenticated source.

Hmmm..... I wonder how easy it is to craft a malicious pickle that
will automatically run code objects just because they are unpickled.
My guess is that it's quite difficult - I've never heard of it *ever*
being done. Someone would have to be *very* malicious to work out how
to do it on the off chance of planting a back door into someone's
machine through a program like yours. No offence intended, but if they
were going to go to all that effort I expect they might aim for
something with a wider audience.

I would expect it to be 'safe enough', but that might not be safe
enough for you !

Creating your own data format is probably the way forward - and
probably not that difficult either.

Regards,

Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html

 

[Shalabh Chaturvedi]

| Warning: The pickle module is not intended to be secure against
| erroneous or maliciously constructed data. Never unpickle data
| received from an untrusted or unauthenticated source.

But since this is for a game and people should be able to
exchange puzzles, it seems a heavy requirement to ask of
the users to check a puzzle file for security hazards.

http://twistedmatrix.com/products/spread#jelly

I haven't used it myself, though. In fact you might be able to use
twisted in other ways as well.

Shalabh

 

[Paul Rubin]

So how do you serialize data in python, when you want
a somewhat secure mechanisme. Preferably where a user
can make a puzzle file by hand in a text editor.

There are a lot of different serialization formats in the Python
library but the general ones are not secure and the secure ones are
not general. You may have to concoct an ad-hoc format just for your
puzzles.

 

[Bengt Richter]

I'm writing a little game, a gridler application, where you
can turn pixmaps into puzzle's and try to solve them. I already
have the data structure for such a puzzle worked out, one of
the problems is writing it to a file and reading it back in.

I first went to the pickle module but there I read this.

| Warning: The pickle module is not intended to be secure against
| erroneous or maliciously constructed data. Never unpickle data
| received from an untrusted or unauthenticated source.

But since this is for a game and people should be able to
exchange puzzles, it seems a heavy requirement to ask of
the users to check a puzzle file for security hazards.


I also thought about writing out a string that, when read
back in and fed to eval would recreate the structure. But
that seems to be just as insecure if not more so.

So how do you serialize data in python, when you want
a somewhat secure mechanisme. Preferably where a user
can make a puzzle file by hand in a text editor.

I would consider saving and retrieving your puzzle info in a simple csv format.
You can invent your own very simple interpreter based on lines of the
form
cmd, whatever...

the csv module has methods and options to control delimiters and quoting etc,
but e.g., by default:
... this, can, be, a, command, format
... cmd, easy to edit with editor, note what happened to spaces
... cmd, arg, "quoted arg", etc
... do, something, else
... push, something on a stack
... set, something,to,a,value
... call, afunction, arg1, arg2, etc
... etc
... """ ...
[]
['this', ' can', ' be', ' a', ' command', ' format']
['cmd', ' easy to edit with editor', ' note what happened to spaces']
['cmd', ' arg', ' "quoted arg"', ' etc']
['do', ' something', ' else']
['push', ' something on a stack']
['set', ' something', 'to', 'a', 'value']
['call', ' afunction', ' arg1', ' arg2', ' etc']
['etc']

Anything that will iterate by lines should be ok to pass to csv.reader
so you can pass an open file, e.g., file('mypuzzlesetup.txt')

Obviously, you can interpret row[0] as operation and do what you like, e.g.,

... abs, 123
... abs, -123
... sum, 1,2,3
... sum, 100,200,5
... xxx, what?
... """.splitlines())
... cmd = getattr(__builtins__, row[0], None)
... if cmd is None: print 're-edit your info:',row
... else:
... args = map(int, row[1:])
... if len(args)==1: args=args[0]
... print row,'=>', cmd(args)
...
['abs', ' 123'] => 123
['abs', ' -123'] => 123
['sum', ' 1', '2', '3'] => 6
['sum', ' 100', '200', '5'] => 305
re-edit your info: ['xxx', ' what?']

The csv module also has stuff to control delimiters and a writer method, etc.
See help(csv) interactively, after importing csv.

Note that no one can introduce an xxx do do anything weird, and if you
validate all the command formats, and don't do int conversions etc without
try/except, etc, you should be able to reject anything not safe, and write
informative error messages. You don't necessarily have to abort on any and
all errors, but you can have options for that if you want to get fancy.

HTH

Regards,
Bengt Richter