Code: Rolling a Container Into a String

K

Kamilche

I want to convert a dict into string form, then back again. After
discovering that eval is insecure, I wrote some code to roll a Python
object, dict, tuple, or list into a string. I've posted it below. Does
anyone know an easier way to accomplish this? Essentially, I want to
avoid doing an 'eval' on a string to get it back into dict form... but
still allow nested structures. (My current code doesn't handle nested
structures.)

I conked out before writing the 'unroll' code. I'm going to go watch
some boob tube with my husband, instead, and leave that code for
another day. If you know of a better way, or feel like writing the
'unroll' code and posting it, by all means, do so! :-D I'll check
Google newsgroups before I tackle the job, to see if some kind soul
took pity on me.

--Kamilche

import types

SimpleTypes = [types.BooleanType, types.FloatType, types.IntType, \
types.LongType, types.NoneType, types.StringType]

_dictdelim1 = "{"
_dictdelim2 = "}"
_listdelim1 = "["
_listdelim2 = "]"
_tupledelim1 = "("
_tupledelim2 = ")"

def roll(item):
"Return a string representation of an object, dict, tuple, or
list."
return _roll2(item, [], {})


def unroll(s):
"Unrolls a string back into a dict, tuple, or list."
if type(s) != types.StringType:
raise Exception("You may only pass strings to this function!")
err = "Error occurred when parsing " + s + "!"
state = 0
container = None
for c in s:
if c == _dictdelim1:
lookfor = _dictdelim2
elif c == _listdelim1:
lookfor = _listdelim2
elif c == _tupledelim1:
lookfor = _tupledelim2
else:
raise Exception(err)
state = 1



def _quoted(s):
' Return a stringized value'
if type(s) != types.StringType:
return str(s)
else:
l = []
s = s.replace("'", "\'")
l.append("'")
l.append(s)
l.append("'")
return ''.join(l)

def _roll2(d, lst, r):
' Function that does the work.'
# Start of _roll2
t = type(d)
if t == types.DictType:
theid = id(d)
if theid in r:
raise Exception("Recursion detected! Stopping now.")
r[theid] = theid
cnt = 0
lst.append(_dictdelim1)
for key in d.keys():
if key[0] != '_':
lst.append(_quoted(key))
lst.append(': ')
t = type(d[key])
if t in SimpleTypes:
lst.append(_quoted(d[key]))
else:
_roll2(d[key], lst, r)
lst.append(", ")
cnt += 1
if cnt > 0:
del lst[-1]
lst.append(_dictdelim2)
elif t in (types.ListType, types.TupleType):
theid = id(d)
if theid in r:
raise Exception("Recursion detected! Stopping now.")
r[theid] = theid
cnt = 0
if t == types.ListType:
lst.append(_listdelim1)
else:
lst.append(_tupledelim1)
for item in d:
if type(item) in SimpleTypes:
lst.append(_quoted(item))
else:
_roll2(item, lst, r)
lst.append(", ")
cnt += 1
if cnt > 0:
del lst[-1]
if t == types.ListType:
lst.append(_listdelim2)
else:
lst.append(_tupledelim2)
elif hasattr(d, '__dict__'):
_roll2(d.__dict__, lst, r)
else:
raise Exception("Unhandled type " + str(t) + \
"! You may only pass dicts, tuples, lists, and
" + \
"objects with a __dict__ to this function!")
return ''.join(lst)




class simple:
pass

def main():
l = ['List1', 'List2', 'List3']
d = {'Dict1': 'd1', 'Dict2': 'd2', 'list': l}
t = ('Tuple1', d, 'Tuple2')

print "It handles dicts, lists, and tuples."
print roll(t), "\n"

o = simple()
o.name = 'the name'
o.password = 'the password'
o.list = ['ol1', 'ol2']
o.dict = {'od1': None, 'od2': 2}
o.tuple = ('tuple1', 'tuple2')
o.float = 1.5
o.long = 12345678901234567890
o.bool = True
o.int = 1

print "It handles objects."
print roll(o), "\n"

print "It won't roll attributes whose name starts with '_'"
o._recursion = o.tuple
print roll(o), "\n"

print "It will raise an exception if it detects recursion."
print "This next one will cause recursion."
o.recursion = o.tuple
print roll(o), "\n"

print "You can't roll simple types."
print roll('a'), "\n"

if __name__ == "__main__":
main()
 
T

Terry Reedy

Kamilche said:
I want to convert a dict into string form, then back again. After
discovering that eval is insecure,

With arbitrary code from an arbitrary source, yes.
If you *know* that you are eval-ing your own safe strings, then no problem.
I wrote some code to roll a Python
object, dict, tuple, or list into a string.

repr(object) already does that for you. Why duplicate the work?

You only need custom a eval function, which might check that string is safe
(no function calls, no list comps) and then eval, or which might do parsing
and construction itself.

Terry J. Reedy
 
D

David Fraser

Terry said:
With arbitrary code from an arbitrary source, yes.
If you *know* that you are eval-ing your own safe strings, then no problem.




repr(object) already does that for you. Why duplicate the work?

You only need custom a eval function, which might check that string is safe
(no function calls, no list comps) and then eval, or which might do parsing
and construction itself.

Terry J. Reedy
Or use the pprint module which does nice pretty-printing

David
 
P

Paul McGuire

Kamilche said:
I want to convert a dict into string form, then back again. After
discovering that eval is insecure, I wrote some code to roll a Python
object, dict, tuple, or list into a string. I've posted it below. Does
anyone know an easier way to accomplish this? Essentially, I want to
avoid doing an 'eval' on a string to get it back into dict form... but
still allow nested structures. (My current code doesn't handle nested
structures.)
unroll is just repr().

Here is a pyparsing routine to "roll up" your data structures, a mere 60
lines or so. It's fairly tolerant of some odd cases, and fully handles
nested data. (Extension to include remaining items, such as boolean data
and scientific notation, is left as an exercise for the reader.) I hope
this is fairly easy to follow - I've dropped in a few comments.

For those of you who've been living in a cave, you can download pyparsing at
http://pyparsing.sourceforge.net.

-- Paul


from pyparsing import Word, ZeroOrMore, OneOrMore, Suppress, Forward, \
quotedString,nums,Combine,Optional,delimitedList,Group

# create a dictionary of ugly data, complete with nested lists, tuples
# and dictionaries, even imaginary numbers!
d1 = {}
d1['a'] = [1,2,3,[4,5,6]]
d1['b'] = (7,8,(9,10),'a',"",'')
d1['c'] = { 'aa' : 1, 'bb' : "lskdj'slkdjf", 'cc':1.232, 'dd':(('z',),) }
d1[('d','e')] = 5+10j

print repr(d1)

testdata = repr(d1)
"""
looks like this:
{'a': [1, 2, 3, [4, 5, 6]], ('d', 'e'): (5+10j), 'c': {'aa': 1, 'cc': 1.232,
'dd': (('z',),), 'bb': "lskdj'slkdjf"}, 'b': (7, 8, (9, 10), 'a', '', '')}
"""

#define low-level data elements
intNum = Word( nums+"+-", nums )
realNum = Combine(intNum + "." + Optional(Word(nums)))
number = realNum | intNum
imagNum = Combine( "(" + number + "+" + number + "j" + ")" )

item = Forward() # set up for recursive grammar definition
tupleDef = Suppress("(") + ( delimitedList( item ) ^
( item + Suppress(",") ) ) + Suppress(")")
listDef = Suppress("[") + delimitedList( item ) + Suppress("]")
keyDef = tupleDef | quotedString | imagNum | number
keyVal = Group( keyDef + Suppress(":") + item )
dictDef = Suppress("{") + delimitedList( keyVal ) + Suppress("}")

item << ( quotedString | number | imagNum |
tupleDef | listDef | dictDef )

# define low-level conversion routines
intNum.setParseAction( lambda s,loc,toks: int(toks[0]) )
realNum.setParseAction( lambda s,loc,toks: float(toks[0]) )
imagNum.setParseAction( lambda s,loc,toks: eval(toks[0]) ) # no built-in to
convert imaginaries?

# strip leading and trailing character from parsed quoted string
quotedString.setParseAction( lambda s,loc,toks: toks[0][1:-1] )

# define list-to-list/tuple/dict routines
evalTuple = lambda s,loc,toks: [ tuple(toks) ]
evalList = lambda s,loc,toks: [ toks.asList() ]
evalDict = lambda s,loc,toks: [ dict([tuple(kv) for kv in toks]) ]

tupleDef.setParseAction( evalTuple )
listDef.setParseAction( evalList )
dictDef.setParseAction( evalDict )

# first element of returned tokens list is the reconstructed list/tuple/dict
results = item.parseString( testdata )[0]
print results

if repr(results) == repr(d1):
print "Eureka!"
else:
print "Compare results for mismatch"
print repr(results)
print repr(d1)
 
?

=?iso-8859-15?Q?Pierre-Fr=E9d=E9ric_Caillaud?=

Use YAML

import yaml

then from your code:

yaml.dump( whatever ) =>

then yaml.loadstring(str)...

It handles objects.
{'name': 'the name', 'tuple': ('tuple1', 'tuple2'), 'int': 1, 'float':
1.5, 'list': ['ol1', 'ol2'], 'long': 12345678901234567890,
'dict': {'od1': None, 'od2': 2}, 'bool': True, 'password': 'the password'}

--- !!__main__.simple # instanciates class for you
bool: True
dict:
od1: ~
od2: 2
float: 1.5
int: 1
list:
- ol1
- ol2
long: 12345678901234567890
name: the name
password: the password
tuple:
- tuple1
- tuple2

It won't roll attributes whose name starts with '_'
{'name': 'the name', 'tuple': ('tuple1', 'tuple2'), 'int': 1, 'float':
1.5, 'list': ['ol1', 'ol2'], 'long': 12345678901234567890,
'dict': {'od1': None, 'od2': 2}, 'bool': True, 'password': 'the password'}

--- !!__main__.simple
_recursion: # yaml does not handle recursion (I have an old version)
- tuple1
- tuple2
bool: True
dict:
od1: ~
od2: 2
float: 1.5
int: 1
list:
- ol1
- ol2
long: 12345678901234567890
name: the name
password: the password
tuple:
- tuple1
- tuple2

It will raise an exception if it detects recursion.
This next one will cause recursion.
--- !!__main__.simple
_recursion:
- tuple1
- tuple2
bool: True
dict:
od1: ~
od2: 2
float: 1.5
int: 1
list:
- ol1
- ol2
long: 12345678901234567890
name: the name
password: the password
recursion:
- tuple1
- tuple2
tuple:
- tuple1
- tuple2
 
O

Oliver Pieper

I want to convert a dict into string form, then back again.

Since you probably don't want to do this just for the joy of
converting something to string an back ...

Maybe the pickle (and cPickle) module does what you are looking for.

Oliver
 
K

Kamilche

Pierre-Frédéric Caillaud said:
It looked interesting, so I downloaded it... and was confronted with
dozens of files, and the need to compile before use... when I was
looking for a simple cross-platform 2 function solution that didn't
take any DLL's. Dang.

Well, it's a new day, maybe I'll be inspired.
 
?

=?iso-8859-15?Q?Pierre-Fr=E9d=E9ric_Caillaud?=

did you download syck or the pure python yaml parser ?
on Linux the pure python module is just a matter of typing "emerge sync"
but I don't know about Syck...
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,769
Messages
2,569,579
Members
45,053
Latest member
BrodieSola

Latest Threads

Top