Code: Rolling a Container Into a String

Discussion in 'Python' started by Kamilche, Jun 25, 2004.

  1. Kamilche

    Kamilche Guest

    I want to convert a dict into string form, then back again. After
    discovering that eval is insecure, I wrote some code to roll a Python
    object, dict, tuple, or list into a string. I've posted it below. Does
    anyone know an easier way to accomplish this? Essentially, I want to
    avoid doing an 'eval' on a string to get it back into dict form... but
    still allow nested structures. (My current code doesn't handle nested
    structures.)

    I conked out before writing the 'unroll' code. I'm going to go watch
    some boob tube with my husband, instead, and leave that code for
    another day. If you know of a better way, or feel like writing the
    'unroll' code and posting it, by all means, do so! :-D I'll check
    Google newsgroups before I tackle the job, to see if some kind soul
    took pity on me.

    --Kamilche

    import types

    SimpleTypes = [types.BooleanType, types.FloatType, types.IntType, \
    types.LongType, types.NoneType, types.StringType]

    _dictdelim1 = "{"
    _dictdelim2 = "}"
    _listdelim1 = "["
    _listdelim2 = "]"
    _tupledelim1 = "("
    _tupledelim2 = ")"

    def roll(item):
    "Return a string representation of an object, dict, tuple, or
    list."
    return _roll2(item, [], {})


    def unroll(s):
    "Unrolls a string back into a dict, tuple, or list."
    if type(s) != types.StringType:
    raise Exception("You may only pass strings to this function!")
    err = "Error occurred when parsing " + s + "!"
    state = 0
    container = None
    for c in s:
    if c == _dictdelim1:
    lookfor = _dictdelim2
    elif c == _listdelim1:
    lookfor = _listdelim2
    elif c == _tupledelim1:
    lookfor = _tupledelim2
    else:
    raise Exception(err)
    state = 1



    def _quoted(s):
    ' Return a stringized value'
    if type(s) != types.StringType:
    return str(s)
    else:
    l = []
    s = s.replace("'", "\'")
    l.append("'")
    l.append(s)
    l.append("'")
    return ''.join(l)

    def _roll2(d, lst, r):
    ' Function that does the work.'
    # Start of _roll2
    t = type(d)
    if t == types.DictType:
    theid = id(d)
    if theid in r:
    raise Exception("Recursion detected! Stopping now.")
    r[theid] = theid
    cnt = 0
    lst.append(_dictdelim1)
    for key in d.keys():
    if key[0] != '_':
    lst.append(_quoted(key))
    lst.append(': ')
    t = type(d[key])
    if t in SimpleTypes:
    lst.append(_quoted(d[key]))
    else:
    _roll2(d[key], lst, r)
    lst.append(", ")
    cnt += 1
    if cnt > 0:
    del lst[-1]
    lst.append(_dictdelim2)
    elif t in (types.ListType, types.TupleType):
    theid = id(d)
    if theid in r:
    raise Exception("Recursion detected! Stopping now.")
    r[theid] = theid
    cnt = 0
    if t == types.ListType:
    lst.append(_listdelim1)
    else:
    lst.append(_tupledelim1)
    for item in d:
    if type(item) in SimpleTypes:
    lst.append(_quoted(item))
    else:
    _roll2(item, lst, r)
    lst.append(", ")
    cnt += 1
    if cnt > 0:
    del lst[-1]
    if t == types.ListType:
    lst.append(_listdelim2)
    else:
    lst.append(_tupledelim2)
    elif hasattr(d, '__dict__'):
    _roll2(d.__dict__, lst, r)
    else:
    raise Exception("Unhandled type " + str(t) + \
    "! You may only pass dicts, tuples, lists, and
    " + \
    "objects with a __dict__ to this function!")
    return ''.join(lst)




    class simple:
    pass

    def main():
    l = ['List1', 'List2', 'List3']
    d = {'Dict1': 'd1', 'Dict2': 'd2', 'list': l}
    t = ('Tuple1', d, 'Tuple2')

    print "It handles dicts, lists, and tuples."
    print roll(t), "\n"

    o = simple()
    o.name = 'the name'
    o.password = 'the password'
    o.list = ['ol1', 'ol2']
    o.dict = {'od1': None, 'od2': 2}
    o.tuple = ('tuple1', 'tuple2')
    o.float = 1.5
    o.long = 12345678901234567890
    o.bool = True
    o.int = 1

    print "It handles objects."
    print roll(o), "\n"

    print "It won't roll attributes whose name starts with '_'"
    o._recursion = o.tuple
    print roll(o), "\n"

    print "It will raise an exception if it detects recursion."
    print "This next one will cause recursion."
    o.recursion = o.tuple
    print roll(o), "\n"

    print "You can't roll simple types."
    print roll('a'), "\n"

    if __name__ == "__main__":
    main()
     
    Kamilche, Jun 25, 2004
    #1
    1. Advertising

  2. Kamilche

    Terry Reedy Guest

    Re: Rolling a Container Into a String

    "Kamilche" <> wrote in message
    news:...
    > I want to convert a dict into string form, then back again. After
    > discovering that eval is insecure,


    With arbitrary code from an arbitrary source, yes.
    If you *know* that you are eval-ing your own safe strings, then no problem.

    >I wrote some code to roll a Python
    > object, dict, tuple, or list into a string.


    repr(object) already does that for you. Why duplicate the work?

    You only need custom a eval function, which might check that string is safe
    (no function calls, no list comps) and then eval, or which might do parsing
    and construction itself.

    Terry J. Reedy
     
    Terry Reedy, Jun 25, 2004
    #2
    1. Advertising

  3. Kamilche

    David Fraser Guest

    Re: Rolling a Container Into a String

    Terry Reedy wrote:
    > "Kamilche" <> wrote in message
    > news:...
    >
    >>I want to convert a dict into string form, then back again. After
    >>discovering that eval is insecure,

    >
    >
    > With arbitrary code from an arbitrary source, yes.
    > If you *know* that you are eval-ing your own safe strings, then no problem.
    >
    >
    >>I wrote some code to roll a Python
    >>object, dict, tuple, or list into a string.

    >
    >
    > repr(object) already does that for you. Why duplicate the work?
    >
    > You only need custom a eval function, which might check that string is safe
    > (no function calls, no list comps) and then eval, or which might do parsing
    > and construction itself.
    >
    > Terry J. Reedy
    >
    >

    Or use the pprint module which does nice pretty-printing

    David
     
    David Fraser, Jun 25, 2004
    #3
  4. Kamilche

    Paul McGuire Guest

    Re: Rolling a Container Into a String

    "Kamilche" <> wrote in message
    news:...
    > I want to convert a dict into string form, then back again. After
    > discovering that eval is insecure, I wrote some code to roll a Python
    > object, dict, tuple, or list into a string. I've posted it below. Does
    > anyone know an easier way to accomplish this? Essentially, I want to
    > avoid doing an 'eval' on a string to get it back into dict form... but
    > still allow nested structures. (My current code doesn't handle nested
    > structures.)
    >

    unroll is just repr().

    Here is a pyparsing routine to "roll up" your data structures, a mere 60
    lines or so. It's fairly tolerant of some odd cases, and fully handles
    nested data. (Extension to include remaining items, such as boolean data
    and scientific notation, is left as an exercise for the reader.) I hope
    this is fairly easy to follow - I've dropped in a few comments.

    For those of you who've been living in a cave, you can download pyparsing at
    http://pyparsing.sourceforge.net.

    -- Paul


    from pyparsing import Word, ZeroOrMore, OneOrMore, Suppress, Forward, \
    quotedString,nums,Combine,Optional,delimitedList,Group

    # create a dictionary of ugly data, complete with nested lists, tuples
    # and dictionaries, even imaginary numbers!
    d1 = {}
    d1['a'] = [1,2,3,[4,5,6]]
    d1['b'] = (7,8,(9,10),'a',"",'')
    d1['c'] = { 'aa' : 1, 'bb' : "lskdj'slkdjf", 'cc':1.232, 'dd':(('z',),) }
    d1[('d','e')] = 5+10j

    print repr(d1)

    testdata = repr(d1)
    """
    looks like this:
    {'a': [1, 2, 3, [4, 5, 6]], ('d', 'e'): (5+10j), 'c': {'aa': 1, 'cc': 1.232,
    'dd': (('z',),), 'bb': "lskdj'slkdjf"}, 'b': (7, 8, (9, 10), 'a', '', '')}
    """

    #define low-level data elements
    intNum = Word( nums+"+-", nums )
    realNum = Combine(intNum + "." + Optional(Word(nums)))
    number = realNum | intNum
    imagNum = Combine( "(" + number + "+" + number + "j" + ")" )

    item = Forward() # set up for recursive grammar definition
    tupleDef = Suppress("(") + ( delimitedList( item ) ^
    ( item + Suppress(",") ) ) + Suppress(")")
    listDef = Suppress("[") + delimitedList( item ) + Suppress("]")
    keyDef = tupleDef | quotedString | imagNum | number
    keyVal = Group( keyDef + Suppress(":") + item )
    dictDef = Suppress("{") + delimitedList( keyVal ) + Suppress("}")

    item << ( quotedString | number | imagNum |
    tupleDef | listDef | dictDef )

    # define low-level conversion routines
    intNum.setParseAction( lambda s,loc,toks: int(toks[0]) )
    realNum.setParseAction( lambda s,loc,toks: float(toks[0]) )
    imagNum.setParseAction( lambda s,loc,toks: eval(toks[0]) ) # no built-in to
    convert imaginaries?

    # strip leading and trailing character from parsed quoted string
    quotedString.setParseAction( lambda s,loc,toks: toks[0][1:-1] )

    # define list-to-list/tuple/dict routines
    evalTuple = lambda s,loc,toks: [ tuple(toks) ]
    evalList = lambda s,loc,toks: [ toks.asList() ]
    evalDict = lambda s,loc,toks: [ dict([tuple(kv) for kv in toks]) ]

    tupleDef.setParseAction( evalTuple )
    listDef.setParseAction( evalList )
    dictDef.setParseAction( evalDict )

    # first element of returned tokens list is the reconstructed list/tuple/dict
    results = item.parseString( testdata )[0]
    print results

    if repr(results) == repr(d1):
    print "Eureka!"
    else:
    print "Compare results for mismatch"
    print repr(results)
    print repr(d1)
     
    Paul McGuire, Jun 25, 2004
    #4
  5. Use YAML

    import yaml

    then from your code:

    yaml.dump( whatever ) =>

    then yaml.loadstring(str)...

    It handles objects.
    {'name': 'the name', 'tuple': ('tuple1', 'tuple2'), 'int': 1, 'float':
    1.5, 'list': ['ol1', 'ol2'], 'long': 12345678901234567890,
    'dict': {'od1': None, 'od2': 2}, 'bool': True, 'password': 'the password'}

    --- !!__main__.simple # instanciates class for you
    bool: True
    dict:
    od1: ~
    od2: 2
    float: 1.5
    int: 1
    list:
    - ol1
    - ol2
    long: 12345678901234567890
    name: the name
    password: the password
    tuple:
    - tuple1
    - tuple2

    It won't roll attributes whose name starts with '_'
    {'name': 'the name', 'tuple': ('tuple1', 'tuple2'), 'int': 1, 'float':
    1.5, 'list': ['ol1', 'ol2'], 'long': 12345678901234567890,
    'dict': {'od1': None, 'od2': 2}, 'bool': True, 'password': 'the password'}

    --- !!__main__.simple
    _recursion: # yaml does not handle recursion (I have an old version)
    - tuple1
    - tuple2
    bool: True
    dict:
    od1: ~
    od2: 2
    float: 1.5
    int: 1
    list:
    - ol1
    - ol2
    long: 12345678901234567890
    name: the name
    password: the password
    tuple:
    - tuple1
    - tuple2

    It will raise an exception if it detects recursion.
    This next one will cause recursion.
    --- !!__main__.simple
    _recursion:
    - tuple1
    - tuple2
    bool: True
    dict:
    od1: ~
    od2: 2
    float: 1.5
    int: 1
    list:
    - ol1
    - ol2
    long: 12345678901234567890
    name: the name
    password: the password
    recursion:
    - tuple1
    - tuple2
    tuple:
    - tuple1
    - tuple2
     
    =?iso-8859-15?Q?Pierre-Fr=E9d=E9ric_Caillaud?=, Jun 25, 2004
    #5
  6. > I want to convert a dict into string form, then back again.

    Since you probably don't want to do this just for the joy of
    converting something to string an back ...

    Maybe the pickle (and cPickle) module does what you are looking for.

    Oliver
     
    Oliver Pieper, Jun 25, 2004
    #6
  7. Kamilche

    Kamilche Guest

    Pierre-Frédéric Caillaud <> wrote in message news:<opr941cav01v4ijd@musicbox>...

    > Use YAML
    >

    It looked interesting, so I downloaded it... and was confronted with
    dozens of files, and the need to compile before use... when I was
    looking for a simple cross-platform 2 function solution that didn't
    take any DLL's. Dang.

    Well, it's a new day, maybe I'll be inspired.
     
    Kamilche, Jun 25, 2004
    #7
  8. did you download syck or the pure python yaml parser ?
    on Linux the pure python module is just a matter of typing "emerge sync"
    but I don't know about Syck...



    On 25 Jun 2004 13:36:31 -0700, Kamilche <> wrote:

    > Pierre-Frédéric Caillaud <> wrote in message
    > news:<opr941cav01v4ijd@musicbox>...
    >
    >> Use YAML
    >>

    > It looked interesting, so I downloaded it... and was confronted with
    > dozens of files, and the need to compile before use... when I was
    > looking for a simple cross-platform 2 function solution that didn't
    > take any DLL's. Dang.
    >
    > Well, it's a new day, maybe I'll be inspired.




    --
    Using Opera's revolutionary e-mail client: http://www.opera.com/m2/
     
    =?iso-8859-15?Q?Pierre-Fr=E9d=E9ric_Caillaud?=, Jun 25, 2004
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. PT
    Replies:
    0
    Views:
    4,200
  2. Brett Sheeran

    Log4j Not Rolling Files

    Brett Sheeran, Feb 9, 2004, in forum: Java
    Replies:
    1
    Views:
    7,429
    Oscar kind
    Feb 23, 2004
  3. Dharmveer Jain
    Replies:
    0
    Views:
    497
    Dharmveer Jain
    Sep 14, 2004
  4. Replies:
    6
    Views:
    188
    Michael Campbell
    Mar 17, 2005
  5. Matt Brooks

    Rolling String Buffer needed

    Matt Brooks, Oct 2, 2009, in forum: Ruby
    Replies:
    5
    Views:
    196
    jimmy Zhang
    Oct 3, 2009
Loading...

Share This Page