RE: [python-list] python application file format

Discussion in 'Python' started by Prasad, Ramit, Sep 28, 2012.

  1. Benjamin Jessup wrote:

    > Hello all,
    >
    > What do people recommend for a file format for a python desktop
    > application? Data is complex with 100s/1000s of class instances, which
    > reference each other.
    >
    > Write the file with struct module? (Rebuild object pointers, safe,
    > compact, portable, not expandable without reserved space)
    >
    > Use cPickle with a module/class whitelist? (Can't easily port, not
    > entirely safe, compact enough, expandable)
    >
    > Use JSON or similar? (Rebuild object pointers, portable, expandable, size?)
    >
    > Any advice is greatly appreciated!


    I would thinkyour options are pickle, json or database (either sqlite or
    something like Postgres). I am unfamiliar with the struct module so I
    cannot comment on its applicability.

    I would guess that your data would be best saved by using a sqlite
    database. Your biggest problemmight be how the different classes are
    referencing each other. If you are using identifiers then any of these
    options will probably work. If you are using aggregation then I know
    that pickle will work (at least for built-in types). JSON will
    keep the structure but duplicate elements.



    >>> a = [ 1,2,3 ]
    >>> b = [ 'a', 'b', 'c' ]
    >>> a.append( b )
    >>> e = [ a,b ]
    >>> s = json.dumps( e )
    >>> eret = json.loads( s )
    >>> id(eret[0][3]), id(eret[1]) # Same result for json and simplejson

    (329443808, 327677272)

    >>> eret[0][3].append( 'o')
    >>> eret[0][3], eret[1]

    ([u'a', u'b', u'c', 'o'], [u'a', u'b', u'c'])

    So pickle will be your easiest option, but I am not sure how well it
    will scale with a large number items. Using sqlite/db should scale well
    but it will take you longer/more effort to create a system for converting
    your objects to and from the DB.



    This email is confidential and subject to important disclaimers and
    conditions including on offers for the purchase or sale of
    securities, accuracy and completenessof information, viruses,
    confidentiality, legal privilege, and legal entity disclaimers,
    available at http://www.jpmorgan.com/pages/disclosures/email.
     
    Prasad, Ramit, Sep 28, 2012
    #1
    1. Advertising

  2. On Saturday, 29 September 2012 02:05:07 UTC+5:30, Prasad, Ramit wrote:
    > Benjamin Jessup wrote:
    >
    > > Hello all,

    >
    > >

    >
    > > What do people recommend for a file format for a python desktop

    >
    > > application? Data is complex with 100s/1000s of class instances, which

    >
    > > reference each other.

    >
    > >

    >
    > > Write the file with struct module? (Rebuild object pointers, safe,

    >
    > > compact, portable, not expandable without reserved space)

    >
    > >

    >
    > > Use cPickle with a module/class whitelist? (Can't easily port, not

    >
    > > entirely safe, compact enough, expandable)

    >
    > >

    >
    > > Use JSON or similar? (Rebuild object pointers, portable, expandable, size?)

    >
    > >

    >
    > > Any advice is greatly appreciated!

    >
    >
    >
    > I would think your options are pickle, json or database (either sqlite or
    >
    > something like Postgres). I am unfamiliar with the struct module so I
    >
    > cannot comment on its applicability.
    >
    >
    >
    > I would guess that your data would be best saved by using a sqlite
    >
    > database. Your biggest problem might be how the different classes are
    >
    > referencing each other. If you are using identifiers then any of these
    >
    > options will probably work. If you are using aggregation then I know
    >
    > that pickle will work (at least for built-in types). JSON will
    >
    > keep the structure but duplicate elements.
    >
    >
    >
    >
    >
    > >>> a = [ 1,2,3 ]

    >
    > >>> b = [ 'a', 'b', 'c' ]

    >
    > >>> a.append( b )

    >
    > >>> e = [ a,b ]

    >
    > >>> s = json.dumps( e )

    >
    > >>> eret = json.loads( s )

    >
    > >>> id(eret[0][3]), id(eret[1]) # Same result for json and simplejson

    >
    > (329443808, 327677272)
    >
    > >>> eret[0][3].append( 'o')

    >
    > >>> eret[0][3], eret[1]

    >
    > ([u'a', u'b', u'c', 'o'], [u'a', u'b', u'c'])
    >
    >
    >
    > So pickle will be your easiest option, but I am not sure how well it
    >
    > will scale with a large number items. Using sqlite/db should scale well
    >
    > but it will take you longer/more effort to create a system for converting
    >
    > your objects to and from the DB.
    >
    >
    >
    >
    >
    >
    >
    > This email is confidential and subject to important disclaimers and
    >
    > conditions including on offers for the purchase or sale of
    >
    > securities, accuracy and completeness of information, viruses,
    >
    > confidentiality, legal privilege, and legal entity disclaimers,
    >
    > available at http://www.jpmorgan.com/pages/disclosures/email.


    Guess I shouldn't read or rely your posts (virus'es could be in them) :-D
     
    Ramchandra Apte, Sep 29, 2012
    #2
    1. Advertising

  3. On Saturday, 29 September 2012 02:05:07 UTC+5:30, Prasad, Ramit wrote:
    > Benjamin Jessup wrote:
    >
    > > Hello all,

    >
    > >

    >
    > > What do people recommend for a file format for a python desktop

    >
    > > application? Data is complex with 100s/1000s of class instances, which

    >
    > > reference each other.

    >
    > >

    >
    > > Write the file with struct module? (Rebuild object pointers, safe,

    >
    > > compact, portable, not expandable without reserved space)

    >
    > >

    >
    > > Use cPickle with a module/class whitelist? (Can't easily port, not

    >
    > > entirely safe, compact enough, expandable)

    >
    > >

    >
    > > Use JSON or similar? (Rebuild object pointers, portable, expandable, size?)

    >
    > >

    >
    > > Any advice is greatly appreciated!

    >
    >
    >
    > I would think your options are pickle, json or database (either sqlite or
    >
    > something like Postgres). I am unfamiliar with the struct module so I
    >
    > cannot comment on its applicability.
    >
    >
    >
    > I would guess that your data would be best saved by using a sqlite
    >
    > database. Your biggest problem might be how the different classes are
    >
    > referencing each other. If you are using identifiers then any of these
    >
    > options will probably work. If you are using aggregation then I know
    >
    > that pickle will work (at least for built-in types). JSON will
    >
    > keep the structure but duplicate elements.
    >
    >
    >
    >
    >
    > >>> a = [ 1,2,3 ]

    >
    > >>> b = [ 'a', 'b', 'c' ]

    >
    > >>> a.append( b )

    >
    > >>> e = [ a,b ]

    >
    > >>> s = json.dumps( e )

    >
    > >>> eret = json.loads( s )

    >
    > >>> id(eret[0][3]), id(eret[1]) # Same result for json and simplejson

    >
    > (329443808, 327677272)
    >
    > >>> eret[0][3].append( 'o')

    >
    > >>> eret[0][3], eret[1]

    >
    > ([u'a', u'b', u'c', 'o'], [u'a', u'b', u'c'])
    >
    >
    >
    > So pickle will be your easiest option, but I am not sure how well it
    >
    > will scale with a large number items. Using sqlite/db should scale well
    >
    > but it will take you longer/more effort to create a system for converting
    >
    > your objects to and from the DB.
    >
    >
    >
    >
    >
    >
    >
    > This email is confidential and subject to important disclaimers and
    >
    > conditions including on offers for the purchase or sale of
    >
    > securities, accuracy and completeness of information, viruses,
    >
    > confidentiality, legal privilege, and legal entity disclaimers,
    >
    > available at http://www.jpmorgan.com/pages/disclosures/email.


    Guess I shouldn't read or rely your posts (virus'es could be in them) :-D
     
    Ramchandra Apte, Sep 29, 2012
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    5
    Views:
    729
    Magnus Lycka
    Oct 27, 2006
  2. Replies:
    1
    Views:
    810
    mlimber
    Sep 11, 2006
  3. Randy Kramer
    Replies:
    2
    Views:
    406
    Randy Kramer
    Jan 12, 2007
  4. Benjamin Jessup

    [python-list] python application file format

    Benjamin Jessup, Sep 26, 2012, in forum: Python
    Replies:
    2
    Views:
    224
    Ramchandra Apte
    Sep 26, 2012
  5. Wesley
    Replies:
    1
    Views:
    160
    Wesley
    Apr 15, 2014
Loading...

Share This Page