shelf-like list?

Discussion in 'Python' started by kj, Aug 10, 2010.

  1. kj

    kj Guest

    I'm looking for a module that implements "persistent lists": objects
    that behave like lists except that all their elements are stored
    on disk. IOW, the equivalent of "shelves", but for lists rather
    than a dictionaries.

    Does anyone know of such a module?

    (I suppose that I could slap together a crude implementation of
    such a thing by wrapping a shelf with suitable methods to simulate
    the list interface. But I'd rather not roll my own if a tested
    implementation already exist.)

    TIA!

    ~K
    kj, Aug 10, 2010
    #1
    1. Advertising

  2. On Tuesday 10 August 2010, it occurred to kj to exclaim:
    > I'm looking for a module that implements "persistent lists": objects
    > that behave like lists except that all their elements are stored
    > on disk. IOW, the equivalent of "shelves", but for lists rather
    > than a dictionaries.
    >
    > Does anyone know of such a module?
    >
    > (I suppose that I could slap together a crude implementation of
    > such a thing by wrapping a shelf with suitable methods to simulate
    > the list interface. But I'd rather not roll my own if a tested
    > implementation already exist.)


    You could simply use pickle to save the data every once in a while.
    Thomas Jollans, Aug 12, 2010
    #2
    1. Advertising

  3. On Aug 12, 1:37 pm, Thomas Jollans <> wrote:
    > On Tuesday 10 August 2010, it occurred to kj to exclaim:
    >
    > > I'm looking for a module that implements "persistent lists": objects
    > > that behave like lists except that all their elements are stored
    > > on disk.  IOW, the equivalent of "shelves", but for lists rather
    > > than a dictionaries.

    . . .
    > You could simply use pickle to save the data every once in a while.


    That is a very reasonable solution.
    Raymond Hettinger, Aug 13, 2010
    #3
  4. kj

    kj Guest

    In <> Raymond Hettinger <> writes:

    >On Aug 12, 1:37=A0pm, Thomas Jollans <> wrote:
    >> On Tuesday 10 August 2010, it occurred to kj to exclaim:
    >>
    >> > I'm looking for a module that implements "persistent lists": objects
    >> > that behave like lists except that all their elements are stored
    >> > on disk. =A0IOW, the equivalent of "shelves", but for lists rather
    >> > than a dictionaries.

    > . . .
    >> You could simply use pickle to save the data every once in a while.


    >That is a very reasonable solution.


    Sorry I don't follow. Some sample code would be helpful.
    TIA,
    ~K
    kj, Aug 15, 2010
    #4
  5. kj

    Chris Rebert Guest

    On Sat, Aug 14, 2010 at 5:13 PM, kj <> wrote:
    > In <> Raymond Hettinger <> writes:
    >>On Aug 12, 1:37=A0pm, Thomas Jollans <> wrote:
    >>> On Tuesday 10 August 2010, it occurred to kj to exclaim:
    >>>
    >>> > I'm looking for a module that implements "persistent lists": objects
    >>> > that behave like lists except that all their elements are stored
    >>> > on disk. =A0IOW, the equivalent of "shelves", but for lists rather
    >>> > than a dictionaries.

    >> . . .
    >>> You could simply use pickle to save the data every once in a while.

    >
    >>That is a very reasonable solution.

    >
    > Sorry I don't follow.  Some sample code would be helpful.


    I would assume something along the lines of (untested):

    from pickle import dump

    MOD_THRESHOLD = 42

    class PersistentList(list):
    def __init__(self, filepath):
    self.filepath = filepath
    self._mod_count = 0

    def save(self):
    with open(self.filepath, 'w') as f:
    dump(self, f)
    self._mod_count = 0

    def append(self, *args, **kwds):
    super(PersistentList, self).append(*args, **kwds)
    self._mod_count += 1
    if self._mod_count >= MOD_THRESHOLD:
    # obviously time-since-last-dump or other
    # more sophisticated metrics might be used instead
    self.save()
    # define similar wrappers for list's other mutator methods:
    __delitem__, __iadd__, __imul__, __setitem__, extend, insert, pop,
    remove, etc.
    # using decorators should help decrease code duplication

    Cheers,
    Chris
    --
    http://blog.rebertia.com
    Chris Rebert, Aug 15, 2010
    #5
  6. > Does anyone know of such a module?

    ZODB supports persistent lists.

    Regards,
    Martin
    Martin v. Loewis, Aug 15, 2010
    #6
  7. kj

    kj Guest

    In <> Chris Rebert <> writes:

    >On Sat, Aug 14, 2010 at 5:13 PM, kj <> wrote:
    >> In <> Ra=

    >ymond Hettinger <> writes:
    >>>On Aug 12, 1:37=3DA0pm, Thomas Jollans <> wrote:
    >>>> On Tuesday 10 August 2010, it occurred to kj to exclaim:
    >>>>
    >>>> > I'm looking for a module that implements "persistent lists": objects
    >>>> > that behave like lists except that all their elements are stored
    >>>> > on disk. =3DA0IOW, the equivalent of "shelves", but for lists rather
    >>>> > than a dictionaries.
    >>> . . .
    >>>> You could simply use pickle to save the data every once in a while.

    >>
    >>>That is a very reasonable solution.

    >>
    >> Sorry I don't follow. =C2=A0Some sample code would be helpful.


    >I would assume something along the lines of (untested):


    >from pickle import dump


    >MOD_THRESHOLD =3D 42


    >class PersistentList(list):
    > def __init__(self, filepath):
    > self.filepath =3D filepath
    > self._mod_count =3D 0


    > def save(self):
    > with open(self.filepath, 'w') as f:
    > dump(self, f)
    > self._mod_count =3D 0


    > def append(self, *args, **kwds):
    > super(PersistentList, self).append(*args, **kwds)
    > self._mod_count +=3D 1
    > if self._mod_count >=3D MOD_THRESHOLD:
    > # obviously time-since-last-dump or other
    > # more sophisticated metrics might be used instead
    > self.save()


    Even though it is saved periodically to disk, it looks like the
    whole list remains in memory all the time? (If so, it's not what
    I'm looking for; the whole point of saving stuff to disk is to keep
    the list's memory footprint low.)

    ~K
    kj, Aug 15, 2010
    #7
  8. kj

    kj Guest

    In <i486al$bk$> "Martin v. Loewis" <> writes:

    >> Does anyone know of such a module?


    >ZODB supports persistent lists.


    Thanks; I'll check it out.

    ~K
    kj, Aug 15, 2010
    #8
  9. kj

    Dave Angel Guest

    kj wrote:
    > <snip>
    >> self.save()
    >>

    >
    > Even though it is saved periodically to disk, it looks like the
    > whole list remains in memory all the time? (If so, it's not what
    > I'm looking for; the whole point of saving stuff to disk is to keep
    > the list's memory footprint low.)
    >
    > ~K
    >
    >

    It sounds like we all made the wrong assumption about your
    requirements. It's not persistence you want, but low memory footprint
    ? Do you or don't you also want the data to survive multiple runs of
    the program?

    If you want the data to be in a persistent disk file, you probably
    should use a database. And if you don't, then you could either increase
    the size of your swapfile, or use a more complex scheme to do a memory
    mapped file while the program is running, and delete it on exit. The
    swapfile is likely to be execute more quickly than anything else you
    could cook up in pure python.

    DaveA
    Dave Angel, Aug 15, 2010
    #9
  10. On 8/15/2010 3:58 AM kj said...
    > In<i486al$bk$> "Martin v. Loewis"<> writes:
    >
    >>> Does anyone know of such a module?

    >
    >> ZODB supports persistent lists.

    >
    > Thanks; I'll check it out.
    >


    I wouldn't expect a low memory footprint however. :)

    Emile
    Emile van Sebille, Aug 15, 2010
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. xz
    Replies:
    1
    Views:
    364
    Mark Newmister
    Apr 7, 2006
  2. Print Guy

    another Book on the shelf

    Print Guy, Aug 19, 2006, in forum: Java
    Replies:
    2
    Views:
    462
    Mindundi
    Aug 20, 2006
  3. Aaron Brady

    shelf membership

    Aaron Brady, Mar 31, 2007, in forum: Python
    Replies:
    3
    Views:
    240
    Peter Otten
    Apr 1, 2007
  4. dE|_
    Replies:
    0
    Views:
    373
  5. Sera Jackson

    I love "shelf" BUT

    Sera Jackson, Jul 28, 2008, in forum: Python
    Replies:
    3
    Views:
    265
    alex23
    Jul 28, 2008
Loading...

Share This Page