Memory error while saving dictionary using pickle

Discussion in 'Python' started by Nagu, Jul 7, 2008.

  1. Nagu

    Nagu Guest

    I am trying to save a dictionary of size 65000X50 to a local file and
    I get the memory error problem.

    How do I go about resolving this? Is there way to partition the pickle
    object and combine later if this is a problem due to limited resources
    (memory) on the machine (it is 32 bit machine Win XP, with 4GB RAM).

    Please advice.

    Thank you,
    Nagu

    Here is the detail description of the error:

    Traceback (most recent call last):
    File "<pyshell#12>", line 1, in <module>
    s = pickle.dumps(itemsim)
    File "C:\Python25\lib\pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
    File "C:\Python25\lib\pickle.py", line 224, in dump
    self.save(obj)
    File "C:\Python25\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    File "C:\Python25\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
    File "C:\Python25\lib\pickle.py", line 663, in _batch_setitems
    save(v)
    File "C:\Python25\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    File "C:\Python25\lib\pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
    File "C:\Python25\lib\pickle.py", line 615, in _batch_appends
    save(x)
    File "C:\Python25\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    File "C:\Python25\lib\pickle.py", line 562, in save_tuple
    save(element)
    File "C:\Python25\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    File "C:\Python25\lib\pickle.py", line 477, in save_float
    self.write(FLOAT + repr(obj) + '\n')
    MemoryError: out of memory
     
    Nagu, Jul 7, 2008
    #1
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.