Memory error while saving dictionary of size 65000X50 using pickle

[Nagu]

I am trying to save a dictionary of size 65000X50 to a local file and
I get the memory error problem.

How do I go about resolving this? Is there way to partition the pickle
object and combine later if this is a problem due to limited resources
(memory) on the machine (it is 32 bit machine Win XP, with 4GB RAM).


Here is the detail description of the error:

Traceback (most recent call last):
File "<pyshell#12>", line 1, in <module>
s = pickle.dumps(itemsim)
File "C:\Python25\lib\pickle.py", line 1366, in dumps
Pickler(file, protocol).dump(obj)
File "C:\Python25\lib\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python25\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python25\lib\pickle.py", line 663, in _batch_setitems
save(v)
File "C:\Python25\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\lib\pickle.py", line 600, in save_list
self._batch_appends(iter(obj))
File "C:\Python25\lib\pickle.py", line 615, in _batch_appends
save(x)
File "C:\Python25\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\lib\pickle.py", line 562, in save_tuple
save(element)
File "C:\Python25\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\lib\pickle.py", line 477, in save_float
self.write(FLOAT + repr(obj) + '\n')
MemoryError: out of memory

 

[Martin v. Löwis]

I am trying to save a dictionary of size 65000X50 to a local file and
I get the memory error problem.

What do you mean by this size specification? When I interpreter X as
multiplication, I can't see a problem: the code

import pickle

d = {}

for i in xrange(65000*50):
d=i
print "Starting dump"
s = pickle.dumps(d)

works just fine for me. Can you please modify it so that it does cause
a problem?

Regards,
Martin