pickling multiple dictionaries

M

manstey

Hi,

I am running a script that produces about 450,000 dictionaries. I tried
putting them into a tuple and then pickling the tuple, but the tuple
gets too big. Can I pickle dictionaries one after another into the same
file and then read them out again?

Cheers,
Matthew
 
H

Hans Georg Krauthaeuser

manstey said:
Hi,

I am running a script that produces about 450,000 dictionaries. I tried
putting them into a tuple and then pickling the tuple, but the tuple
gets too big. Can I pickle dictionaries one after another into the same
file and then read them out again?

Cheers,
Matthew
If you don't know, just try it:

In [1]:import pickle
In [2]:d1={'a':1}
In [3]:d2={'b':2}
In [4]:pfile=file('test.p','wb')
In [5]:pickle.dump(d1,pfile)
In [6]:pickle.dump(d2,pfile)
In [7]:pfile.close()
In [8]:del d1
In [9]:del d2
In [10]:pfile=file('test.p','rb')
In [11]:d1=pickle.load(pfile)
In [12]:d1
Out[12]:{'a': 1}
In [13]:d2=pickle.load(pfile)
In [14]:d2
Out[14]:{'b': 2}

If your data is *really* large, have a look to PyTables
(http://www.pytables.org/moin).

Regards,
Hans Georg
 
M

manstey

Thanks very much. How large is *really* large for making pytables
worthwhile. Our python script generates an xml file of about 450Mb. Is
pytables worth using then?
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top