Sharing Data in Python

R

raunakgup90

I have some Pickled data, which is stored on disk, and it is about 100 MB in size.

When my python program is executed, the picked data is loaded using the cPickle module, and all that works fine.

If I execute the python multiple times using python main.py for example, each python process will load the same data multiple times, which is the correct behaviour.

How can I make it so, all new python process share this data, so it is only loaded a single time into memory?

asked the same question on SO, but could not get any constructive responses.. http://stackoverflow.com/questions/10550870/sharing-data-in-python/10551845
 
M

Miki Tebeka

How can I make it so, all new python process share this data, so it is only loaded a single time into memory?
You can have one process as server and client ask for parts of data.
You might be able to do something smart with mmap but I can't think of a way.

I Linux systems, if you first load the data and then fork, the OS will keep all the read only data shared.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,755
Messages
2,569,535
Members
45,007
Latest member
obedient dusk

Latest Threads

Top