jeanmichel at sequans
May 15, 2012, 2:36 AM
Post #3 of 4
raunakgup90 [at] gmail wrote:
> I have some Pickled data, which is stored on disk, and it is about 100 MB in size.
> When my python program is executed, the picked data is loaded using the cPickle module, and all that works fine.
> If I execute the python multiple times using python main.py for example, each python process will load the same data multiple times, which is the correct behaviour.
> How can I make it so, all new python process share this data, so it is only loaded a single time into memory?
> asked the same question on SO, but could not get any constructive responses.. http://stackoverflow.com/questions/10550870/sharing-data-in-python/10551845
Well a straightforward way of solving this is to use threads.
Have your main program spawn threads for processing data.
Now what triggers a thread is up to you, your main program can listen to
commands, or catch keyboard events ...
If you want to make sure your using all CPU cores, you may want to use
It emulates threads with subprocesses, and features a way to share memory.
I do tend to prefer processes over threads, I find them easier to
monitor and control.