Share Large, Read-Only Numpy Array Between Multiprocessing Processes
I have a 60GB SciPy Array (Matrix) I must share between 5+
Process objects. I've seen numpy-sharedmem and read this discussion on the SciPy list. There seem to be two approaches--
numpy-sharedmem and using a
multiprocessing.RawArray() and mapping NumPy
numpy-sharedmem seems to be the way to go, but I've yet to see a good reference example. I don't need any kind of locks, since the array (actually a matrix) will be read-only. Now, due to its size, I'd like to avoid a copy. It sounds like the correct method is to create the only copy of the array as a
sharedmem array, and then pass it to the
Process objects? A couple of specific questions:
What's the best way to actually pass the sharedmem handles to sub-
Process()es? Do I need a queue just to pass one array around? Would a pipe be better? Can I just pass it as an argument to the
Process()subclass's init (where I'm assuming it's pickled)?
In the discussion I linked above, there's mention of
numpy-sharedmemnot being 64bit-safe? I'm definitely using some structures that aren't 32-bit addressable.
Are there tradeoff's to the
RawArray()approach? Slower, buggier?
Do I need any ctype-to-dtype mapping for the numpy-sharedmem method?
Does anyone have an example of some OpenSource code doing this? I'm a very hands-on learned and it's hard to get this working without any kind of good example to look at.
If there's any additional info I can provide to help clarify this for others, please comment and I'll add. Thanks!
This needs to run on Ubuntu Linux and Maybe Mac OS, but portability isn't a huge concern.
@Velimir Mlaker gave a great answer. I thought I could add some bits of comments and a tiny example.
(I couldn't find much documentation on sharedmem - these are the results of my own experiments.)
- Do you need to pass the handles when the subprocess is starting, or after it has started? If it's just the former, you can just use the
Process. This is potentially better than using a global variable.
- From the discussion page you linked, it appears that support for 64-bit Linux was added to sharedmem a while back, so it could be a non-issue.
- I don't know about this one.
- No. Refer to example below.
#!/usr/bin/env python from multiprocessing import Process import sharedmem import numpy def do_work(data, start): data[start] = 0; def split_work(num): n = 20 width = n/num shared = sharedmem.empty(n) shared[:] = numpy.random.rand(1, n) print "values are %s" % shared processes = [Process(target=do_work, args=(shared, i*width)) for i in xrange(num)] for p in processes: p.start() for p in processes: p.join() print "values are %s" % shared print "type is %s" % type(shared) if __name__ == '__main__': split_work(4)
values are [ 0.81397784 0.59667692 0.10761908 0.6736734 0.46349645 0.98340718 0.44056863 0.10701816 0.67167752 0.29158274 0.22242552 0.14273156 0.34912309 0.43812636 0.58484507 0.81697513 0.57758441 0.4284959 0.7292129 0.06063283] values are [ 0. 0.59667692 0.10761908 0.6736734 0.46349645 0. 0.44056863 0.10701816 0.67167752 0.29158274 0. 0.14273156 0.34912309 0.43812636 0.58484507 0. 0.57758441 0.4284959 0.7292129 0.06063283] type is <type 'numpy.float64'>
This related question might be useful.
★ Back to homepage or read more recommendations: