How to do parallel programming in Python
For C++, we can use OpenMP to do parallel programming; however, OpenMP will not work for Python. What should I do if I want to parallel some parts of my python program?
The structure of the code may be considered as:
solve2 are two independent function. How to run this kind of code in parallel instead of in sequence in order to reduce the running time? Hope someone can help me. Thanks very much in advance. The code is:
def solve(Q, G, n): i = 0 tol = 10 ** -4 while i < 1000: inneropt, partition, x = setinner(Q, G, n) outeropt = setouter(Q, G, n) if (outeropt - inneropt) / (1 + abs(outeropt) + abs(inneropt)) < tol: break node1 = partition node2 = partition G = updateGraph(G, node1, node2) if i == 999: print "Maximum iteration reaches" print inneropt
Where setinner and setouter are two independent functions. That's where I want to parallel...
You can use the multiprocessing module. For this case I might use a processing pool:
from multiprocessing import Pool pool = Pool() result1 = pool.apply_async(solve1, [A]) # evaluate "solve1(A)" asynchronously result2 = pool.apply_async(solve2, [B]) # evaluate "solve2(B)" asynchronously answer1 = result1.get(timeout=10) answer2 = result2.get(timeout=10)
This will spawn processes that can do generic work for you. Since we did not pass
processes, it will spawn one process for each CPU core on your machine. Each CPU core can execute one process simultaneously.
If you want to map a list to a single function you would do this:
args = [A, B] results = pool.map(solve1, args)
Don't use threads because the GIL locks any operations on python objects.