python 多进程加速for循环示例,包括如何传参。
对于for循环而言,可以将整个长度进行切分成如上的args_list,再对各个Process传递不同的index值即可。
·
# SuperFastPython.com
# execute tasks in parallel in a for loop
from time import sleep
from random import random
from multiprocessing import Process
# execute a task
def task(arg, i, string_):
print("arg=" + str(arg) + " i=" + str(i) + "string_" + string_)
# generate a random value between 0 and 1
value = random()
# block for a fraction of a second
sleep(value + arg[0])
print(sum(arg))
# report a message
print(f'.done {arg[0]}, generated {value}', flush=True)
# protect the entry point
if __name__ == '__main__':
pool_num = 3
args_list = [[1,2,3,4], [5,6,7,8], [9,10,11,12]]
string_ = "res"
processes = [Process(target=task, args=(args_list[i], i, string_)) for i in range(pool_num)]
# start all processes
for process in processes:
process.start()
# wait for all processes to complete
for process in processes:
process.join()
# report that all tasks are completed
print('Done', flush=True)
过程就是定义几个Process,每个Process执行不同的task。task的区别在于for循环中不同的索引。另外各个Process如果输出文件,那么应写到不同的文件里避免不必要的麻烦。
args_list的由来:对于for循环而言,可以将整个长度进行切分成如上的args_list,再对各个Process传递不同的index值即可。如下所示:
split_size = int(len(accx_cuts) / pool_num)
print("allocate indexes")
args_list = []
for i in range(pool_num):
data_list = [i * split_size + k for k in range(split_size)]
print(i, data_list[0:5], data_list[-5:])
args_list.append(data_list)
processes = [Process(target=task, args=(args_list[i], i, accx_cuts, accy_cuts, accz_cuts)) for i in range(pool_num)]
更多推荐
所有评论(0)