标准库为我们提供了concurrent.futures模块,它提供了ThreadPoolExecutor和ProcessPoolExecutor两个类,
实现了对threading和multiprocessing的进一步抽象,对编写线程池/进程池提供了直接的支持。
注:pool.submit(func,*args,**kwargs),关于入参:
func: 表示调用的自定义函数名
*args: 表示该自定义函数名需要的入参:元组、单个参数形式,
如pool.submit(func,arg1,arg2,arg3=3)
表示该自定义函数的传参形式为:func(arg1,arg2,arg3=3)
**kwargs:表示该自定义函数名需要的入参:字典参数形式
举例1:ProcessPoolExecutor
from concurrent.futures import ProcessPoolExecutor
import time
def task(name):
print("name:", name)
time.sleep(2)
if __name__ == "__main__":
time1 = time.time()
with ProcessPoolExecutor(5) as pool:
future_result1 = pool.submit(task, "name1")
future_result2 = pool.submit(task, "name2")
future_result3 = pool.submit(task, "name3")
#pool.shutdown(wait=True)
time2 = time.time()
print("time2 - time1", time2 - time1)
"""
结果:由结果看出,程序执行结果并不是累计等待2+2+2=6秒,而是并发执行
name: name1
name: name2
name: name3
time2 - time1 2.439988851547241
解释:
ProcessPoolExecutor(5)创建一个进程池,容量为5,并一共创建三个进程(future_result1,future_result2,future_result3),
然后再线程池队列里并行执行该三个进程.
且pool.shutdown(wait=True)表示创建的三个进程都执行完毕才执行该程序的主进程,这里使用with语句默认调用了pool.shutdown(wait=True),因此可以省略;
且pool.shutdown(wait=True)的wait参数值默认即为True
实际断言方法并行时用到如下方式:
with ProcessPoolExecutor(5) as pool:
future_result1 = pool.submit(task, "name1") #但是submit提交时是异步的,这样就不用等待提交得到结果再去下一行提交代码
future_result2 = pool.submit(task, "name2")
future_result3 = pool.submit(task, "name3")
future_result1.result() #同步调用
future_result2.result() #同步调用
future_result3.result() #同步调用
"""
举例2:
from concurrent.futures import ProcessPoolExecutor
import time
def task(name):
print("name:", name)
time.sleep(2)
if __name__ == "__main__":
time1 = time.time()
pool = ProcessPoolExecutor(5)
for i in range(3):
future_result1 = pool.submit(task, "name{}".format(i))
pool.shutdown(wait=True)
print("main")
time2 = time.time()
print("time2 - time1", time2 - time1)
"""
结果:由结果看出,程序执行结果并不是累计等待2+2+2=6秒,而是并发执行
且pool.shutdown(wait=True)的参数为True,表示先执行完创建的子进程,再执行主进程 main
name: name0
name: name1
name: name2
main
time2 - time1 2.2973177433013916
pool.shutdown(wait=False)的结果:
main
time2 - time1 0.26048994064331055
name: name0
name: name1
name: name2
"""
举例3:使用submit同步调用(实际接口测试中使用过)
同步调用:提交/调用一个任务,然后就在原地等着,等到该任务执行完毕拿到结果,再执行下一行代码
from concurrent.futures import ProcessPoolExecutor
import time
import os
def task(name):
print("{}子进程{}:".format(name,os.getpid()))
time.sleep(2)
return name+str(2)
if __name__ == "__main__":
time1 = time.time()
pool = ProcessPoolExecutor(5)
results = []
for i in range(3):
future_result = pool.submit(task, "name{}".format(i))
res=future_result.result() #同步调用,等到结果后再执行下一行代码
print(res)
print("主进程{}".format(os.getpid()))
time2 = time.time()
print("time2 - time1", time2 - time1)
"""
同步调用结果:由结果得知,每个进程得出了结果才去执行下一行代码
name0子进程37512:
name02
name1子进程35884:
name12
name2子进程33600:
name22
主进程36168
time2 - time1 6.2372283935546875
"""
举例4:使用submit异步调用
异步调用: 提交/调用一个任务,不在原地等着,直接执行下一行代码
from concurrent.futures import ProcessPoolExecutor
import time
import os
def task(name):
print("{}子进程{}:".format(name,os.getpid()))
time.sleep(2)
return name+str(2)
if __name__ == "__main__":
time1 = time.time()
pool = ProcessPoolExecutor(5)
results = []
for i in range(3):
future_result = pool.submit(task, "name{}".format(i))
results.append(future_result) #异步调用,不用等到结果,直接执行下一行代码
for result in results:
print(result.result())
print("主进程{}".format(os.getpid()))
time2 = time.time()
print("time2 - time1", time2 - time1)
"""
异步调用结果:由结果得治,每个进程执行完成之后,并未等待结果,而是直接执行下一行代码
name0子进程36116:
name1子进程31972:
name2子进程660:
name02
name12
name22
主进程42280
time2 - time1 2.342352867126465
"""
网友评论