美文网首页
并发编程之concurrent.futures模块

并发编程之concurrent.futures模块

作者: Chaweys | 来源:发表于2021-12-29 01:49 被阅读0次

标准库为我们提供了concurrent.futures模块,它提供了ThreadPoolExecutor和ProcessPoolExecutor两个类,
实现了对threading和multiprocessing的进一步抽象,对编写线程池/进程池提供了直接的支持。
注:pool.submit(func,*args,**kwargs),关于入参:
func:    表示调用的自定义函数名
*args:   表示该自定义函数名需要的入参:元组、单个参数形式,
          如pool.submit(func,arg1,arg2,arg3=3)
          表示该自定义函数的传参形式为:func(arg1,arg2,arg3=3)
          
**kwargs:表示该自定义函数名需要的入参:字典参数形式


举例1:ProcessPoolExecutor
from concurrent.futures import ProcessPoolExecutor
import time

def task(name):
    print("name:", name)
    time.sleep(2)

if __name__ == "__main__":
    time1 = time.time()
    with ProcessPoolExecutor(5) as pool:
        future_result1 = pool.submit(task, "name1")
        future_result2 = pool.submit(task, "name2")
        future_result3 = pool.submit(task, "name3")

        #pool.shutdown(wait=True)

    time2 = time.time()
    print("time2 - time1", time2 - time1)
"""
结果:由结果看出,程序执行结果并不是累计等待2+2+2=6秒,而是并发执行
name: name1
name: name2
name: name3
time2 - time1 2.439988851547241

解释:
ProcessPoolExecutor(5)创建一个进程池,容量为5,并一共创建三个进程(future_result1,future_result2,future_result3),
然后再线程池队列里并行执行该三个进程.
且pool.shutdown(wait=True)表示创建的三个进程都执行完毕才执行该程序的主进程,这里使用with语句默认调用了pool.shutdown(wait=True),因此可以省略;
且pool.shutdown(wait=True)的wait参数值默认即为True


实际断言方法并行时用到如下方式:
with ProcessPoolExecutor(5) as pool:
    future_result1 = pool.submit(task, "name1")  #但是submit提交时是异步的,这样就不用等待提交得到结果再去下一行提交代码
    future_result2 = pool.submit(task, "name2")
    future_result3 = pool.submit(task, "name3")
    
    future_result1.result() #同步调用
    future_result2.result() #同步调用
    future_result3.result() #同步调用

"""


举例2:
from concurrent.futures import ProcessPoolExecutor
import time

def task(name):
    print("name:", name)
    time.sleep(2)

if __name__ == "__main__":
    time1 = time.time()
    pool = ProcessPoolExecutor(5)

    for i in range(3):
        future_result1 = pool.submit(task, "name{}".format(i))
    pool.shutdown(wait=True)

    print("main")
    time2 = time.time()
    print("time2 - time1", time2 - time1)
"""
结果:由结果看出,程序执行结果并不是累计等待2+2+2=6秒,而是并发执行
且pool.shutdown(wait=True)的参数为True,表示先执行完创建的子进程,再执行主进程 main
name: name0
name: name1
name: name2
main
time2 - time1 2.2973177433013916

    
pool.shutdown(wait=False)的结果:
main
time2 - time1 0.26048994064331055
name: name0
name: name1
name: name2
"""



举例3:使用submit同步调用(实际接口测试中使用过)
同步调用:提交/调用一个任务,然后就在原地等着,等到该任务执行完毕拿到结果,再执行下一行代码
from concurrent.futures import ProcessPoolExecutor
import time
import os

def task(name):
    print("{}子进程{}:".format(name,os.getpid()))
    time.sleep(2)
    return name+str(2)

if __name__ == "__main__":
    time1 = time.time()
    pool = ProcessPoolExecutor(5)

    results = []
    for i in range(3):
        future_result = pool.submit(task, "name{}".format(i))
        res=future_result.result()  #同步调用,等到结果后再执行下一行代码
        print(res)

    print("主进程{}".format(os.getpid()))
    time2 = time.time()
    print("time2 - time1", time2 - time1)
"""
同步调用结果:由结果得知,每个进程得出了结果才去执行下一行代码
name0子进程37512:
name02
name1子进程35884:
name12
name2子进程33600:
name22
主进程36168
time2 - time1 6.2372283935546875
"""
    
    

举例4:使用submit异步调用
异步调用: 提交/调用一个任务,不在原地等着,直接执行下一行代码
from concurrent.futures import ProcessPoolExecutor
import time
import os

def task(name):
    print("{}子进程{}:".format(name,os.getpid()))
    time.sleep(2)
    return name+str(2)

if __name__ == "__main__":
    time1 = time.time()
    pool = ProcessPoolExecutor(5)

    results = []
    for i in range(3):
        future_result = pool.submit(task, "name{}".format(i))
        results.append(future_result)   #异步调用,不用等到结果,直接执行下一行代码

    for result in results:
        print(result.result())

    print("主进程{}".format(os.getpid()))
    time2 = time.time()
    print("time2 - time1", time2 - time1)
    
"""
异步调用结果:由结果得治,每个进程执行完成之后,并未等待结果,而是直接执行下一行代码
name0子进程36116:
name1子进程31972:
name2子进程660:
name02
name12
name22
主进程42280
time2 - time1 2.342352867126465
"""





相关文章

网友评论

      本文标题:并发编程之concurrent.futures模块

      本文链接:https://www.haomeiwen.com/subject/tzamqrtx.html