异步爬虫: async/await 与 aiohttp的使用

什么是 aiohttp？
一个异步的 HTTP 客户端服务端框架，基于 asyncio 的异步模块。
可用于实现异步爬虫，更快于 requests 的同步爬虫。

import asyncio
import aiohttp    
import time

async def get(url):
    async with aiohttp.ClientSession() as session:
        response = await session.get(url)
        result = await response.text()
        return result

async def request():
    url = 'http://127.0.0.1:8000/app02/index/'    #自己用django框架写个接口睡眠1秒钟
    print('Waiting for', url)
    result = await get(url)
    print('Get response from', url, 'Result:', result)

start_time = time.time()
tasks = [asyncio.ensure_future(request()) for _ in range(5)]
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
end = time.time()
print('耗时：', end - start_time)

打印结果：

Waiting for http://127.0.0.1:8000/app02/index/
Waiting for http://127.0.0.1:8000/app02/index/
Waiting for http://127.0.0.1:8000/app02/index/
Waiting for http://127.0.0.1:8000/app02/index/
Waiting for http://127.0.0.1:8000/app02/index/
Get response from http://127.0.0.1:8000/app02/index/ Result: ["app02", "oldboy"]
Get response from http://127.0.0.1:8000/app02/index/ Result: ["app02", "oldboy"]
Get response from http://127.0.0.1:8000/app02/index/ Result: ["app02", "oldboy"]
Get response from http://127.0.0.1:8000/app02/index/ Result: ["app02", "oldboy"]
Get response from http://127.0.0.1:8000/app02/index/ Result: ["app02", "oldboy"]
Cost time: 1.018911361694336