Python 爬虫 - Instagram异步协程！

作者: 14e61d025165 | 来源:发表于2019-05-17 16:02 被阅读0次

Python 爬虫 - Instagram异步协程！
Python爬虫 - Instagram异步协程
Python Scraping ———09.24.2017
python异步协程(aiohttp,asyncio)
Python进程+协程——从零开始搭建异步爬虫（1）
Python进程+协程——从零开始搭建异步爬虫（2）
python爬虫11：gevent queue
Python爬虫进阶 | 异步协程
Python协程之asyncio
这怕是全网最详细的异步IO之协程详解！

首先分析网站图片加载流程，taeri__taeri 应该有人认识这个网红。ins照片一次只加载了一定数量的照片，往下翻又会加载，毫无疑问看 xhr

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1558080057176" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

在预览栏里可以看到json数据， display_url 就是照片的链接，只要获取到这个就行了

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1558080057179" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

Python学习交流群：1004391443，这里有资源共享，技术解答，还有小编从最基础的Python资料到项目实战的学习资料都有整理，希望能帮助你更了解python，学习python。

2、分析请求参数

回到 headers 看看请求用了哪些参数；就两个， quer_hash 和 variables

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1558080057183" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

variables 是一个json，里面有 id、first、after 这三项；为了不麻烦。。我直接说这三个是啥玩意儿，有兴趣的可以自己分析

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">id：user id 即用户id
first：这次请求加载照片数量
after：end cursor 这个参数是为了判断上一页的，没有这个就一直加载的第一页，而本页会带有一个end cursor参数来进行下一页请求
</pre>

还有一点，需要加上cookie，这个应该不用多说了

3、程序流程

请求啥的都分析好了，接下来就来分析程序怎么写，我现在的需求是这样的

给定一个用户名，尽快获取该用户所有照片并下载到instagram//这个目录下

要求中说了需要尽快，那么单线程就算了，太没效率的，就一个普通网红照片好歹也有几百张；这里用到asyncio，流程是这样的

首先下载和获取图片链接弄成两个任务，在等待获取的时候我先去执行下载里的任务，在等待下载的时候我可以获取下载链接，这就是一个生产者消费者模式，这里涉及到了通信，我用queue。

二、代码

用到的lib：

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import json
import multiprocessing
import sys
from urllib.parse import urljoin
import aiohttp
import asyncio
import os
import re
from pathlib import Path
import requests
复制代码
</pre>

init:

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">def init(self, username, maxtasks=200):
self.username = username
self.maxtasks = maxtasks # 最大任务数
self.queue = asyncio.Queue(maxsize=maxtasks * 2)
# 配置代理，没有科学上网没法访问ins
os.environ['http_proxy'] = PROXY
os.environ['https_proxy'] = PROXY
self.session = aiohttp.ClientSession(trust_env=True, headers=HEADERS)
复制代码
</pre>

首先获取user id：

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">async def get_shared_data(self):
"""
获取 shared data
:return:
"""
try:
async with self.session.get(ROOT_URL + self.username) as resp:
html = await resp.text()
if html is not None and '_sharedData' in html:
shared_data = html.split("window._sharedData = ")[1].split(
";</script>")[0]
if not shared_data: # 没有shared data可以直接终止程序了
print('!!!!!!!')
exit(1)
return json.loads(shared_data)
except Exception:
pass
async def init(self):
"""
初始化必要参数
:return:
"""
user = (await self.get_shared_data())['entry_data']['ProfilePage'][0]['graphql']['user']
if not user:
print('user is none.')
exit(1)
self.user_id = user['id'] # user id
self.count = user['edge_owner_to_timeline_media']['count'] # 照片数量
复制代码
</pre>

生产者：

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">async def produce_download_urls(self, max=50):
"""
获取每一页的所有照片链接
:param max: 一次要获取照片数量
:return:
"""
end_cursor = '' # 加载第一页
while True:
pic_params = {
'query_hash':
'f2405b236d85e8296cf30347c9f08c2a', # query_hash 可以固定一个值
'variables':
'{{"id":"{0}","first":{1},"after":"{2}"}}'.format(
self.user_id, max, end_cursor),
}
pic_url = ROOT_URL + 'graphql/query/'
async with self.session.get(pic_url, params=pic_params) as resp:
json = await resp.json()
edge_media = json['data']['user'][
'edge_owner_to_timeline_media']
edges = edge_media['edges']
if edges:
for edge in edges:
await self.queue.put(edge['node']['display_url']) # queue通信
has_next_page = edge_media['page_info']['has_next_page'] # json中有一个has next page项，其值是 true或false，用来判断是否有下一页
if has_next_page:
end_cursor = edge_media['page_info']['end_cursor'] # 获取 end cursor
else:
break
复制代码
</pre>

消费者：

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">async def download(self):
"""
下载照片
:return:
"""
while not (self.producer.done() and self.queue.empty()): # 生产任务是否没有完成以及queue队列是否不为空
url = await self.queue.get() # 获取照片链接
filename = PATH / url.split('?')[0].split('/')[-1]
async with self.session.get(url) as resp:
with filename.open('wb') as f:
async for chunk in resp.content.iter_any():
f.write(chunk)
self.queue.task_done() # 表示刚刚排队的任务已完成（就是用get取出的照片url下载完成了）
print('.', end='', flush=True)
复制代码
</pre>

run：

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">async def run(self):
"""
:return:
"""
print('Preparing...')
print('Initializing...')
await self.init()
print('User id: %r.' % self.user_id)
print('Total %r photos.' % self.count)
print('-'50)
self.producer = asyncio.create_task(self.produce_download_urls())
print('Downloading...', end='', flush=True)
await asyncio.gather((self.download() for _ in range(self.maxtasks))) # asyncio.gather和asyncio.wait差不多，具体百度
复制代码
</pre>

check：

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">def check(_):
"""
检测照片数量。。。太菜了不知道怎么停止只能这样了（逃；
"""
print('Start check...')
with requests.get(urljoin(ROOT_URL, USERNAME), headers=HEADERS,
proxies={'http': 'http://localhost:80001', 'https': 'https://localhost:8001'}) as resp:
pattern = '"edge_owner_to_timeline_media":.?{"count":(.*?),"page_info"'
count = int(re.findall(pattern, resp.text)[0])
while True:
files = len(os.listdir(PATH))
print('Check files:%r' % files)
if files == count:
# print('Total %r photos download done.' % count)
print('\nProduce done, Total %r photos, plz wait save done :)' % count)
sys.exit(0)
复制代码
</pre>

main：

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">async def main():
ins = Instagram(USERNAME)
try:
await ins.run()
finally:
await ins.close()
复制代码
</pre>

if name == 'main':

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">if name == 'main':
try:
p = multiprocessing.Process(target=check, args=(0,))
p.start()
future = asyncio.ensure_future(main())
loop = asyncio.get_event_loop()
loop.run_until_complete(future)
loop.close()
except KeyboardInterrupt:
pass
复制代码
</pre>

运行：

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1558080057216" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

网友评论

本文标题：Python 爬虫 - Instagram异步协程！

本文链接：https://www.haomeiwen.com/subject/cseoaqtx.html

延伸阅读

深度阅读

您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

Python 爬虫 - Instagram异步协程！

相关文章