以爬取qq音乐评论为例
动态网页中的部分数据并不在返回的HTML文件中,因此按照静态网页的爬取方式无法获得数据。
F12调出检查元素面板,选择Network,清空内容,在网页中选择下一页评论,这样就可以筛选出评论所请求的数据。
根据Size和Name可以很容易地判断出评论内容的数据,选择评论数据。
其中Headers下的General下的Request URL就是获取评论的网址,为简化网址,把网址问号后的部分转换成json格式(json就是字符串类型的字典或列表,可以在不同语言编程过程中传递数据),借助于该网站(https://www.convertonline.io/convert/query-string-to-json)可以自动把网址的参数部分转化成json格式,这时request请求可以简写成下面这样:
params = {# json中字符串必须使用双引号
"g_tk_new_20200303":"5381",
"g_tk":"5381",
"loginUin":"0",
"hostUin":"0",
"format":"json",
"inCharset":"utf8",
"outCharset":"GB2312",
"notice":"0",
"platform":"yqq.json",
"needNewCode":"0",
"cid":"205360772",
"reqtype":"2",
"biztype":"1",
"topid":"212877900",
"cmd":"8",
"needmusiccrit":"0",
"pagenum":"1",
"pagesize":"25",
"lasthotcommentid":" song_212877900_1152921504825029402_1592083838 ",
"domain":"qq.com",
"ct":"24",
"cv":"10101010"
}
res = requests.get('https://c.y.qq.com/base/fcgi-bin/fcg_global_comment_h5.fcg', headers=headers, params=params)
data = res.json()
这些参数在Headers下的Query String parameters也有体现:
可以多观察几页评论的参数变化爬取多页评论,其中pagenum是评论页码,lasthotcommentid记录上页最后一个评论的id,防止翻页时出现新评论使得下页前几个评论重复。
最终data是字典类型,其内容就是preview的内容:
利用Python中字典的使用方法就可以获得需要的数据了。
爬取前五页评论的代码:
import requests
import time
headers = {
'user-agent':''
}
lasthotcommentid =""
for pagenumin range(5):
params = {# json中字符串必须使用双引号
"g_tk_new_20200303":"5381",
"g_tk":"5381",
"loginUin":"0",
"hostUin":"0",
"format":"json",
"inCharset":"utf8",
"outCharset":"GB2312",
"notice":"0",
"platform":"yqq.json",
"needNewCode":"0",
"cid":"205360772",
"reqtype":"2",
"biztype":"1",
"topid":"212877900",
"cmd":"8",
"needmusiccrit":"0",
"pagenum":pagenum,
"pagesize":"25",
"lasthotcommentid":lasthotcommentid,
"domain":"qq.com",
"ct":"24",
"cv":"10101010"
}
res = requests.get('https://c.y.qq.com/base/fcgi-bin/fcg_global_comment_h5.fcg', headers=headers, params=params)
data = res.json()
comment_list = data['comment']['commentlist']
for commentin comment_list:
print(comment['nick'], comment['rootcommentcontent'])
lasthotcommentid = comment_list[-1]['commentid']
print("---------------------------------------------")
time.sleep(0.5)
网友评论