Python爬虫-爬取杀破狼豆瓣影评并小作分析~

也是前几天看到一个公众号推了一篇文章，是爬取战狼的影评。今天自己也来试一下
我选择爬的是《杀破狼》

image.png

然后就是打开短评页面，可以看到comment-item，这就是影评了

image.png

现在已经找到想要的了，但是这仅仅是第一页的，可以看到一共有六千多条记录，那么怎么拿到其他的呢，页面拉到下方的后页，可以看到地址栏变成了下面的这个地址

image.png

所以可以知道limit应该是每页记录数，start是从第几条开始，知道这个我们就知道了所有的地址啦

url_list = ['https://movie.douban.com/subject/26826398/comments?' \
            'start={}&limit=20&sort=new_score&status=P' .format(x)for x in range(0, 6317, 20)]

爬取过程就是利用bs4拿到想要的就ok

response = requests.get(url=url, headers=header)
            response.encoding = 'utf-8'
            html = BeautifulSoup(response.text, 'html.parser')
            comment_items = html.select('div.comment-item')
            for item in comment_items:
                comment = item.find('p')

然后把爬取的文本写入txt中最后用来作数据分析

image.png

要作数据分析首先到网上找个停用词表，然后利用jieba来分析，代码如下（这里也是看了罗罗攀的文章：http://www.jianshu.com/p/b277199346ae)

def fenci():
    path = '/Users/mocokoo/Documents/shapolang.txt'
    with open(path, mode='r', encoding='utf-8') as f:
        content = f.read()
        analyse.set_stop_words('/Users/mocokoo /Documents/tycibiao.txt')
        tags = analyse.extract_tags(content, topK=100, withWeight=True)
        for item in tags:
            print(item[0] + '\t' + str(int(item[1] * 1000)))

image.png
最后利用这个网站来制作一下输出结果
https://wordart.com/create image.png

最后附上完整代码：

#!usr/bin/env python3
# -*- coding:utf-8-*-

import requests
from bs4 import BeautifulSoup
import jieba.analyse as analyse

header = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Connection': 'keep-alive',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 '
                  '(KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
}


url_list = ['https://movie.douban.com/subject/26826398/comments?' \
            'start={}&limit=20&sort=new_score&status=P' .format(x)for x in range(0, 6317, 20)]

# 爬取所有短评写入文件中


def get_comments():
    with open(file='/Users/mocokoo/Documents/shapolang.txt', mode='w', encoding='utf-8') as f:
        i = 1
        for url in url_list:
            print('正在爬取杀破狼影评第_%d_页' % i)
            response = requests.get(url=url, headers=header)
            response.encoding = 'utf-8'
            html = BeautifulSoup(response.text, 'html.parser')
            comment_items = html.select('div.comment-item')
            for item in comment_items:
                comment = item.find('p')
                f.write(comment.get_text().strip() + '\n')
            print('第_%d_页完成' % i)
            i += 1
# 分词


def fenci():
    path = '/Users/mocokoo/Documents/shapolang.txt'
    with open(path, mode='r', encoding='utf-8') as f:
        content = f.read()
        analyse.set_stop_words('/Users/mocokoo/Documents/tycibiao.txt')
        tags = analyse.extract_tags(content, topK=100, withWeight=True)
        for item in tags:
            print(item[0] + '\t' + str(int(item[1] * 1000)))

if __name__ == '__main__':
    get_comments() # 将影评写入文档中
    # fenci()