需求分析

简书文章的分析功能比较弱，只能按照热度排序，从页面上看，热度指的是点赞数。

热度

可文章还有其他的分析维度：阅读数，评论数, 点赞数。简书并没有提供对这些维度的分析。

既然如此，就自己撸起袖子干吧...

实现的需求很简单：将自己简书文章的阅读、评论、点赞、打赏、标题、发布时间抓取下来，存入数据库，再进行分析展示

效果如下：

简书文章分析.gif

以上只是最简单的展示，可以自定义其他数据分析效果

具体实现

数据抓取

使用python抓取页面数据，抓取之前先分析页面的html结构

分析html结构

具体实现代码：

# -*- coding: utf-8 -*-
import requests
import pyquery
import time
import datetime
import pymysql

# 数据库连接信息
conn = pymysql.connect(host='127.0.0.1', user='root', passwd=None, db='test', charset='utf8')
cur = conn.cursor()

user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' \
             ' (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'
headers = {"User-Agent": "user-agent:%s" % user_agent}
page = 0
flag = True

while flag:
    baseUrl = 'https://www.jianshu.com/u/f9338eda7dda?page='
    page = int(page) + 1
    url = baseUrl + str(page)
    print(url)

    # 抓取数据
    req = requests.get(url, headers=headers, timeout=2)
    pageText = req.text
    pq = pyquery.PyQuery(pageText)
    contents = pq('li')
    for x in contents:
        el = pq(x)
        title = el.find('a.title').text()
        if title:
            nodeId = el.attr('data-note-id')

            # data-note-id为空时，表示文章已抓取完毕，此时退出循环
            if nodeId is None:
                flag = False
                break

            link = 'https://www.jianshu.com' + el.find('a.title').attr('href') # 文章链接
            postTime = el.find('span.time').attr('data-shared-at') # 发布时间
            dateTime = datetime.datetime.strptime(postTime, "%Y-%m-%dT%H:%M:%S+08:00")
            create_time = int(time.mktime(dateTime.timetuple()))
            read_num = el.find('i.ic-list-read').parent().text() # 阅读数
            comment_num = el.find('i.ic-list-comments').parent().text() # 评论数
            like_num = el.find('i.ic-list-like').parent().text() # 点赞数
            money_num = el.find('i.ic-list-money').parent().text() # 打赏数
            if money_num is '':
                money_num = 0

            # 数据入库
            analyze_time = int(time.time())
            sql = "insert into analyze_article \
                  (title, link, create_time, analyze_time, read_num, like_num, comment_num, money_num) values \
                  ('%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s')" % \
                  (title, link, create_time, analyze_time, read_num, like_num, comment_num, money_num)
            cur.execute(sql)
            conn.commit()

    # 暂停1秒，避免被简书的反爬虫拦截
    time.sleep(1)

php读取数据

爬虫将数据入库后，用php作为服务端读取数据表数据

极简单的数据读取脚本，无需解释，直接贴代码

<?php
header("Access-Control-Allow-Origin:*"); // 如果客户端和服务端不同域，要加上这行代码，不然会报跨域错误
$con=mysqli_connect("localhost","root","","test");
$analyzeTime = strtotime(date('Y-m-m', time())) - 3600 * 24;
$sql="SELECT * FROM analyze_article where analyze_time >= $analyzeTime";
$order = '';
if (isset($_GET['read_num'])) {
    $order = " order by read_num desc";
}
if (isset($_GET['like_num'])) {
    $order = " order by like_num desc";
}
if (isset($_GET['comment_num'])) {
    $order = " order by comment_num desc";
}
if (isset($_GET['money_num'])) {
    $order = " order by money_num desc";
}
$sql .= $order;
$result=mysqli_query($con,$sql);
$data=mysqli_fetch_all($result, MYSQLI_ASSOC);
mysqli_free_result($result);
mysqli_close($con);
echo json_encode($data, true);

前端使用vue.js展现

php后端返回json数据，vue.js将json数据解析展现到页面

<!doctype html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <link href="https://cdn.bootcss.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet">
    <script src="https://cdn.jsdelivr.net/npm/vue/dist/vue.js"></script>
    <script src="https://unpkg.com/axios/dist/axios.min.js"></script>
    <title>简书文章分析</title>
    <style>
        .container {
            margin-top: 2%;
        }
    </style>
</head>
<body>
<div class="container">
    <div id="app">
        <h3 class="text-center">简书文章分析</h3>
        <table class="table table-bordered table-hover">
            <tr>
                <th>标题</th>
                <th><a href="" @click.prevent="changeOrder('read_num')" class="text-info">阅读</a></th>
                <th><a href="" @click.prevent="changeOrder('like_num')" class="text-danger">点赞</a></th>
                <th><a href="" @click.prevent="changeOrder('comment_num')" class="text-warning">评论</a></th>
                <th><a href="" @click.prevent="changeOrder('money_num')" class="text-success">打赏</a></th>
            </tr>
            <tr v-for="item in list">
                <td><a :href="item.link" target="_blank">{{ item.title }}</a></td>
                <td>{{ item.read_num }}</td>
                <td>{{ item.like_num }}</td>
                <td>{{ item.comment_num }}</td>
                <td>{{ item.money_num }}</td>
            </tr>
        </table>
    </div>
</div>
<script>
    let url = 'http://local.php.com/jianshu.php';
    let vm = new Vue({
        el: '#app',
        data: {
            list: []
        },
        methods: {
            changeOrder: function (sign) {
                let reqUrl = url + '?' + sign + '=1'
                axios.get(reqUrl, {})
                    .then(function (response) {
                        vm.$data.list = response.data;
                    })
            },
        }
    });
    axios.get(url, {})
        .then(function (response) {
            vm.$data.list = response.data;
        })
        .catch(function (error) {
            console.log(error);
        })
        .then(function () {
            // always executed
        });
</script>
</body>
</html>

对于vue.js不熟悉的同学，推荐查看：实例学习vue.js目录