Python爬取App指南

作者: 不存在的一角 | 来源:发表于2019-01-27 20:30 被阅读15次

Python爬取App指南
python爬取手机app
Python爬虫Scrapy(五)_Spiders
使用python爬取抖音app视频（appium可以操控手机）
爬虫douyin相关
以『公众号』为例，手把手教你爬取PC端数据
Python用Mitmproxy+Appium实现APP抓取
python爬虫学习手册-服务器渲染（基础库urllib）熟悉
Python爬取视频指南
用Python爬取手机APP

前菜

之前跟大家分享了许多爬虫的经验，当然很多我也是边学习边复盘这些掌握的知识，基本上来说我们已经可以上至九天揽月，下入五洋捉鳖

咳咳，事实上我们已经试过爬取的形式有文本、图片、音频、视频这些，而且基本上都是从网页上获取的，那么今天会带大家来看看如何从App来获取数据！

汤

工具：Fiddler、Android手机
环境：Python3、Scrapy

主菜

这里我们主要通过Fiddler对手机进行抓包来分析（当然有的人可能用的Charles，这里就不同时介绍了，因为方法很多，但思路是相似的），正常情况下是无法抓到手机上的数据的，所以在这之前我们需要配置一下

安装Fiddler，这个网上也可以下，或者在公众号回复【pye】里面也有
对Fiddler配置，打开Fiddler, Tools-> Fiddler Options (配置完后记得要重启Fiddler)，选中"Decrpt HTTPS traffic", Fiddler就可以截获HTTPS请求，选中"Allow remote computers to connect"，是允许别的机器把HTTP/HTTPS请求发送到Fiddler上来，这里注意下面的端口号 8888

再次注意重启Fiddler，接着打开命令行窗口，输入ipconfig查看本机的IP，这里我就不展示出来了，此时我们已经知道了IP以及端口号

接着打开手机，进入WLAN设置，修改已连接的网络（确保一下手机和PC是连接在同一个局域网中），设置代理为手动，填入本机的IP于服务器主机名，服务器端口为8888，然后保存即可

接着我们要下载安全证书，使用Android手机的浏览器打开：http://本机IP:8888，点"FiddlerRoot certificate" 然后下载证书

之后点击下载的证书可以命名证书便于区别（额...你可能觉得...没什么意义）

到这里我们的配置就结束了，接着打开Fiddler来看看，可以点击你手机上的一个App，就会看到Fiddler已经抓到了数据包

这里我首先试了下打开Boss直聘，搜索Python爬虫岗位的信息，看看抓到的包中的数据之怎样的，结果

可以看到虽然抓到了包，但是其中的数据是加密了的，没法获取对我们有用的信息，然后看了几个其他的url也是加密的，于是换了一个App，这次打开的是兼职猫

然后我在全部兼职中，给定范围，临时工，地点徐汇

那么我们来看看Fiddler是否抓到了这些兼职的数据

答案是肯定的，事实上你只要点击App的某个按钮，Fiddler都会抓到数据包，你也可以尝试下点击“我的”、“简历”这些就可以看到对应的操作在Fiddler中抓到的包，可能看的不太清晰，我们把这个链接拿出来放到浏览器上看能不能看到数据

首先通过右侧的Headers，我们可以知道这是一个Post请求，但是当我们在浏览器直接打开这个链接，也同样能看到我们想看的数据

图上所显示的就是对应的第一条兼职的信息，来点进去看看职位详情

同样这时候我们看看Fiddler最新抓到的

额...是不是看不太清...但是这就是对应的包，我发四！

我们来总结一下App抓包的流程：

1、首先需要配置Fiddler以及下载安全证书到你的Android手机
2、接着打开你想要获取数据的App，随意滑动、点击，Fiddler就能抓到发送的请求
3、再从这些刚刚获取的请求中筛选，就能找到所对应的数据
4、当然一般都会进行加密，这时候你要再想获取数据可能就得换种方式，从网页等入手
5、正常的话就可以拿到这个请求对应的链接，这时候再通过代码根据对应的请求方式，测试需要带上的请求参数，最后发起请求，就可以获得我们想要的数据

甜点

上面已经讲述了App抓包的流程，而这里我们来看一部分代码的演示，当然用的是Scrapy这个爬虫框架，虽然我平时喜欢写爬虫直接上，相当于一个脚本，这样对我来说直接简单，但是提高速度什么的也就需要自己开多进程、协程等

而用Scrapy就很好的解决了速度问题，其内部是基于Twisted框架，所以是以异步的方式发送请求，但是还需要你根据自己的要求改动某些参数

当然这也是你要做为一名虫师所必须要掌握的，所以之后的案例，代码我会尽量克服自己，多用Scrapy写，当然除了一些小东西，毕竟杀鸡焉用牛刀

首先是创建一个Scrapy项目，可以在命令行中运行如下指令

python3 -m scrapy startproject projectname

cd projectname

python3 -m scrapy genspider spidername "爬取网站的域名，like : xxx.com"

这样就创建了一个Scrapy项目，至于我为什么要加上 python3 -m 是因为我的机子同时装了Python2跟3，为了区别调用，若你只有Python3，可以不用加

这是Scrapy项目的基本结构

接着是基本的配置，一般都要在settings.py中令

ROBOTSTXT_OBEY = False  # 默认为True

这是不遵守网站的机器人协议，一般搜索引擎在访问一个网站的资源时都会首先访问这个协议，协议中规定了搜索引擎只能访问哪些内容，而我们的爬虫一般不会去遵守，你可以试试在网站的主网址下跟上/robots.txt，就像： https://www.bilibili.com/robots.txt 跑偏了跑偏了......

还需要修改开启这里的User-Agent，改为手机的请求头

USER_AGENT = 'Dalvik/2.1.0 (Linux; U; Android 7.0; KNT-AL10 Build/HUAWEIKNT-AL10)'

其他暂时可以不用动，接下来是 jzm.py 部分

# -*- coding: utf-8 -*-
import scrapy
from scrapy import Request
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings


class JzmSpider(scrapy.Spider):
    name = 'jzm'  # 爬虫名称
    allowed_domains = ['jianzhimao.com']  # 爬虫允许的域名范围
    
    # 重写了start_url 的内容，yield回调parse来解析返回的响应
    def start_requests(self):
        # 这是兼职猫，徐汇临时工兼职的数据的URL，从Fiddler抓包获得
        start_url = 'http://api.yh.jianzhimao.com/v35http/job/pageNewJob?order=10&token=f4b68d09a22bf4ee14f96623673b4a54&city=%E4%B8%8A%E6%B5%B7&mc=69DDU16519018674&system=1&timestamp=1548567219385&versions=5.8.1&request_tag=e6fc923518c9c5b313eddc45beaa2ab5&userid=13821503&lng=121.611041&usedRecommend_data=&area=%E5%BE%90%E6%B1%87&statistics_data=%7B%22userid%22%3A%2213821503%22%2C%22sys_info_sign%22%3A%222fdfc484a2b33b26874ac15e62246a66%22%2C%22app_version%22%3A%225.8.1%22%7D&sm=f694185c7ecbfffda31e734e91fc0dd4&channel=huawei&type=%E4%B8%B4%E6%97%B6%E5%B7%A5&page=1&lat=31.149018'
        yield Request(url=start_url, callback=self.parse)

    def parse(self, response):
        # 打印返回的结果
        print(response.text)


if __name__ == "__main__":
    # 这里这样做的好处是不用通过命令行 crawl 来运行Scrapy项目，可以直接右键Run
    process = CrawlerProcess(get_project_settings())
    # 参数为爬虫的名称
    process.crawl('jzm')
    process.start()

结果，注意右滑

C:\Python3\python3.exe D:/Python/PycharmProject/Work/Jzmao/Jzmao/spiders/jzm.py
2019-01-27 19:57:05 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: Jzmao)
2019-01-27 19:57:05 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.5, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 17:26:49) [MSC v.1900 32 bit (Intel)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0h  27 Mar 2018), cryptography 2.2.2, Platform Windows-8.1-6.3.9600-SP0
2019-01-27 19:57:05 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'Jzmao', 'NEWSPIDER_MODULE': 'Jzmao.spiders', 'SPIDER_MODULES': ['Jzmao.spiders'], 'USER_AGENT': 'Dalvik/2.1.0 (Linux; U; Android 7.0; KNT-AL10 Build/HUAWEIKNT-AL10)'}
2019-01-27 19:57:05 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2019-01-27 19:57:15 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-01-27 19:57:15 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-01-27 19:57:15 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-01-27 19:57:15 [scrapy.core.engine] INFO: Spider opened
2019-01-27 19:57:15 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-01-27 19:57:15 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2019-01-27 19:57:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://api.yh.jianzhimao.com/v35http/job/pageNewJob?order=10&token=f4b68d09a22bf4ee14f96623673b4a54&city=%E4%B8%8A%E6%B5%B7&mc=69DDU16519018674&system=1&timestamp=1548567219385&versions=5.8.1&request_tag=e6fc923518c9c5b313eddc45beaa2ab5&userid=13821503&lng=121.611041&usedRecommend_data=&area=%E5%BE%90%E6%B1%87&statistics_data=%7B%22userid%22%3A%2213821503%22%2C%22sys_info_sign%22%3A%222fdfc484a2b33b26874ac15e62246a66%22%2C%22app_version%22%3A%225.8.1%22%7D&sm=f694185c7ecbfffda31e734e91fc0dd4&channel=huawei&type=%E4%B8%B4%E6%97%B6%E5%B7%A5&page=1&lat=31.149018> (referer: None)
{"total":56,"entity":{"feedFlowRecommend":[]},"status":0,"msg":"ok","list":[{"id":2970091,"jobType":0,"days":180,"deadline":1563465599,"sex":"不限","tab":"1","type":"临时工","title":"爱建园3期连锁超市日结小时工寒假工","city":"上海","area":"徐汇","longitude":"121.438984","latitude":"31.182868","salary":"250元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":804,"isshow":1,"mi":16785,"userId":10963508,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548574990,"work_time":1547913600,"createtime":"2019-01-20 10:14:32","open_addr":"上海市徐汇爱建园","clearing_cycle":"日结","miToString":"16785.0","tabList":[{"tab_name":"日结","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2979916,"jobType":0,"days":181,"deadline":1564070399,"sex":"不限","tab":"1","type":"临时工","title":"徐家汇接听客服240","city":"上海","area":"徐汇","longitude":"121.443481","latitude":"31.200208","salary":"240元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":166,"isshow":1,"mi":16914,"userId":10739109,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548569959,"work_time":1548432000,"createtime":"2019-01-26 13:39:01","open_addr":"上海市徐汇徐家汇地铁附近","clearing_cycle":"月结","miToString":"16914.0","tabList":[{"tab_name":"客服","tab_color":"#866437"}],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2973256,"jobType":0,"days":38,"deadline":1551369599,"sex":"不限","tab":"1","type":"临时工","title":"1号线徐家汇服务员丶洗碗180/天日结","city":"上海","area":"徐汇","longitude":"121.443476","latitude":"31.200206","salary":"18元/小时","welfare":"1,2,3,4,5,","isrecommended":0,"is_filter_ios":0,"click_times":344,"isshow":1,"mi":16914,"userId":13341593,"is_filter_home_page":1,"privilege_tag":0,"start_time":1548555683,"work_time":1548086400,"createtime":"2019-01-22 03:38:19","open_addr":"上海市徐汇徐家汇地铁5号口","clearing_cycle":"日结","miToString":"16914.0","tabList":[{"tab_name":"日结","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2980235,"jobType":0,"days":6,"deadline":1548950399,"sex":"不限","tab":"2,3","type":"临时工","title":"日结200/天","city":"上海","area":"徐汇","longitude":"121.446234","latitude":"31.16915","salary":"200元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":537,"isshow":1,"mi":15829,"userId":13523363,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548549860,"work_time":1548432000,"createtime":"2019-01-26 19:30:51","open_addr":"上海市徐汇","clearing_cycle":"完工结","miToString":"15829.0","tabList":[{"tab_name":"日结","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2963613,"jobType":0,"days":44,"deadline":1551369599,"sex":"不限","tab":"1","type":"临时工","title":"1号线徐家汇洗碗工180/天日结","city":"上海","area":"徐汇","longitude":"121.443476","latitude":"31.200206","salary":"18元/小时","welfare":"1,2,3,4,5,","isrecommended":0,"is_filter_ios":0,"click_times":970,"isshow":1,"mi":16914,"userId":13341593,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548508604,"work_time":1547568000,"createtime":"2019-01-16 23:57:47","open_addr":"上海市徐汇徐家汇地铁5号口","clearing_cycle":"日结","miToString":"16914.0","tabList":[{"tab_name":"日结","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2980217,"jobType":0,"days":3,"deadline":1548777599,"sex":"男","tab":"2,3","type":"临时工","title":"顺丰短派200一天","city":"上海","area":"徐汇","longitude":"121.446234","latitude":"31.16915","salary":"200元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":375,"isshow":1,"mi":15829,"userId":10189553,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548501268,"work_time":1548518400,"createtime":"2019-01-26 19:14:28","open_addr":"上海市徐汇","clearing_cycle":"完工结","miToString":"15829.0","tabList":[{"tab_name":"日结","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2976450,"jobType":0,"days":32,"deadline":1550937599,"sex":"不限","tab":"1","type":"临时工","title":"温泉管包吃住5000","city":"上海","area":"徐汇","longitude":"121.443476","latitude":"31.200206","salary":"5000元/月","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":379,"isshow":1,"mi":16914,"userId":10189553,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548383539,"work_time":1548172800,"createtime":"2019-01-23 22:07:19","open_addr":"上海市徐汇徐家汇","clearing_cycle":"完工结","miToString":"16914.0","tabList":[],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2976793,"jobType":0,"days":180,"deadline":1563811199,"sex":"不限","tab":"1","type":"临时工","title":"兼职一手机日结一在家可做","city":"上海","area":"徐汇","longitude":"121.446234","latitude":"31.16915","salary":"300元/天","welfare":"1,2,3,4,5,","isrecommended":0,"is_filter_ios":0,"click_times":277,"isshow":1,"mi":15829,"userId":11286085,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548297528,"work_time":1548259200,"createtime":"2019-01-24 10:38:48","open_addr":"上海市徐汇加微信18170621123","clearing_cycle":"日结","miToString":"15829.0","tabList":[{"tab_name":"日结","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2908763,"jobType":0,"days":6,"deadline":1549641599,"sex":"不限","tab":"2,3","type":"临时工","title":"管吃住160/天服务员上海就近分配","city":"上海","area":"徐汇","longitude":"121.446234","latitude":"31.16915","salary":"160元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":1518,"isshow":1,"mi":15829,"userId":6888856,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548236183,"work_time":1549123200,"createtime":"2018-12-28 17:30:11","open_addr":"上海市徐汇就近分配","clearing_cycle":"日结","miToString":"15829.0","tabList":[],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2974897,"jobType":0,"days":180,"deadline":1563638399,"sex":"不限","tab":"1","type":"临时工","title":"裕德路连锁超市高价日结小时工寒假工","city":"上海","area":"徐汇","longitude":"121.441868","latitude":"31.191101","salary":"250元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":366,"isshow":1,"mi":16749,"userId":10963508,"is_filter_home_page":1,"privilege_tag":0,"start_time":1548160028,"work_time":1548086400,"createtime":"2019-01-22 20:27:08","open_addr":"上海市徐汇裕德路58号","clearing_cycle":"日结","miToString":"16749.0","tabList":[{"tab_name":"寒假工","tab_color":"#4756ec"}],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2794265,"jobType":0,"days":141,"deadline":1554047999,"sex":"不限","tab":"1","type":"临时工","title":"大型餐饮酒店每天要人日结18一个小时","city":"上海","area":"徐汇","longitude":"121.443125","latitude":"31.203978","salary":"180元/天","welfare":"1,2,3,4,5","isrecommended":0,"is_filter_ios":0,"click_times":13205,"isshow":1,"mi":17091,"userId":13808431,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548132357,"work_time":1541865600,"createtime":"2018-11-10 15:44:46","open_addr":"上海市上海徐汇汇银广场","clearing_cycle":"完工结","miToString":"17091.0","tabList":[{"tab_name":"日结","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"认证企业","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2946398,"jobType":0,"days":42,"deadline":1550678399,"sex":"不限","tab":"1","type":"临时工","title":"宜芝多136-200/天寒假工","city":"上海","area":"徐汇","longitude":"121.446235","latitude":"31.169152","salary":"200元/天","welfare":"5","isrecommended":0,"is_filter_ios":0,"click_times":664,"isshow":1,"mi":15829,"userId":1006298,"is_filter_home_page":1,"privilege_tag":0,"start_time":1548062002,"work_time":1547049600,"createtime":"2019-01-10 19:33:00","open_addr":"上海市徐汇全上海各地铁站就近分配","clearing_cycle":"月结","miToString":"15829.0","tabList":[{"tab_name":"寒假工","tab_color":"#4756ec"}],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2971579,"jobType":0,"days":39,"deadline":1551369599,"sex":"不限","tab":"1","type":"临时工","title":"上海宜芝多门店就近安排","city":"上海","area":"徐汇","longitude":"121.446234","latitude":"31.16915","salary":"5000元/月","welfare":"5,","isrecommended":0,"is_filter_ios":0,"click_times":224,"isshow":1,"mi":15829,"userId":14353534,"is_filter_home_page":0,"privilege_tag":0,"start_time":1548036331,"work_time":1548000000,"createtime":"2019-01-21 10:05:31","open_addr":"上海市徐汇","clearing_cycle":"月结","miToString":"15829.0","tabList":[],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2970422,"jobType":0,"days":26,"deadline":1550246399,"sex":"不限","tab":"1","type":"临时工","title":"急招寒假工餐厅服务员","city":"上海","area":"徐汇","longitude":"121.487899","latitude":"31.249162","salary":"150元/天","welfare":"1,2","isrecommended":0,"is_filter_ios":0,"click_times":167,"isshow":1,"mi":16149,"userId":3549404,"is_filter_home_page":0,"privilege_tag":0,"start_time":1547961776,"work_time":1548000000,"createtime":"2019-01-20 13:22:56","open_addr":"上海市上海徐汇","clearing_cycle":"完工结","miToString":"16149.0","tabList":[{"tab_name":"寒假工","tab_color":"#4756ec"}],"vip_tab_list":[{"tab_name":"认证企业","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2970334,"jobType":0,"days":182,"deadline":1563638399,"sex":"不限","tab":"1","type":"临时工","title":"兼职一手机日结一在家可做","city":"上海","area":"徐汇","longitude":"121.446235","latitude":"31.169152","salary":"300元/天","welfare":"1,2,3,4,5","isrecommended":0,"is_filter_ios":0,"click_times":759,"isshow":1,"mi":15829,"userId":11286085,"is_filter_home_page":0,"privilege_tag":0,"start_time":1547957457,"work_time":1547913600,"createtime":"2019-01-20 12:10:57","open_addr":"上海市上海徐汇加微信18170621123","clearing_cycle":"日结","miToString":"15829.0","tabList":[{"tab_name":"日结","tab_color":"#d226cf"}],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2960246,"jobType":0,"days":180,"deadline":1563119999,"sex":"不限","tab":"1","type":"临时工","title":"徐汇区连锁超市日结小时工","city":"上海","area":"徐汇","longitude":"121.441015","latitude":"31.191503","salary":"250元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":738,"isshow":1,"mi":16839,"userId":10963508,"is_filter_home_page":0,"privilege_tag":0,"start_time":1547864800,"work_time":1547568000,"createtime":"2019-01-16 01:04:01","open_addr":"上海市徐汇裕德路","clearing_cycle":"日结","miToString":"16839.0","tabList":[],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2948610,"jobType":0,"days":49,"deadline":1551369599,"sex":"不限","tab":"1","type":"临时工","title":"711便利店寒假兼职","city":"上海","area":"徐汇","longitude":"121.443476","latitude":"31.200206","salary":"20元/小时","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":2222,"isshow":1,"mi":16914,"userId":10189553,"is_filter_home_page":0,"privilege_tag":0,"start_time":1547819343,"work_time":1547136000,"createtime":"2019-01-11 16:54:47","open_addr":"上海市徐汇徐家汇","clearing_cycle":"完工结","miToString":"16914.0","tabList":[{"tab_name":"可长期","tab_color":"#3a45eb"}],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2964534,"jobType":0,"days":0,"deadline":1550764799,"sex":"不限","tab":"2","type":"临时工","title":"高薪百丽鞋子营业员就近分配","city":"上海","area":"徐汇","longitude":"121.427217","latitude":"31.191935","salary":"185元/天","welfare":"2,","isrecommended":0,"is_filter_ios":0,"click_times":216,"isshow":1,"mi":18115,"userId":10258217,"is_filter_home_page":0,"privilege_tag":0,"start_time":1547704536,"work_time":1550678400,"createtime":"2019-01-17 13:55:36","open_addr":"上海市徐汇柳州路928号","clearing_cycle":"完工结","miToString":"18115.0","tabList":[{"tab_name":"包住","tab_color":"#b71144"}],"vip_tab_list":[{"tab_name":"认证企业","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2906710,"jobType":0,"days":31,"deadline":1548950399,"sex":"男","tab":"1","type":"临时工","title":"顺丰短派兼职","city":"上海","area":"徐汇","longitude":"121.443476","latitude":"31.200206","salary":"180元/天","welfare":"","isrecommended":0,"is_filter_ios":0,"click_times":2358,"isshow":1,"mi":16914,"userId":10189553,"is_filter_home_page":0,"privilege_tag":0,"start_time":1547651147,"work_time":1546272000,"createtime":"2018-12-28 07:21:18","open_addr":"上海市徐汇徐家汇","clearing_cycle":"完工结","miToString":"16914.0","tabList":[{"tab_name":"包住","tab_color":"#b71144"}],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0},{"id":2947376,"jobType":0,"days":39,"deadline":1550764799,"sex":"不限","tab":"1","type":"临时工","title":"罗森便利店寒假工187/ 天","city":"上海","area":"徐汇","longitude":"121.455922","latitude":"31.195215","salary":"187元/天","welfare":"5","isrecommended":0,"is_filter_ios":0,"click_times":617,"isshow":1,"mi":15615,"userId":195218,"is_filter_home_page":0,"privilege_tag":0,"start_time":1547646860,"work_time":1547395200,"createtime":"2019-01-11 11:04:15","open_addr":"上海市徐汇罗森便利店就近分配","clearing_cycle":"月结","miToString":"15615.0","tabList":[],"vip_tab_list":[{"tab_name":"热点招聘","tab_color":"#ff9c27"}],"is_nationwide":0}]}
2019-01-27 19:57:15 [scrapy.core.engine] INFO: Closing spider (finished)
2019-01-27 19:57:15 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 789,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 14370,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 1, 27, 11, 57, 15, 591689),
 'log_count/DEBUG': 2,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2019, 1, 27, 11, 57, 15, 177682)}
2019-01-27 19:57:15 [scrapy.core.engine] INFO: Spider closed (finished)

Process finished with exit code 0

以上就是今天的内容，之后也会更多的来讲解Scrapy，敬请期待

print('微信公众号搜索 "猿狮的单身日常" ，Java技术升级、虫师修炼，我们 不见不散!')
print('也可以扫下方二维码哦~')

猿狮的单身日常

Python爬取App指南
前菜之前跟大家分享了许多爬虫的经验，当然很多我也是边学习边复盘这些掌握的知识，基本上来说我们已经可以上至九天揽月...
python爬取手机app
python爬取手机app
Python爬虫Scrapy(五)_Spiders
更多内容请参考：Python学习指南 Spider Spider类定义了如何爬取某个网站(或某些)网站。包括了爬取...
使用python爬取抖音app视频（appium可以操控手机）
记录一下如何用python爬取app数据，本文以爬取抖音视频app为例。编程工具：pycharm app抓包工具...
爬虫douyin相关
记录一下如何用python爬取app数据，本文以爬取抖音视频app为例。编程工具：pycharm app抓包工具...
以『公众号』为例，手把手教你爬取PC端数据
之前教过大家如何爬取手机APP数据【以某乎为实战案例，教你用Python爬取手机App数据[http://mp.w...
Python用Mitmproxy+Appium实现APP抓取
记录一下如何用python爬取app数据，本文以爬取抖音视频app为例！转载请说明出处。编程工具：pycharm...
python爬虫学习手册-服务器渲染（基础库urllib）熟悉
今天我们来说说python爬虫的第一步！爬，爬这一步分为网站爬取和APP爬取，而网站爬取里面按照渲染方式，有分为服...
Python爬取视频指南
前言前两天尔羽说让我爬一下菜鸟窝的教程视频，这次就跟大家来说说Python爬取视频的经验正文 https://...
用Python爬取手机APP
前言如果你以为python只可以爬取web网页，那就大错特错了，本篇文章教你如何爬取手机app的信息。 Char...