三十五. Scrapy爬虫框架

作者: 橄榄的世界 | 来源:发表于2018-02-28 20:11 被阅读0次

Scrapy爬虫框架：一个为了爬取网站信息，提取结构性数据而编写的应用爬虫框架，该框架集数据字段定义、网络请求和解析、数据获取和处理等为一体，极大地方便了爬虫的编写过程。

1.下面以小猪短租网的信息为例，讲述Scrapy爬虫项目的创建。
①在CMD命令窗口中输入信息：

F:
cd F:\01_Python\03_Scrapy   #选择路径
scrapy startproject xiaozhu    #建立爬虫项目，名为xiaozhu

image.png

②通过pycharm查看xiaozhu项目下的所有文件：

image.png

③在spiders文件夹下新建xiaozhuspiders.py，用于编写爬虫代码：

image.png

2.Scrapy文件介绍

如上图，最顶层的xiaozhu文件夹为项目名，第二层由与项目同名的文件夹xiaozhu和scrapy.cfg组成，其中xiaozhu文件夹就是模块，通常叫包，所有的爬虫代码都在这个包中添加，scrapy是这个项目的设置文件，其中的内容如下：

# Automatically created by: scrapy startproject
#
# For more information about the [deploy] section see:
# https://scrapyd.readthedocs.io/en/latest/deploy.html

[settings]
default = xiaozhu.settings

[deploy]
#url = http://localhost:6800/
project = xiaozhu

除了注释部分，这个配置文件声明了两件事：
①【settings】定义设置文件的位置为：xiaozhu模块下面的settings.py
②【deploy】定义项目名称为：xiaozhu
第三层由5个Python文件文件和spiders文件夹构成。spiders文件夹实际上也是一个模块。而5个Python文件夹中，init.py是空文件，主要用于Python导入使用。middlewares.py是spiders的中间件，主要负责对Request对象和Response对象进行处理，属于可选件。

-此处着重介绍其他3个Python文件items.py、pipelines.py、settings.py的使用：
①items.py文件：用于定义爬取的字段信息。自动生成的代码如下：

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class XiaozhuItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    pass

②pipelines.py文件：用于爬虫数据的清洗和入库操作。

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html


class XiaozhuPipeline(object):
    def process_item(self, item, spider):
        return item

③settings.py文件:用于对爬虫项目的一些设置，例如请求头的填写，设置pipelines.py爬取爬虫数据等。

# -*- coding: utf-8 -*-

# Scrapy settings for xiaozhu project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://doc.scrapy.org/en/latest/topics/settings.html
#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://doc.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'xiaozhu'

SPIDER_MODULES = ['xiaozhu.spiders']
NEWSPIDER_MODULE = 'xiaozhu.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'xiaozhu (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'xiaozhu.middlewares.XiaozhuSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#    'xiaozhu.middlewares.XiaozhuDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See https://doc.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
#    'xiaozhu.pipelines.XiaozhuPipeline': 300,
#}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

第4层为spiders模块下的2个Python文件，init.py与前面所述一致，xiaozhuspider.py是前面新建的Python文件，用于爬虫代码的编写。

综上所述，使用Scrapy爬虫框架相当于往里面填空，把scrapy项目中对应的文件代码补全即可实现爬虫。通常，我们设置以下4个文件即可完成爬虫任务。

items.py：用于定义字段。
xiaozhuspider.py：用于爬虫数据的获取。
pipelines.py：用于爬虫数据的处理。
settings.py：用于爬虫的设置。

3.下面爬取小组短租网的一页住房信息。
爬取字段为：标题、地址、价格、出租类型、居住人数和床位数。

image.png

①items.py文件：文件开头的注释部分不需要修改，其他部分替换掉，即可完成爬虫的字段信息。

import scrapy

class XiaozhuItem(scrapy.Item):
    # define the fields for your item here:
    title = scrapy.Field()
    address = scrapy.Field()
    price = scrapy.Field()
    lease_type = scrapy.Field()
    suggestion = scrapy.Field()
    bed = scrapy.Field()

②xiaozhuspider.py文件：爬虫代码的编写。

from scrapy.spiders import CrawlSpider   #CrawlSpider作为xiaozhu的父类
from scrapy.selector import Selector     #Selector用于解析请求网页后返回的数据，与Lxml库的用法一样
from xiaozhu.items import XiaozhuItem    #XiaozhuItem是items.py中定义爬虫字段的类

class xiaozhu(CrawlSpider):
    name = "xiaozhu"   #定义Spider的名字，在Scrapy爬虫运行时会用到：scrapy crawl xiaozhu
    start_urls = ['http://gz.xiaozhu.com/fangzi/26121722503.html'] # 默认情况下，spider会以start_urls中的链接为入口开始爬取

    def parse(self, response): #response是请求网页返回的数据
        item = XiaozhuItem()   #初始化item
        html = Selector(response)  #导入Selector用于解析数据
        title = html.xpath('//div[@class="pho_info"]/h4/em/text()').extract()[0]   #唯一不同，需要加上.extract()来提取信息
        address = html.xpath('//div[@class="pho_info"]/p/span/text()').extract()[0].strip()
        price = html.xpath('//div[@class="day_l"]/span/text()').extract()[0]
        lease_type = html.xpath('//li[@class="border_none"]/h6/text()').extract()[0]
        suggestion = html.xpath('//h6[@class="h_ico2"]/text()').extract()[0]
        bed = html.xpath('//h6[@class="h_ico3"]/text()').extract()[0]

        item['title'] = title
        item['address'] = address
        item['price'] = price
        item['lease_type'] = lease_type
        item['suggestion'] = suggestion
        item['bed'] = bed

        yield item

③pipelines.py文件：将获取的item存入TXT文件中。

class XiaozhuPipeline(object):
    def process_item(self, item, spider):
        f = open('F:/xiaozhu.txt','a+')
        f.write(item['title'] + '\n')
        f.write(item['address'] + '\n')
        f.write(item['price'] + '\n')
        f.write(item['lease_type'] + '\n')
        f.write(item['suggestion'] + '\n')
        f.write(item['bed'] + '\n')
        return item

④settings.py文件：在原有代码上恢复下述已经生成的代码，用于指定爬取的信息使用pipelines.py处理。

# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'xiaozhu.pipelines.XiaozhuPipeline': 300,
}

4.Scrapy爬虫的运行
在xiaozhu文件夹中，输入以下命令：
scrapy crawl xiaozhu

image.png

TXT文件内容如下：

image.png

当然，也可以在xiaozhu文件夹中新建一个main.py的文件，这样直接运行main.py即可运行爬虫程序。
代码如下：

from scrapy import cmdline
cmdline.execute("scrapy crawl xiaozhu".split())

如果设置了使用“C:\Python36\python.exe”作为默认打开py文件的程序，直接双击main.py即可生成结果。

网友评论

Python爬虫

本文标题：三十五. Scrapy爬虫框架

本文链接：https://www.haomeiwen.com/subject/ggnbxftx.html

延伸阅读

深度阅读

您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

三十五. Scrapy爬虫框架

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

Python爬虫