美文网首页python-Scrapy
爬虫入门(4)-Scrapy框架简单入门

爬虫入门(4)-Scrapy框架简单入门

作者: Maxim_Tian | 来源:发表于2017-05-16 20:25 被阅读49次

    scrapy作为一个强大的爬虫框架,就不多作介绍。
    今天刚入门scrapy,所以做个简单的使用

    Scrapy中文文档

    http://scrapy-chs.readthedocs.io/zh_CN/latest/index.html

    Ubuntu安装scrapy

    在Ubuntu下安装scrapy,需要的命令为:

    pip install zope.interface
    pip install twisted
    pip install pyopenssl
    pip install scrapy
    

    开始

    1. 使用scrapy创建文件夹命令:

    scrapy startproject doubanTest

    命令执行完系统将在当前目录下生成doubanTest文件

    2. 看看最后的代码文件目录(开始时创建不存在douban_spider.py,main.py文件)
    3. 网页提取的元素需要在items.py定义
    # -*- coding: utf-8 -*-
    
    # Define here the models for your scraped items
    #
    # See documentation in:
    # http://doc.scrapy.org/en/latest/topics/items.html
    
    import scrapy
    
    
    class DoubantestItem(scrapy.Item):
        # define the fields for your item here like:
        # name = scrapy.Field()
        title = scrapy.Field() # 抓取电影标题
        content = scrapy.Field() # 抓取电影简介
    
    

    比如我在代码中定义了抓取电影的标题和电影的简介

    关于item的使用,官方文档这么描述:

    Item用法类似字典dict

    4. 在settings.py定义一些防反爬元素
    # -*- coding: utf-8 -*-
    
    # Scrapy settings for doubanTest project
    #
    # For simplicity, this file contains only settings considered important or
    # commonly used. You can find more settings consulting the documentation:
    #
    #     http://doc.scrapy.org/en/latest/topics/settings.html
    #     http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
    #     http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
    
    BOT_NAME = 'doubanTest'
    
    SPIDER_MODULES = ['doubanTest.spiders']
    NEWSPIDER_MODULE = 'doubanTest.spiders'
    
    
    # Crawl responsibly by identifying yourself (and your website) on the user-agent
    USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36'
    
    FEED_URI = 'doubanmovie.csv' # 将信息存储到doubanmovie.csv文件中
    FEED_FORMAT = 'CSV' # 存储的文件类型
    
    5. 在spiders下创建douban_spider.py,编写主要代码
    #-*_coding:utf8-*-
    
    import scrapy
    from scrapy.spider import Spider
    from scrapy.http import Request
    from scrapy.selector import Selector
    from doubanTest.items import DoubantestItem
    
    class doubanSpider(Spider):
        name="doubanSpider" # 爬虫程序名称
        start_urls = ['https://movie.douban.com/top250'] # URL访问列表
    
        def parse(self, response):
            # print response.body
            item = DoubantestItem() # 定义一个Item对象
            selector = Selector(response)
            Movies = selector.xpath('//div[@class="info"]')
            for each in Movies:
                title = each.xpath('div[@class="hd"]/a/span/text()').extract() # 获取电影标题
                full_title = "".join(title)
                content = each.xpath('div[@class="bd"]/p/text()').extract() # 获取电影简介
    
                # item类似dict, 它的key名称必须与items.py里的名称一致
                item['title'] = full_title
                item['content'] = ';'.join(content).strip()
                yield item
    
    6. 在doubanTest目录下创建一个main.py

    ok,就当作scrapy入门例子吧

    好好学习天天向上~

    相关文章

      网友评论

        本文标题:爬虫入门(4)-Scrapy框架简单入门

        本文链接:https://www.haomeiwen.com/subject/aizlxxtx.html