Scrapy 爬取图片

1.创建Scrapy项目

scrapy startproject CrawlMeiziTu

scrapy genspider MeiziTuSpider  https://movie.douban.com/top250

image.png

2.cd到文件目录

cd CrawlMeizitu

image.png

3.创建爬虫,并设定初始爬取网页地址

scrapy genspider Meizitu http://www.meizitu.com/a/more_1.html

image.png

项目结构：

image.png

4.新建main.py文件

from scrapy import cmdline
cmdline.execute("scrapy crawl Meizitu".split())

image.png

5.编辑setting文件 主要设置USER_AGENT,下载路径，下载延迟时间

BOT_NAME = 'CrawlMeiziTu'

SPIDER_MODULES = ['CrawlMeiziTu.spiders']
NEWSPIDER_MODULE = 'CrawlMeiziTu.spiders'

#存储位置
IMAGES_STORE = '/Users/vincentwen/Downloads/img/meizitu/'

#模拟浏览器
USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'

#下载时间延迟
DOWNLOAD_DELAY = 0.3


ROBOTSTXT_OBEY = True

ITEM_PIPELINES = {
   'CrawlMeiziTu.pipelines.CrawlmeizituPipeline': 300,
}

image.png

6.编辑item，Items主要用来存取通过Spider程序抓取的信息。由于我们爬取妹子图，所以要抓取每张图片的名字，图片的连接，标签等等

import scrapy


class CrawlmeizituItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    #title文件夹名
    title = scrapy.Field()
    url = scrapy.Field()
    tags = scrapy.Field()
    #图片链接地址
    src = scrapy.Field()
    #alt为图片名
    alt = scrapy.Field()
    

7编辑Pipelines
Pipelines主要对items里面获取的信息进行处理。比如说根据title创建文件夹或者图片的名字，根据图片链接下载图片。

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html


import os
import requests

from CrawlMeiziTu.settings import IMAGES_STORE

class CrawlmeizituPipeline(object):
    def process_item(self, item, spider):
        fold_name = "".join(item['title'])

        header = {

            'USER-Agent': 'User-Agent:Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36',
            'Cookie': 'b963ef2d97e050aaf90fd5fab8e78633',
        }
        images = []

        #所有图片都放在一个文件夹下
        dir_path = '{}'.format(IMAGES_STORE)
        if not os.path.exists(dir_path) and len(item['src']) != 0:
            os.mkdir(dir_path)
        if len(item['src']) == 0:
            with open('..//check.txt', 'a+') as fp:
                fp.write("".join(item['title']) + ":" + "".join(item['url']))
                fp.write("\n")
        for jpg_url, name, num in zip(item['src'], item['alt'], range(0, 100)):
            file_name = name + str(num)
            file_path = '{}//{}'.format(dir_path, file_name)
            images.append(file_path)
            if os.path.exists(file_path) or os.path.exists(file_name):
                continue

            with open('{}//{}.jpg'.format(dir_path, file_name), 'wb') as f:
                req = requests.get(jpg_url, headers=header)
                f.write(req.content)
        return item

image.png

8.编辑Meizitu的主程序。

---

觉得文章有用，请用支付宝扫描，领取一下红包！打赏一下

支付宝红包码