一、常用命令

• 安装：pip install scrapy
• 创建项目：scrapy startproject myproject
• 创建spider：scrapy genspider mydomain mydomain.com
• 运行spider：scrapy crawl mydomain

二、基础功能

1、通过脚本运行Spider

import subprocess
 
def run_scrapy_spider(spider_name):
    try:
        subprocess.run(['scrapy', 'crawl', spider_name])
        print(f"Spider {spider_name} finished.")
    except Exception as e:
        print(f"Error running spider: {e}")
 
# 使用函数运行爬虫，替换'your_spider_name'为你的爬虫名
run_scrapy_spider('your_spider_name')

2、Spider匹配多个类型URL

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
 
class MySpider(CrawlSpider):
    name = 'my_spider'
    
    # 初始种子URL
    start_urls = ['http://example.com/', 'http://example.com/posts']
    
    # 定义第一种页面的爬取规则
    # 元祖最后一个元素的逗号不能省略，否则报错
    rules = (
        # follow=True表示会跟随页面中提取的新链接继续爬取。
        Rule(LinkExtractor(allow=r'/details/\d+'), callback='parse_details', follow=True),
        # 定义第二种页面的爬取规则
        Rule(LinkExtractor(allow=r'/posts/\d+'), callback='parse_post', follow=True),
    )
    
    def parse_details(self, response):
        # 解析第一种类型的页面
        pass
    
    def parse_post(self, response):
        # 解析第二种类型的页面
        pass

3、一个pipeline处理多个item

class MultipleItemPipeline:
    def process_item(self, item, spider):
        rules = {
            MyspiderItem: self.handle_first_item,
        }
        for key, view in rules.items():
            if isinstance(item, key):
                view(item)
                break
        return item
 
    def handle_first_item(self, item):
        # 处理FirstItem
        pass
 
    def handle_second_item(self, item):
        # 处理SecondItem
        pass

4、设置User-Agent

设置随机User-Agent

5、Setting常用配置

# 日志级别
LOG_LEVEL = 'CRITICAL'
# 设置Proxy
HTTP_PROXY = 'http://xxx'
# 机器人协议
ROBOTSTXT_OBEY = False