scrapy爬虫出现403

在settings文件中添加下面一段

USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36'

就算在爬虫代码中增加了请求头，也要在设置中增加

headers = {
            'Host': 'channel.chinanews.com',
            'Referer': 'https://www.chinanews.com/',
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36'
        }
        yield scrapy.Request(url=url, headers=headers, callback=self.parse)