scrapy带cookie的几种方式

发起post请求

    def start_requests(self):
        u = 'https://www.sunstar.com.ph/premium'
        FormData = {
            'action': 'login',
            'redirect_link': 'https://www.sunstar.com.ph/premium',
            'login_input': 'zhanghao',
            'password': 'mima',
        }
        yield scrapy.FormRequest(url=u, formdata=FormData, callback=self.login_all)

    def login_all(self, response):
        self.logger.info(f'我成功登录: {response.url}')
        if response.status == 200:
            # cookiejar.extract_cookies(response, response.request)
            # with open('cookies.txt', 'w') as f:
            #     for cookie in cookiejar:
            #         f.write(str(cookie) + '\n')
            for start_url in sunstar_p_url():
                yield scrapy.Request(url=start_url, callback=self.parse)
settings.py

COOKIES_ENABLED = False
COOKIES_DEBUG = True
COOKIES_DEBUG默认值为False,如果启用，Scrapy将记录所有在request(Cookie 请求头)发送的cookies及response接收到的cookies(Set-Cookie 接收头)。

以下需要设置settings中的Cookies_enabled=True
直接复制cookie

settings.py添加cookies.png

重写start_requests中添加cookie

image.png

去中间件文件中找downloadmiddleware这个类，修改process_request，添加request.cookies={}即可

修改process_request